text: This paper studies new off-policy policy optimization algorithm using relative entropy objective and use EM algorithm to solve it.  The general idea is not new, aka, formulating the MDP problem as a probabilistic inference problem.  There are some technical questions: 1. For parametric EM case, there is asymptotic convergence guarantee to local optima case;  However, for nonparametric EM case, there is no guarantee for that.  This is the biggest concern I have for the theoretical justification of the paper. 2. In section 4, it is said that Retrace algorithm from Munos et al. (2016) is used for policy evaluation.  This is not true.  The Retrace algorithm, is per se, a value iteration algorithm.  I think the author could say using the policy evaluation version of Retrace, or use the truncated importance weights technique as used in Retrace algorithm, which is more accurate. Besides, a minor point: Retrace algorithm is not off-policy stable with function approximation, as shown in several recent papers, such as “Convergent Tree-Backup and Retrace with Function Approximation”.  But this is a minor point if the author doesn’t emphasize too much about off-policy stability. 3. The shifting between the unconstrained multiplier formulation in Eq.9 to the constrained optimization formulation in Eq.10 should be clarified.  Usually, an in-depth analysis between the choice of \lambda in multiplier formulation and the \epsilon in the constraint should be discussed,  which is necessary for further theoretical analysis.  4. The experimental conclusions are conducted without sound evidence.  For example, the author claims the method to be 'highly data efficient' compared with existing approaches,  however, there is no strong evidence supporting this claim.  Overall, although the motivation of this paper is interesting,  I think there is still a lot of details to improve.