text: In this paper, the authors investigate variance reduction techniques for agents with multi-dimensional policy outputs, in particular when they are conditionally independent ('factored'). With the increasing focus on applying RL methods to continuous control problems and RTS type games, this is an important problem and this technique seems like an important addition to the RL toolbox. The paper is well written, the method is easy to implement, and the algorithm seems to have clear positive impact on the presented experiments. - The derivations in pages 4-6 are somewhat disconnected from the rest of the paper: the optimal baseline derivation is very standard (even if adapted to the slightly different situation situated here), and for reasons highlighted by the authors in this paper, they are not often used; the 'marginalized' baseline is more common, and indeed, the authors adopt this one as well. In light of this (and of the paper being quite a bit over the page limit)- is this material (4.2->4.4) mostly not better suited for the appendix? Same for section 4.6 (which I believe is not used in the experiments). - The experimental section is very strong; regarding the partial observability experiments, assuming actions are here factored as well, I could see four baselines (two choices for whether the baseline has access to the goal location or not, and two choices for whether the baseline has access to the vector $a_{-i}$). It's not clear which two baselines are depicted in 5b - is it possible to disentangle the effect of providing $a_{-i}$ and the location of the hole to the baseline? (side note: it is an interesting idea to include information not available to the agent as input to the baseline though it does feel a bit 'iffy' ; the agent requires information to train, but is not provided the information to act. Out of curiosity, is it intended as an experiment to verify the need for better baselines? Or as a 'fair' training procedure?) - Minor: in equation 2- is the correct exponent not t'? Also since $\rho_\pi$ is define with a scaling $(1-\gamma)$ (to make it an actual distribution), I believe the definition of $\eta$ should also be multiplied by $(1-\gamma)$ (as well as equation 2).