text: The authors present two autoregressive models for sampling action probabilities from a factorized discrete action space.  On a multi-agent gridworld task and a multi-agent multi-armed bandit task, the proposed method seems to benefit from their lower-variance entropy estimator for exploration bonus.  A few key citations were missing - notably the LSTM model they propose is a clear instance of an autoregressive density estimator, as in PixelCNN, WaveNet and other recently popular deep architectures.  In that context, this work can be viewed as applying deep autoregressive density estimators to policy gradient methods.  At least one of those papers ought to be cited.  It also seems like a simple, obvious baseline is missing from their experiments - simply independently outputting D independent softmaxes from the policy network.  Without that baseline it's not clear that any actual benefit is gained by modeling the joint distribution between actions, especially since the optimal policy for an MDP is provably deterministic anyway.  The method could even be made to capture dependencies between different actions by adding a latent probabilistic layer in the middle of the policy network, inducing marginal dependencies between different actions.  A direct comparison against one of the related methods in the discussion section would help better contextualize the paper as well.  A final point on clarity of presentation - in keeping with the convention in the field, the readability of the tables could be improved by putting the top-performing models in bold, and Table 2 should almost certainly be replaced by a boxplot.