text: The authors present a new RL algorithm for sparse reward tasks.  The work is fairly novel in its approach,  combining a learned reward estimator with a contextual bandit algorithm for exploration/exploitation.  The paper was mostly clear in its exposition,  however some additional information of the motivation for why the said reduction is better than simpler alternatives would help.  \n\nPros\n1. The results on bandit structured prediction problems are pretty good\n 2. The idea of a learnt credit assignment function, and using that to separate credit assignment from the exploration/exploitation tradeoff is good.  \n\nCons: \n1. The method seems fairly more complicated than PPO / A2C,  yet those methods seem to perform equally well on the RL problems (Figure 2.).  It also seems to be designed only for discrete action spaces.\n 2. Reslope Boltzmann performs much worse than Reslope Bootstrap,  thus having a bag of policies helps.  However, in the comparison in Figures 2 and 3, the policy gradient methods dont have the advantage of using a bag of policies.  A fairer comparison would be to compare with methods that use ensembles of Q-functions.  (like this https://arxiv.org/abs/1706.01502 by Chen et al.).  The Q learning methods in general would also have better sample efficiency than the policy gradient methods.\n 3. The method claims to learn an internal representation of a denser reward function for the sparse reward problem,  however the experimental analysis of this is pretty limited (Section 5.3).  It would be useful to do a more thorough investigation of whether it learnt a good credit assignment function in the games.  One way to do this would be to check the qualitative aspects of the function in a well understood game, like Blackjack.\n\n Suggestions:\n1. What is the advantage of the method over a simple RL method that predicts a reward at every step (such that the dense rewards add up to match the sparse reward for the episode), and uses this predicted dense reward to perform RL?  This, and also a bigger discussion on prior bandit learning methods like LOLS will help under the context for why we\u2019re performing the reduction stated in the paper. \n\nSignificance: While the method is novel and interesting, the experimental analysis and the explanations in the paper leave it unclear as to whether its significant compared to prior work.