text: This paper proposes a new theoretically-motivated method for combining reinforcement learning and imitation learning for acquiring policies that are as good as or superior to the expert.  The method assumes access to an expert value function (which could be trained using expert roll-outs) and uses the value function to shape the reward function and allow for truncated-horizon policy search.  The algorithm can gracefully handle suboptimal demonstrations/value functions,  since the demonstrations are only used for reward shaping,  and the experiments demonstrate faster convergence and better performance compared to RL and AggreVaTeD on a range of simulated control domains.  The paper is well-written and easy to understand. My main feedback is with regard to the experiments: I appreciate that the experiments used 25 random seeds!  This provides a convincing evaluation. It would be nice to see experimental results on even higher dimensional domains such as the ant, humanoid, or vision-based tasks,  since the experiments seem to suggest that the benefit of the proposed method is diminished in the swimmer and hopper domains compared to the simpler settings. Since the method uses demonstrations,  it would be nice to see three additional comparisons: (a) training with supervised learning on the expert roll-outs, (b) initializing THOR and AggreVaTeD (k=1) with a policy trained with supervised learning, and (c) initializing TRPO with a policy trained with supervised learning.  There doesn't seem to be any reason not to initialize in such a way, when expert demonstrations are available,  and such an initialization should likely provide a significant speed boost in training for all methods. How many demonstrations were used for training the value function in each domain?  I did not see this information in the paper. With regard to the method and discussion: The paper discusses the connection between the proposed method and short-horizon imitation and long-horizon RL, describing the method as a midway point.  It would also be interesting to see a discussion of the relation to inverse RL,  which considers long-term outcomes from expert demonstrations.  For example, MacGlashn & Littman propose a midway point between imitation and inverse RL [1]. Theoretically, would it make sense to anneal k from small to large? (to learn the most effectively from the smallest amount of experience) [1] https://www.ijcai.org/Proceedings/15/Papers/519.pdf Minor feedback: - The RHS of the first inequality in the proof of Thm 3.3 seems to have an error in the indexing of i and exponent, which differs from the line before and line after