text: This paper examines the effects of RL in an augmented action space which includes sequences of actions (e.g. meta actions) as well as the primitive actions defined by the MDP. The authors extend a GPU-based A3C implementation to include meta actions and show that their algorithm can achieve better sample complexity / and higher performance in most cases. The paper is well written, but fails to mention the relationship between meta-actions and the Options framework (Sutton et al 99). In particular, it seems that meta-actions can just be viewed as a set of predefined options given to the agent. Much prior work has studied how to combine options with Deep RL. To name a few: Multi-Level Discovery of Deep Options (Fox et al 17), Classifying Options for Deep RL (Arulkumaran 16), and Deep Exploration via Bootstrapped DQN (Osband et al). The former even learns the options rather than pre-defining them. This connection needs to be made explicit. I have concerns regarding the results in this paper: •	 Why on Qbert is the switching agent able to do so much better than both IU and DU? I suspect the curves may not be averaged over enough trials and results may be noisy, as it seems this shouldn’t be possible. Results curves should show the standard deviation or variance of the 3 runs. •	I am concerned this approach will not scale to games that have more actions than the 4 games explored. The concern is that A4C exponentially increases the size of the action space as a function of k. Cartpole as well as the explored Atari games all have relatively small action spaces, so I think it is critical to show that it scales to games with larger spaces as well. My concern is that in larger actions spaces, the gains A4C gets from meta-actions will be outweighed by the difficulty of having to learn with so many different actions. •	What is the value for k used in Atari experiments? Overall, I was very excited about this paper after reading the introduction. I really like the idea of allowing the network to decide on sequences of actions, and I think many games do have opportunity to identify and re-use combos of primitive actions (e.g. to stay between lanes in Beam Rider). However, I don’t think the architecture, algorithm, and results live up to this motivation. Simply augmenting the action space with all possible sequences of actions begs for a better solution. Pros:•	The authors show that in certain domains, exploration can be aided by pre-defined meta actions. •	The authors introduce an algorithm to squeeze more gradient updates out of a meta-action (DU-A4C). This is related to the insight that meta actions can be thought of as sequences of primitive actions. Cons:•	Relationship to Options is not identified. •	Results are only given for games with small action spaces. Unclear how the method scales to larger action spaces. •	Method for augmenting action space is not particularly interesting. •	Heuristic switching is somewhat undesirable. It would be nice to understand why DU stops working well and how to improve it.