text: The paper treats the interesting problem of long term video prediction in complex video streams.  I think the approach of adding more structure to their representation before making longer term prediction is also a reasonable one.  Their approach combines an RNN that predicts an encoding of scene and then generating an image prediction using a VAN (Reed et al.).  They show some results on the Human3.6M and the Robot Push dataset.  I find the submission lacking clarity in many places.  The main lack of clarity source I think is about what the contribution is.  There are sparse mentions in the introduction  but I think it would be much more forceful and clear if they would present VAN or Villegas et al method separately and then put the pieces together for their method in a separate section.  This would allow the author to clearly delineate their contribution and maybe why those choices were made.  Also the use of hierarchical is non-standard and leads to confusion I recommend maybe "semantical" or better "latent structured" instead.  Smaller ambiguities in wording are also in the paper :  e.g. related work -> long term prediction "in this work" refers to the work mentioned but could as well be the work that they are presenting.   I find some of the claims not clearly backed by a thorough evaluation and analysis.  Claiming to be able to produce encodings of scenes that work well at predicting many steps into the future is a very strong claim.  I find the few images provided very little evidence for that fact.  I think a toy example where this is clearly the case  because we know exactly the factors of variations and they are inferred by the algorithm automatically or some better ones are discovered by the algorithm,  that would make it a very strong submission.  Reed et al. have a few examples that could be adapted to this setting and the resulting representation, analyzed appropriately, would shed some light into whether this is the right approach for long term video prediction and what are the nobs that should be tweaked in this system.  In the current format, I think that the authors are on a good path  and I hope my suggestions will help them improve their submission,  but as it stands I recommend rejection from this conference.