text: The paper investigates the iterative estimation view on gated recurrent networks (GNN).  Authors observe that the average estimation error between a given hidden state and the last hidden state  gradually decreases toward zeros.  This suggest that GNN are bias toward an identity mapping and learn to preserve the activation through time. Given this observation, authors then propose RIN, a new RNN parametrization where the hidden to hidden matrix is decomposed as a learnable weight matrix plus the identity matrix. Authors evaluate their RIN on the adding, sequential MNIST and the baby tasks and show that their IRNN outperforms the IRNN and LSTM models. Questions:- Section 2 suggests that use of the gate  in GNNs encourages to learn an identity mapping.  Does the average iteration error behaves differently in case of a tanh-RNN ? - It seems from Figure 4 (a) that the average estimation error is higher for RIN than IRNN and LSTM and only decrease toward zero at the very end. What could explain this phenomenon? - While the LSTM baseline matches the results of Le et al.,  later work such as Recurrent Batch Normalization or Unitary Evolution RNN have demonstrated much better performance with a vanilla LSTM on those tasks (outperforming both IRNN and RIN).  What could explain this difference in the performances? - Unless I am mistaken, Gated Orthogonal Recurrent Units: On Learning to Forget from Jing et al. also reports better performances for the LSTM (and GRU) baselines that outperform RIN on the baby tasks with mean performances of 58.2 and 56.0 for GRU and LSTM respectively? - Quality/Clarity:The paper is well written and pleasant to read - Originality:Looking at RNN from an iterative refinement point of view seems novel. - Significance:While looking at RNN from an iterative estimation is interesting,  the experimental part does not really show what are the advantages of the propose RIN.  In particular, the LSTM baseline seems to weak compared to other works.