text: Summary: The paper proposes a learnable skimming mechanism for RNN. The model decides whether to send the word to a larger heavy-weight RNN or a light-weight RNN. The heavy-weight and the light-weight RNN each controls a portion of the hidden state. The paper finds that with the proposed skimming method, they achieve a significant reduction in terms of FLOPS. Although it doesn’t contribute to much speedup on modern GPU hardware, there is a good speedup on CPU, and it is more power efficient. Contribution: - The paper proposes to use a small RNN to read unimportant text. Unlike (Yu et al., 2017), which skips the text, here the model decides between small and large RNN. Pros: - Models that dynamically decide the amount of computation make intuitive sense and are of general interests. - The paper presents solid experimentation on various text classification and question answering datasets. - The proposed method has shown reasonable reduction in FLOPS and CPU speedup with no significant accuracy degradation (increase in accuracy in some tasks). - The paper is well written, and the presentation is good. Cons: - Each model component is not novel. The authors propose to use Gumbel softmax, but does compare other gradient estimators. It would be good to use REINFORCE to do a fair comparison with (Yu et al., 2017 ) to see the benefit of using small RNN. - The authors report that training from scratch results in unstable skim rate, while Half pretrain seems to always work better than fully pretrained ones. This makes the success of training a bit adhoc, as one need to actively tune the number of pretraining steps. - Although there is difference from (Yu et al., 2017), the contribution of this paper is still incremental. Questions: - Although it is out of the scope for this paper to achieve GPU level speedup, I am curious to know some numbers on GPU speedup. - One recommended task would probably be text summarization, in which the attended text can contribute to the output of the summary. Conclusion: - Based on the comments above, I recommend Accept