text: The manuscript proposes a new framework for inference in RNN based upon the Bayes by Backprop (BBB) algorithm.   In particular, the authors propose a new framework to "sharpen" the posterior. In particular, the hierarchical prior in (6) and (7) frame an interesting modification to directly learning a multivariate normal variational approximation.   In the experimental results, it seems clear that this approach is beneficial, but it's not clear as to why.   In particular, how does the variational posterior change as a result of the hierarchical prior?   It seems that (7) would push the center of the variational structure back towards the MAP point and reduces the variance of the output of the hierarchical prior;  however, with the two layers in the prior it's unclear what actually is happening.   Carefully explaining *what* the authors believe is happening and exploring how it changes the variational approximation in a classic modeling framework would be beneficial to understanding the proposed change and evaluating it.   As a final point, the authors state, "as long as the improvement along the gradient is great than the KL loss incurred...this method is guaranteed to make progress towards optimizing L."   Do the authors mean that the negative log-likelihood will be improved in this case?   Or the actual optimization?   Improving the negative log-likelihood seems straightforward,  but I am confused by what the authors mean by optimization. The new evaluation metric proposed in Section 6.1.1 is confusing,  and I do not understand what the metric is trying to capture.   This needs significantly more detail and explanation.   Also, it is unclear to me what would happen when you input data examples that are opposite to the original input sequence;  in particular, for many neural networks the predictions are unstable outside of the input domain and inputting infeasible data leads to unusable outputs.   It's completely feasible that these outputs would just be highly uncertain,  and I'm not sure how you can ascribe meaning to them.   The authors should not compare to the uniform prior as a baseline for entropy.   It's much more revealing to compare it to the empirical likelihoods of the words.