text: This paper proposes training binary and ternary weight distribution networks through the local reparametrization trick and continuous optimization.  The argument is that due to the central limit theorem (CLT) the distribution on the neuron pre-activations is approximately Gaussian, with a mean given by the inner product between the input and the mean of the weight distribution and a variance given by the inner product between the squared input and the variance of the weight distribution.  As a result, the parameters of the underlying discrete distribution can be optimized via backpropagation by sampling the neuron pre-activations with the reparametrization trick.  The authors further propose appropriate initialisation schemes and regularization techniques to either prevent the violation of the CLT or to prevent underfitting.  The method is evaluated on multiple experiments. This paper proposed a relatively simple idea for training networks with discrete weights that seems to work in practice.  My main issue is that while the authors argue about novelty,  the first application of CLT for sampling neuron pre-activations at neural networks with discrete r.v.s is performed at [1].  While [1] was only interested in faster convergence and not on optimization of the parameters of the underlying distribution,  the extension was very straightforward.  I would thus suggest that the authors update the paper accordingly.  Other than that, I have some other comments: - The L2 regularization on the distribution parameters for the ternary weights is a bit ad-hoc;  why not penalise according to the entropy of the distribution which is exactly what you are trying to achieve?  - For the binary setting you mentioned that you had to reduce the entropy thus added a “beta density regulariser”.  Did you add R(p) or log R(p) to the objective function?  Also, with alpha, beta = 2 the beta density is unimodal with a peak at p=0.5;  essentially this will force the probabilities to be close to 0.5, i.e. exactly what you are trying to avoid.  To force the probability near the endpoints you have to use alpha, beta < 1 which results into a “bowl” shaped Beta distribution.  I thus wonder whether any gains you observed from this regulariser are just an artifact of optimization. - I think that a baseline (at least for the binary case) where you learn the weights with a continuous relaxation, such as the concrete distribution, and not via CLT would be helpful.  Maybe for the network to properly converge the entropy for some of the weights needs to become small (hence break the CLT).  [1] Wang & Manning, Fast Dropout Training.