text: Summary:This paper proposes a simple recipe to preserve proximity to zero mean for activations in deep neural networks. The proposal is to replace the non-linearity in half of the units in each layer with its "bipolar" version -- one that is obtained by flipping the function on both axes. The technique is tested on deep stacks of recurrent layers, and on convolutional networks with depth of 28, showing that improved results over the baseline networks are obtained. Clarity: The paper is easy to read. The plots in Fig. 2 and the appendix are quite helpful in improving presentation. The experimental setups are explained in detail. Quality and significance: The main idea from this paper is simple and intuitive. However, the experiments to support the idea do not seem to match the motivation of the paper. As stated in the beginning of the paper, the motivation behind having close to zero mean activations is that this is expected to speed up training using gradient descent. However, the presented results focus on the performance on held-out data instead of improvements in training speed. This is especially the case for the RNN experiments. For the CIFAR-10 experiment, the training loss curves do show faster initial progress in learning. However, it is unclear that overall training time can be reduced with the help of this technique. To evaluate this speed up effect, the dependence on the choice of learning rate and other hyperparameters should also be considered. Nevertheless, it is interesting to note the result that the proposed approach converts a deep network that does not train into one which does in many cases. The method appears to improve the training for moderately deep convolutional networks without batch normalization (although this is tested on a single dataset), but is not practically useful yet since the regularization benefits of Batch Normalization are also taken away.