text: The paper addresses the problem of learning the form of the activation functions in neural networks. The authors propose to place Gaussian process (GP) priors on the functional form of each activation function (each associated with a hidden layer and unit) in the neural net. This  somehow allows to non-parametrically infer from the data the "shape" of the activation functions needed for a specific problem. The paper then proposes an inference framework (to approximately marginalize out all GP functions)  based on sparse GP methods that use inducing points and variational inference. The inducing point approximation used here is very efficient since all GP functions depend on a scalar input (as any activation function!) and therefore by just placing the inducing points in a dense grid gives a fast and accurate representation/compression of all GPs in terms of the inducing function values (denoted by U in the paper). Of course then inference involves approximating the finite posterior over inducing function values U and the paper make use of the standard Gaussian approximations. In general I like the idea and I believe that it can lead to a very useful model. However, I have found the current paper quite preliminary and incomplete. The authors need to address the following: First (very important): You need to show experimentally how your method compares against regular neural nets (with specific fixed forms for their activation functions such relus etc). At the moment in the last section you mention "We have validated networks of Gaussian Process Neurons in a set of experiments, the details of which we submit in a subsequent publication. In those experiments, our model shows to be significantly less prone to overfitting than a traditional feed-forward network of same size, despite having more parameters." ===>  Well all this needs to be included in the same paper. Secondly: Discuss the connection with Deep GPs (Damianou and Lawrence 2013). Your method seems to be connected with Deep GPs although there appear to be important differences as well. E.g. you place GPs on the scalar activation functions in an otherwise  heavily parametrized neural network (having interconnection weights between layers) while deep GPs model the full hidden layer mapping as a single GP (which does not require interconnection weights). Thirdly:  You need to better explain the propagation of uncertainly in section 3.2.2  and the central limit of distribution in section 3.2.1. This is the technical part of your paper which is a non-standard approximation. I will suggest to give a better intuition of the whole idea and move a lot of mathematical details to the appendix.