text: This paper proposes a family of first-order stochastic optimization schemes based on (1)  normalizing (batches of) stochastic gradient descents and (2) choosing from a step size updating scheme.  The authors argue that iterative first-order optimization algorithms can be interpreted as a choice of an update direction and a step size,  so they suggest that one should always normalize the gradient when computing the direction and then choose a step size using the normalized gradient.  \n\nThe presentation in the paper is clear,  and the exposition is easy to follow. The authors also do a good job of presenting related work and putting their ideas in the proper context.  The authors also test their proposed method on many datasets, which is appreciated.\n\n However, I didn't find the main idea of the paper to be particularly compelling.  The proposed technique is reasonable on its own,  but the empirical results do not come with any measure of statistical significance.  The authors also do not analyze the sensitivity of the different optimization algorithms to hyperparameter choice, opting to only use the default.  Moreover, some algorithms were used as benchmarks on some datasets but not others.  For a primarily empirical paper, every state-of-the-art algorithm should be used as a point of comparison on every dataset considered.  These factors altogether render the experiments uninformative in comparing the proposed suite of algorithms to state-of-the-art methods.  The theoretical result in the convex setting is also not data-dependent, despite the fact that it is the normalized gradient version of AdaGrad, which does come with a data-dependent convergence guarantee.\n\n Given the suite of optimization algorithms in the literature and in use today, any new optimization framework should either demonstrate improved (or at least matching) guarantees in some common (e.g. convex) settings or definitively outperform state-of-the-art methods on problems that are of widespread interest.  Unfortunately, this paper does neither.  \n\nBecause of these points, I do not feel the quality, originality, and significance of the work to be high enough to merit acceptance.  \n\nSome specific comments: \np. 2: \"adaptive feature-dependent step size has attracted lots of attention\".  When you apply feature-dependent step sizes, you are effectively changing the direction of the gradient,  so your meta learning formulation, as posed, doesn't make as much sense. \np. 2: "we hope the resulting methods can benefit from both techniques\".  What reason do you have to hope for this?  Why should they be complimentary?  Existing optimization techniques are based on careful design and coupling of gradients or surrogate gradients, with specific learning rate schedules.  Arbitrarily mixing the two doesn't seem to be theoretically well-motivated. \np. 2: \"numerical results shows that normalized gradient always helps to improve the performance of the original methods when the network structure is deep\".  It would be great to provide some intuition for this.  \np. 2: \"we also provide a convergence proof under this framework when the problem is convex and the stepsize is adaptive\".  The result that you prove guarantees a \\theta(\\sqrt{T}) convergence rate.  On the other hand, the AdaGrad algorithm guarantees a data-dependent bound that is O(\\sqrt{T})  but can also be much smaller.  This suggests that there is no theoretical motivation to use NGD with an adaptive step size over AdaGrad. \np. 2-3: \"NGD can find a \\eps-optimal solution....when the objective function is quasi-convex. ....extended NGD for upper semi-continuous quasiconvex objective functions...\".  This seems like a typo.  How are results that go from quasi-convex to upper semi-continuous quasi-convex an extension? \np. 3: There should be a reference for RMSProp. \np. 3: \"where each block of parameters x^i can be viewed as parameters associated to the ith layer in the network\".  Why is layer parametrization (and later on normalization) a good way idea?  There should be either a reference or an explanation. \np. 4: \"x=(x_1, x_2, \\ldots, x_B)\".  Should these subscripts be superscripts? \np. 4: \"For all the algorithms, we use their default settings.\"  This seems insufficient for an empirical paper,  since most problems often involve some amount of hyperparameter tuning.  How sensitive is each method to the choice of hyperparameters?  What about the impact of initialization? \np. 4-8: None of the experimental results have error bars or any measure of statistical significance. \np. 5: \"NG... is a variant of the NG_{UNIT} method\".  This method is never motivated. \np. 5-6: Why are SGD and Adam used for MNIST but not on CIFAR?  \np. 5: \"we chose the best heyper-paerameter from the 56 layer residual network.\"  Apart from the typos, are these parameters chosen from the training set or the test set?  \np. 6: Why isn't Adam tested on ImageNet?