text: The paper proposes a novel way of compressing gradient updates for distributed SGD, in order to speed up overall execution. While the technique is novel as far as I know (eq. (1) in particular), many details in the paper are poorly explained (I am unable to understand) and experimental results do not demonstrate that the problem targeted is actually alleviated. More detailed remarks: 1: Motivating with ImageNet taking over a week to train seems misplaced when we have papers claiming to train ImageNet in 1 hour, 24 mins, 15 mins... 4.1: Lemma 4.1 seems like you want B > 1, or clarify definition of V_B. 4.2: This section is not fully comprehensible to me. - It seems you are confusingly overloading the term gradient and words derived (also in other parts or the paper). What is "maximum value of gradients in a matrix"? Make sure to use something else, when talking about individual elements of a vector (which is constructed as an average of gradients), etc. - Rounding: do you use deterministic or random rounding? Do you then again store the inaccuracy? - I don't understand definition of d. It seems you subtract logarithm of a gradient from a scalar. - In total, I really don't know what is the object that actually gets communicated, and consequently when you remark that this can be combined with QSGD and the more below it, I don't understand it. This section has to be thoroughly explained, perhaps with some illustrative examples. 4.3: allgatherv remark: does that mean that this approach would not scale well to higher number of workers? 4.4: Remarks about quantization and mantissa manipulation are not clear to me again, or what is the point in doing so. Possible because the problems above. 5: I think this section is not too useful unless you can accompany it with actual efficient implementation and contrast the practical performance. 6: Given that I don't understand how you compress the information being communicated, it is hard to believe the utility of the method. The objective was to speed up training time because communication is bottleneck. If you provide 12,000x compression, is it any more practically useful than providing 120x compression? What would be the difference in runtime? Such questions are never discussed. Further, if in the implementation you discuss masking mantissa, I have serious concern about whether the compression protocol is feasible to implement efficiently, without writing some extremely low-level code. I think the soundness of work addressing this particular problem is damaged if not implemented properly (compared to other kinds of works in current ML related research). Therefore I highly recommend including proper time comparison with a baseline in the future. Further, I don't understand 2 things about the Tables. a) how do you combine the proposed method with Momentum in SGD? This is not discussed as far as I can see. b) What is "QSGD, 2bit" If I remember QSGD protocol correctly, there's no natural mapping of 2bit to its parameters.