text: This paper provides theoretical and empirical motivations for removing the top few principle components of commonly-used word embeddings. The paper is well-written and I enjoyed reading it.  However, it does not explain how significant this result is beyond that of (Bullinaria and Levy, 2012),  who also removed the top N dimensions when benchmarking SVD-factorized word embeddings.  From what I can see, this paper provides a more detailed explanation of the phenomenon ("why" it works),  which is supported with both theoretical results and a series of empirical analyses, as well as "updating" the benchmarks and methods from the pre-neural era.  Although this contribution is relatively incremental,  I find the depth of this work very interesting,  and I think future work could perhaps rely on these insights to create better embedding algorithms that directly enforce isotropy. I have two concerns regarding the empirical section, which may be resolvable fairly quickly: 1) Are the embedding vectors L2 normalized before using them in each task?  This is known to significantly affect performance.  I am curious whether removing the top PCs is redundant or not given L2 normalization. 2) Most of the benchmarks used in this paper are "toy" tasks.  As Schnabel et al (2015) and Tsvetkov et al (2015) showed, there is often little correlation between success on these benchmarks and improvement of downstream NLP tasks.  I would like to measure the change in performance on a major NLP task that heavily relies on pre-trained word embeddings such as SQuAD. Minor Comments:* The last sentence in the first paragraph ("The success comes from the geometry of the representations...") is not true;  the success stems from the ability to capture lexical similarity.  Levy and Goldberg (2014) showed that searching for the closest word vector to (king - man + woman) is equivalent to optimizing a linear combination of 3 similarity terms [+(x,king), -(x,man), +(x, woman)].  This explanation was further demonstrated by Linzen (2016) who showed that even when removing the negative term (x, man), many analogies can still be solved, i.e. by looking for a word that is similar both to "king" and to "woman".  Add to that the fact that the analogy trick works best when the vectors are L2 normalized;  if they are all on the unit sphere, what is the geometric interpretation of (king - man + woman), which is not on the unit sphere?  I suggest removing this sentence and other references to linguistic regularities from this paper,  since they are controversial at best, and distract from the main findings. * This is also related to Bullinaria and Levy's (2012) finding that downweighting the eigenvalue matrix in SVD-based methods improves their performance.  Levy et al (2015) showed that keeping the original eigenvalues can actually degenerate SVD-based embeddings.  Perhaps there is a connection to the findings in this paper?