text: This paper describes an extension to the recently introduced Transformer networks which shows better convergence properties and also improves results on standard machine translation benchmarks.  This is a great paper  -- it introduces a relatively simple extension of Transformer networks which only adds very few parameters and speeds up convergence and achieves better results.  It would have been good to also add a motivation for doing this  (for example, this idea can be interpreted as having a variable number of attention heads which can be blended in and out with a single learned parameter, hence making it easier to use the parameters where they are needed).  Also, it would be interesting to see how important the concatenation weight and the addition weight are relative to each other --  do you possibly get the same results even without the concatenation weight?  A suggested improvement: Please check the references in the introduction and see if you can find earlier ones --  for example, language modeling with RNNs has been done for a very long time, not just since 2017 which are the ones you list;  similar for speech recognition etc. (which probably has been done since 1993!).