before_revision: Non-autoregressive neural machine translation achieves remarkable inference acceleration compared to autoregressive models. However, current non-autoregressive models still fall behind their autoregressive counterparts in prediction accuracy. We attribute the accuracy gaps to two disadvantages of non-autoregressive models : a) learning simultaneous generation under the overly strong conditional independence assumption; b) lacking explicit target language modeling. In this paper , we propose Glancing Transformer (GLAT) to address the above disadvantages, which reduces the difficulty of learning simultaneous generation and introduces explicit target language modeling in the non-autoregressive setting at the same time . Experiments on several benchmarks demonstrate that our approach significantly improves the accuracy of non-autoregressive models without sacrificing any inference efficiency . In particular, GLAT achieves 30.91 BLEU on WMT 2014 German-English, which narrows the gap between autoregressive models and non-autoregressive models to less than 0.5 BLEU score .
after_revision: Although non-autoregressive models with one-iteration generation achieve remarkable inference speed-up, they still fall behind their autoregressive counterparts in prediction accuracy. The non-autoregressive models with the best accuracy currently rely on multiple decoding iterations, which largely sacrifice the inference speed of non-autoregressive models . Inspired by the way of learning word dependencies in autoregressive and iterative-decoding models , we propose Glancing Transformer (GLAT) with a glancing language model (GLM), which learns to capture the word dependency gradually . Experiments on three benchmarks demonstrate that our approach can significantly improve the accuracy of non-autoregressive models without multiple decoding iterations . In particular, GLAT achieves state-of-the-art results among non-iterative models and even outperforms top iterative counterparts in some specific benchmarks .
edit_actions: R
edit_actions: Non-autoregressive neural machine translation achieves remarkable inference acceleration compared to autoregressive models. However, current
edit_actions: Although
edit_actions: coherence
edit_actions: coherence coherence coherence
edit_actions: R
edit_actions: models
edit_actions: models with one-iteration generation achieve remarkable inference speed-up, they
edit_actions: meaning-changed
edit_actions: meaning-changed clarity meaning-changed
edit_actions: R
edit_actions: We attribute the accuracy gaps to two disadvantages
edit_actions: The non-autoregressive models with the best accuracy currently rely on multiple decoding iterations, which largely sacrifice the inference speed
edit_actions: meaning-changed
edit_actions: meaning-changed clarity meaning-changed
edit_actions: R
edit_actions: : a) learning simultaneous generation under the overly strong conditional independence assumption; b) lacking explicit target language modeling. In this paper
edit_actions: . Inspired by the way of learning word dependencies in autoregressive and iterative-decoding models
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: R
edit_actions: to address the above disadvantages, which reduces the difficulty of learning simultaneous generation and introduces explicit target language modeling in the non-autoregressive setting at the same time
edit_actions: with a glancing language model (GLM), which learns to capture the word dependency gradually
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed clarity
edit_actions: R
edit_actions: several
edit_actions: three
edit_actions: clarity
edit_actions: clarity clarity meaning-changed
edit_actions: R
edit_actions: significantly improves
edit_actions: can significantly improve
edit_actions: clarity
edit_actions: clarity style clarity
edit_actions: R
edit_actions: sacrificing any inference efficiency
edit_actions: multiple decoding iterations
edit_actions: clarity
edit_actions: clarity clarity meaning-changed
edit_actions: R
edit_actions: 30.91 BLEU on WMT 2014 German-English, which narrows the gap between autoregressive models and non-autoregressive models to less than 0.5 BLEU score
edit_actions: state-of-the-art results among non-iterative models and even outperforms top iterative counterparts in some specific benchmarks
edit_actions: clarity
edit_actions: clarity clarity coherence