before_revision: Although non-autoregressive models with one-iteration generation achieve remarkable inference speed-up, they still fall behind their autoregressive counterparts in prediction accuracy. The non-autoregressive models with the best accuracy currently rely on multiple decoding iterations, which largely sacrifice the inference speed of non-autoregressive models. Inspired by the way of learning word dependencies in autoregressive and iterative-decoding models, we propose Glancing Transformer (GLAT) with a glancing language model (GLM), which learns to capture the word dependency gradually . Experiments on three benchmarks demonstrate that our approach can significantly improve the accuracy of non-autoregressive models without multiple decoding iterations. In particular, GLAT achieves state-of-the-art results among non-iterative models and even outperforms top iterative counterparts in some specific benchmarks .
after_revision: Recent work on non-autoregressive neural machine translation (NAT) aims at improving the efficiency by parallel decoding without sacrificing the quality. However, existing NAT methods are either inferior to Transformer or require multiple decoding passes, leading to reduced speedup. We propose the Glancing Language Model (GLM), a method to learn word interdependency for single-pass parallel generation models. With GLM, we develop Glancing Transformer (GLAT) for machine translation. With only single-pass parallel decoding, GLAT is able to generate high-quality translation with 8-15 times speedup . Experiments on multiple WMT language directions show that GLAT outperforms all previous single pass non-autoregressive methods, and is nearly comparable to Transformer, reducing the gap to 0.25-0.9 BLEU points .
edit_actions: R
edit_actions: Although
edit_actions: Recent work on
edit_actions: clarity
edit_actions: style clarity clarity
edit_actions: R
edit_actions: models with one-iteration generation achieve remarkable inference speed-up, they still fall behind their autoregressive counterparts in prediction accuracy. The non-autoregressive models with the best accuracy currently rely on multiple decoding iterations, which largely sacrifice the inference speed of non-autoregressive models. Inspired by the way of learning word dependencies in autoregressive and iterative-decoding models, we propose
edit_actions: neural machine translation (NAT) aims at improving the efficiency by parallel decoding without sacrificing the quality. However, existing NAT methods are either inferior to Transformer or require multiple decoding passes, leading to reduced speedup. We propose the Glancing Language Model (GLM), a method to learn word interdependency for single-pass parallel generation models. With GLM, we develop
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed clarity
edit_actions: R
edit_actions: with a glancing language model (GLM), which learns to capture the word dependency gradually
edit_actions: for machine translation. With only single-pass parallel decoding, GLAT is able to generate high-quality translation with 8-15 times speedup
edit_actions: meaning-changed
edit_actions: clarity meaning-changed meaning-changed
edit_actions: R
edit_actions: three benchmarks demonstrate that our approach can significantly improve the accuracy of
edit_actions: multiple WMT language directions show that GLAT outperforms all previous single pass
edit_actions: clarity
edit_actions: clarity clarity meaning-changed
edit_actions: R
edit_actions: models without multiple decoding iterations. In particular, GLAT achieves state-of-the-art results among non-iterative models and even outperforms top iterative counterparts in some specific benchmarks
edit_actions: methods, and is nearly comparable to Transformer, reducing the gap to 0.25-0.9 BLEU points
edit_actions: meaning-changed
edit_actions: clarity meaning-changed meaning-changed