before_revision: We propose a novel method to sparsify attention in the Transformer model by learning to select the most-informative token representations , thus leveraging the model's information bottleneck with twofold strength. A careful analysis shows that the contextualization of encoded representations in our model is significantly more effective than in the original Transformer. We achieve a notable reduction in memory usage due to an improved differentiable top-k operator , making the model suitable to process long documents, as shown on an example of a summarization task .
after_revision: We propose a novel method to sparsify attention in the Transformer model by learning to select the most-informative token representations during the training process, thus focusing on task-specific parts of the input. A reduction of quadratic time and memory complexity to sublinear was achieved due to a robust differentiable top-k operator . For example, our experiments on a challenging summarization task of long documents show that our method is much faster and up to 16 times more memory efficient while significantly outperforming both dense and state-of-the-art sparse transformer models. The method can be effortlessly applied to many models used in NLP and CV, simultaneously with other improvements since representation pooling addresses a different aspect of the attention's complexity problem .
edit_actions: R
edit_actions: , thus leveraging the model's information bottleneck with twofold strength. A careful analysis shows that the contextualization of encoded representations in our model is significantly more effective than in the original Transformer. We achieve a notable reduction in memory usage due to an improved
edit_actions: during the training process, thus focusing on task-specific parts of the input. A reduction of quadratic time and memory complexity to sublinear was achieved due to a robust
edit_actions: clarity
edit_actions: clarity coherence clarity
edit_actions: R
edit_actions: , making the model suitable to process long documents, as shown on an example of a summarization task
edit_actions: . For example, our experiments on a challenging summarization task of long documents show that our method is much faster and up to 16 times more memory efficient while significantly outperforming both dense and state-of-the-art sparse transformer models. The method can be effortlessly applied to many models used in NLP and CV, simultaneously with other improvements since representation pooling addresses a different aspect of the attention's complexity problem
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed fluency