before_revision: Lead bias is a common phenomenon in news summarization, where early parts of an article often contain the most salient information. While many algorithms exploit this fact in summary generation, it has a detrimental effect on teaching the model to discriminate and extract important information  . We propose that the lead bias can be leveraged in  a simple and effective way in our favor to pretrain abstractive news summarization models on large-scale unlabeled corpus : predicting the leading sentences using the rest of an article. Via careful data cleaning and filtering , our transformer-based pretrained model without any finetuning achieves remarkable results over various news summarization tasks. With further finetuning, our model outperforms many competitive baseline models. Human evaluations further show the effectiveness of our method .
after_revision: Lead bias is a common phenomenon in news summarization, where early parts of an article often contain the most salient information. While many algorithms exploit this fact in summary generation, it has a detrimental effect on teaching the model to discriminate and extract important information in general . We propose that the lead bias can be leveraged in our favor in a simple and effective way to pre-train abstractive news summarization models on large-scale unlabeled news corpora : predicting the leading sentences using the rest of an article. We collect a massive news corpus and conduct data cleaning and filtering via statistical analysis. We then apply the proposed self-supervised pre-training to existing generation models BART and T5 for domain adaptation. Via extensive experiments on six benchmark datasets, we show that this approach can dramatically improve the summarization quality and achieve state-of-the-art results for zero-shot news summarization without any fine-tuning. For example, in the DUC2003 dataset, the ROUGE-1 score of BART increases 13.7\% after the lead-bias pre-training. We deploy the model in Microsoft News and provide public APIs as well as a demo website for multi-lingual news summarization .
edit_actions: A
edit_actions: in general
edit_actions: clarity
edit_actions: clarity meaning-changed clarity
edit_actions: A
edit_actions: our favor in
edit_actions: meaning-changed
edit_actions: clarity meaning-changed meaning-changed
edit_actions: R
edit_actions: in our favor to pretrain
edit_actions: to pre-train
edit_actions: clarity
edit_actions: clarity clarity fluency
edit_actions: R
edit_actions: corpus
edit_actions: news corpora
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: R
edit_actions: Via careful
edit_actions: We collect a massive news corpus and conduct
edit_actions: clarity
edit_actions: clarity clarity coherence
edit_actions: R
edit_actions: , our transformer-based pretrained model without any finetuning achieves remarkable results over various news summarization tasks. With further finetuning, our model outperforms many competitive baseline models. Human evaluations further show the effectiveness of our method
edit_actions: via statistical analysis. We then apply the proposed self-supervised pre-training to existing generation models BART and T5 for domain adaptation. Via extensive experiments on six benchmark datasets, we show that this approach can dramatically improve the summarization quality and achieve state-of-the-art results for zero-shot news summarization without any fine-tuning. For example, in the DUC2003 dataset, the ROUGE-1 score of BART increases 13.7\% after the lead-bias pre-training. We deploy the model in Microsoft News and provide public APIs as well as a demo website for multi-lingual news summarization
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed