before_revision: ROUGE is the de facto criterion for summarization research. However, its two major drawbackslimit the research and application of automated summarization systems . First, ROUGE favors lexical similarity instead of semantic similarity , making it especially unfit for abstractive summarization . Second, ROUGE cannot function without a reference summary, which is expensive or impossible to obtain in many cases . Therefore , we introduce a new end-to-end metric system for summary quality assessment by leveraging the semantic similarities of words and/or sentences in deep learning. Models trained in our framework can evaluate a summary directly against the input document , without the need of a reference summary . The proposed approach exhibits very promising results on gold-standard datasets and suggests its great potential to future summarization research. The scores from our models have correlation coefficients up to 0.54 with human evaluations on machine generated summaries in TAC2010. Its performance is also very close to ROUGE metrics' .
after_revision: Canonical automatic summary evaluation metrics, such as ROUGE, suffer from two drawbacks . First, semantic similarity and linguistic quality are not captured well . Second, a reference summary, which is expensive or impossible to obtain in many cases , is needed. Existing efforts to address the two drawbacks are done separately and have limitations. To holistically address them , we introduce an end-to-end approach for summary quality assessment by leveraging sentence or document embedding and introducing two negative sampling approaches to create training data for this supervised approach . The proposed approach exhibits promising results on several summarization datasets of various domains including news, legislative bills, scientific papers, and patents. When rating machine-generated summaries in TAC2010, our approach outperforms ROUGE in terms of linguistic quality, and achieves a correlation coefficient of up to 0.5702 with human evaluations in terms of modified pyramid scores. We hope our approach can facilitate summarization research or applications when reference summaries are infeasible or costly to obtain, or when linguistic quality is a focus .
edit_actions: R
edit_actions: ROUGE is the de facto criterion for summarization research. However, its two major drawbackslimit the research and application of automated summarization systems
edit_actions: Canonical automatic summary evaluation metrics, such as ROUGE, suffer from two drawbacks
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: R
edit_actions: ROUGE favors lexical similarity instead of semantic similarity , making it especially unfit for abstractive summarization
edit_actions: semantic similarity and linguistic quality are not captured well
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: D
edit_actions: ROUGE cannot function without
edit_actions: coherence
edit_actions: coherence clarity coherence
edit_actions: R
edit_actions: . Therefore
edit_actions: , is needed. Existing efforts to address the two drawbacks are done separately and have limitations. To holistically address them
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: R
edit_actions: a new
edit_actions: an
edit_actions: clarity
edit_actions: clarity coherence clarity
edit_actions: R
edit_actions: metric system
edit_actions: approach
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: R
edit_actions: the semantic similarities of words and/or sentences in deep learning. Models trained in our framework can evaluate a summary directly against the input document , without the need of a reference summary
edit_actions: sentence or document embedding and introducing two negative sampling approaches to create training data for this supervised approach
edit_actions: meaning-changed
edit_actions: clarity meaning-changed meaning-changed
edit_actions: D
edit_actions: very
edit_actions: clarity
edit_actions: coherence clarity clarity
edit_actions: R
edit_actions: gold-standard datasets and suggests its great potential to future summarization research. The scores from our models have correlation coefficients up to 0.54
edit_actions: several summarization datasets of various domains including news, legislative bills, scientific papers, and patents. When rating machine-generated summaries in TAC2010, our approach outperforms ROUGE in terms of linguistic quality, and achieves a correlation coefficient of up to 0.5702
edit_actions: meaning-changed
edit_actions: style meaning-changed meaning-changed
edit_actions: R
edit_actions: on machine generated summaries in TAC2010. Its performance is also very close to ROUGE metrics'
edit_actions: in terms of modified pyramid scores. We hope our approach can facilitate summarization research or applications when reference summaries are infeasible or costly to obtain, or when linguistic quality is a focus
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed clarity