before_revision: In the automatic evaluation of generative question answering (GenQA) systems, it is difficult to assess the correctness of generated answers due to the free-form of the answer. Moreover, there is a lack of benchmark datasets to evaluate the suitability of existing metrics in terms of correctness. To study a better metric for GenQA, we first create high-quality human judgments of correctness on two standard GenQA datasets. Using our human-evaluation datasets, we show that widely used n-gram similarity metrics do not correlate with human judgments . To alleviate this problem, we propose  a new metric for evaluating the correctness of GenQA. Specifically, our new metric assigns different weights to each token via keyphrase prediction, thereby judging whether a generated answer sentence captures the key meaning of the reference answer. Our proposed metric shows a significantly higher correlation with human judgments than existing metrics in various datasets.
after_revision: In the automatic evaluation of generative question answering (GenQA) systems, it is difficult to assess the correctness of generated answers due to the free-form of the answer. Especially, widely used n-gram similarity metrics often fail to discriminate the incorrect answers since they equally consider all of the tokens . To alleviate this problem, we propose KPQA-metric, a new metric for evaluating the correctness of GenQA. Specifically, our new metric assigns different weights to each token via keyphrase prediction, thereby judging whether a generated answer sentence captures the key meaning of the reference answer. To evaluate our metric, we create high-quality human judgments of correctness on two GenQA datasets. Using our human-evaluation datasets, we show that our proposed metric has a significantly higher correlation with human judgments than existing metrics . The code is available at URL
edit_actions: R
edit_actions: Moreover, there is a lack of benchmark datasets to evaluate the suitability of existing metrics in terms of correctness. To study a better metric for GenQA, we first create high-quality human judgments of correctness on two standard GenQA datasets. Using our human-evaluation datasets, we show that
edit_actions: Especially,
edit_actions: coherence
edit_actions: coherence clarity coherence
edit_actions: R
edit_actions: do not correlate with human judgments
edit_actions: often fail to discriminate the incorrect answers since they equally consider all of the tokens
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: A
edit_actions: KPQA-metric,
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: R
edit_actions: Our proposed metric shows
edit_actions: To evaluate our metric, we create high-quality human judgments of correctness on two GenQA datasets. Using our human-evaluation datasets, we show that our proposed metric has
edit_actions: meaning-changed
edit_actions: coherence meaning-changed meaning-changed
edit_actions: R
edit_actions: in various datasets.
edit_actions: . The code is available at URL
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed