before_revision: Pre-trained language models ( PTLM) have impressive performance on commonsense inference benchmarks, but their ability to practically employ commonsense to communicate with humans is fiercely debated. Prior evaluations of PTLMs have focused on factual world knowledge or the ability to reason when the necessary knowledge is provided explicitly. However, effective communication with humansrequires inferences based on implicit commonsense relationships, and robustness despite paraphrasing . In the pursuit of advancing fluid human-AI communication, we propose a new challenge, RICA  , that evaluates the capabilities of making commonsense inferences and the robustness of these inferences to language variations. In our work , we develop a systematic procedure to probe PTLMs across three different evaluation settings. Extensive experiments on our generated probe sets  show that PTLMs perform no better than random guessing (even with fine-tuning) , are heavily impacted by statistical biases, and are not robust to perturbation attacks. Our framework and probe sets can help future work improve PTLMs ' inference abilities and robustness to linguistic variations--bringing us closer to more fluid communication .
after_revision: Pre-trained language models ( PTLMs) have achieved impressive performance on commonsense inference benchmarks, but their ability to employ commonsense to make robust inferences, which is crucial for effective communications with humans, is debated . In the pursuit of advancing fluid human-AI communication, we propose a new challenge, RICA : Robust Inference capability based on Commonsense Axioms , that evaluates robust commonsense inference despite textual perturbations. To generate data for this challenge , we develop a systematic and scalable procedure using commonsense knowledge bases and probe PTLMs across two different evaluation settings. Extensive experiments on our generated probe sets with more than 10k statements show that PTLMs perform no better than random guessing on the zero-shot setting , are heavily impacted by statistical biases, and are not robust to perturbation attacks. We also find that fine-tuning on similar statements offer limited gains, as PTLMs still fail to generalize to unseen inferences. Our new large-scale benchmark exposes a significant gap between PTLMs and human-level language understanding and offers a new challenge for PTLMs to demonstrate commonsense .
edit_actions: R
edit_actions: PTLM) have
edit_actions: PTLMs) have achieved
edit_actions: style
edit_actions: clarity style style
edit_actions: D
edit_actions: practically
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: R
edit_actions: communicate with humans is fiercely debated. Prior evaluations of PTLMs have focused on factual world knowledge or the ability to reason when the necessary knowledge is provided explicitly. However, effective communication with humansrequires inferences based on implicit commonsense relationships, and robustness despite paraphrasing
edit_actions: make robust inferences, which is crucial for effective communications with humans, is debated
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: A
edit_actions: : Robust Inference capability based on Commonsense Axioms
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: R
edit_actions: the capabilities of making commonsense inferences and the robustness of these inferences to language variations. In our work
edit_actions: robust commonsense inference despite textual perturbations. To generate data for this challenge
edit_actions: clarity
edit_actions: clarity meaning-changed clarity
edit_actions: R
edit_actions: procedure to
edit_actions: and scalable procedure using commonsense knowledge bases and
edit_actions: meaning-changed
edit_actions: clarity meaning-changed meaning-changed
edit_actions: R
edit_actions: three
edit_actions: two
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: A
edit_actions: with more than 10k statements
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: R
edit_actions: (even with fine-tuning)
edit_actions: on the zero-shot setting
edit_actions: clarity
edit_actions: clarity meaning-changed clarity
edit_actions: R
edit_actions: Our framework and probe sets can help future work improve PTLMs ' inference abilities and robustness to linguistic variations--bringing us closer to more fluid communication
edit_actions: We also find that fine-tuning on similar statements offer limited gains, as PTLMs still fail to generalize to unseen inferences. Our new large-scale benchmark exposes a significant gap between PTLMs and human-level language understanding and offers a new challenge for PTLMs to demonstrate commonsense
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed