before_revision: Pre-trained language models (PTLM) have greatly improved performance on commonsense inference benchmarks, however, it remains unclear whether they share a human's ability to consistently make correct inferences under perturbations. Prior studies of PTLMs have found inference deficits, but have failed to provide a systematic means of understanding whether these deficits are due to low inference abilities or poor inference robustness . In this work, we address this gap by developing a procedure that allows for the systematized probing of both PTLMs' inference abilities and robustness. Our procedure centers around the methodical creation of logically-equivalent, but syntactically-different sets of probes, of which we create a corpus of 14,400 probes coming from 60 logically-equivalent sets that can be used to probe PTLMs in three task settings. We find that despite the recent success of large PTLMs on commonsense benchmarks, their performances on our probes are no better than random guessing (even with fine-tuning) and are heavily dependent on biases--the poor overall performance, unfortunately, inhibits us from studying robustness. We hope our approach and initial probe set will assist future work in improving PTLMs' inference abilities , while also providing a probing set to test robustness under several linguistic variations--code and data will be released .
after_revision: Pre-trained language models (PTLM) have impressive performance on commonsense inference benchmarks, but their ability to practically employ commonsense to communicate with humans is fiercely debated. Prior evaluations of PTLMs have focused on factual world knowledge or the ability to reason when the necessary knowledge is provided explicitly. However, effective communication with humans requires inferences based on implicit commonsense relationships, and robustness despite paraphrasing. In the pursuit of advancing fluid human-AI communication, we propose a new challenge, RICA, that evaluates the capabilities of making commonsense inferences and the robustness of these inferences to language variations. In our work, we develop a systematic procedure to probe PTLMs across three different evaluation settings. Extensive experiments on our generated probe sets show that PTLMs perform no better than random guessing (even with fine-tuning) , are heavily impacted by statistical biases, and are not robust to perturbation attacks. Our framework and probe sets can help future work improve PTLMs' inference abilities and robustness to linguistic variations--bringing us closer to more fluid communication .
edit_actions: R
edit_actions: greatly improved
edit_actions: impressive
edit_actions: clarity
edit_actions: clarity style clarity
edit_actions: R
edit_actions: however, it remains unclear whether they share a human's ability to consistently make correct inferences under perturbations. Prior studies
edit_actions: but their ability to practically employ commonsense to communicate with humans is fiercely debated. Prior evaluations
edit_actions: clarity
edit_actions: clarity style clarity
edit_actions: R
edit_actions: found inference deficits, but have failed to provide a systematic means of understanding whether these deficits are due to low inference abilities or poor inference robustness . In this work, we address this gap by developing a procedure that allows for the systematized probing of both PTLMs' inference abilities and robustness. Our procedure centers around the methodical creation of logically-equivalent, but syntactically-different sets of probes, of which we create a corpus of 14,400 probes coming from 60 logically-equivalent sets that can be used
edit_actions: focused on factual world knowledge or the ability to reason when the necessary knowledge is provided explicitly. However, effective communication with humans requires inferences based on implicit commonsense relationships, and robustness despite paraphrasing. In the pursuit of advancing fluid human-AI communication, we propose a new challenge, RICA, that evaluates the capabilities of making commonsense inferences and the robustness of these inferences to language variations. In our work, we develop a systematic procedure
edit_actions: meaning-changed
edit_actions: clarity meaning-changed meaning-changed
edit_actions: R
edit_actions: in three task settings. We find that despite the recent success of large PTLMs on commonsense benchmarks, their performances on our probes are
edit_actions: across three different evaluation settings. Extensive experiments on our generated probe sets show that PTLMs perform
edit_actions: clarity
edit_actions: clarity clarity style
edit_actions: R
edit_actions: and are heavily dependent on biases--the poor overall performance, unfortunately, inhibits us from studying robustness. We hope our approach and initial probe set will assist future work in improving
edit_actions: , are heavily impacted by statistical biases, and are not robust to perturbation attacks. Our framework and probe sets can help future work improve
edit_actions: clarity
edit_actions: clarity coherence coherence
edit_actions: R
edit_actions: , while also providing a probing set to test robustness under several linguistic variations--code and data will be released
edit_actions: and robustness to linguistic variations--bringing us closer to more fluid communication
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed