before_revision: Recent studies indicate that NLU models are prone to rely on shortcut features for prediction  . As a result, these models could potentially fail to generalize to real-world out-of-distribution scenarios . In this work, we show that the shortcut learning behavior can be explained by the long-tailed phenomenon . There are two findings: 1) Trained NLU models have strong preference for features located at the head of the long-tailed distribution, and 2) Shortcut features are picked up during very early few iterations of the model training. These two observations are further employed to formulate a measurement which can quantify the shortcut degree of each training sample. Based on this shortcut measurement, we propose a shortcut mitigation framework  , to suppress the model from making overconfident predictions for samples with large shortcut degree. Experimental results on three NLU benchmarks demonstrate that our long-tailed distribution explanation accurately reflects the shortcut learning behavior of NLU models. Experimental analysis further indicates that our method can improve the generalization accuracy on OOD data, while preserving the accuracy on in distribution test data.
after_revision: Recent studies indicate that NLU models are prone to rely on shortcut features for prediction , without achieving true language understanding . As a result, these models fail to generalize to real-world out-of-distribution data . In this work, we show that the words in the NLU training set can be modeled as a long-tailed distribution . There are two findings: 1) NLU models have strong preference for features located at the head of the long-tailed distribution, and 2) Shortcut features are picked up during very early few iterations of the model training. These two observations are further employed to formulate a measurement which can quantify the shortcut degree of each training sample. Based on this shortcut measurement, we propose a shortcut mitigation framework LGTR , to suppress the model from making overconfident predictions for samples with large shortcut degree. Experimental results on three NLU benchmarks demonstrate that our long-tailed distribution explanation accurately reflects the shortcut learning behavior of NLU models. Experimental analysis further indicates that LGTR can improve the generalization accuracy on OOD data, while preserving the accuracy on in-distribution data.
edit_actions: A
edit_actions: , without achieving true language understanding
edit_actions: meaning-changed
edit_actions: meaning-changed coherence meaning-changed
edit_actions: D
edit_actions: could potentially
edit_actions: clarity
edit_actions: clarity clarity style
edit_actions: R
edit_actions: scenarios
edit_actions: data
edit_actions: clarity
edit_actions: style clarity clarity
edit_actions: R
edit_actions: shortcut learning behavior can be explained by the
edit_actions: words in the NLU training set can be modeled as a
edit_actions: clarity
edit_actions: style clarity clarity
edit_actions: R
edit_actions: phenomenon
edit_actions: distribution
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: D
edit_actions: Trained
edit_actions: clarity
edit_actions: clarity clarity fluency
edit_actions: A
edit_actions: LGTR
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: R
edit_actions: our method
edit_actions: LGTR
edit_actions: clarity
edit_actions: clarity coherence clarity
edit_actions: R
edit_actions: in distribution test
edit_actions: in-distribution
edit_actions: clarity
edit_actions: clarity clarity clarity