before_revision: Transformer-based language models have taken many fields in NLP by storm. BERT and its derivatives dominate most of the existing evaluation benchmarks, including those for Word Sense Disambiguation (WSD), thanks to their ability in capturing context-sensitive semantic nuances. However, there is still little knowledge about their capabilities and potential limitations for encoding and recovering word senses. In this article, we provide an in-depth quantitative and qualitative analysis of the celebrated BERT model with respect to lexical ambiguity. One of the main conclusions of our analysis is that BERT captures high-level sense distinctions accurately , even when a limited number of examples is available for each word sense. Our analysis also reveals that in some cases language models come close to solving coarse-grained noun disambiguation under ideal conditions in terms of availability of training data and computing resources. However, this scenario rarely occurs in real-world settings and, hence, many practical challenges remain even in the coarse-grained setting. We also perform an in-depth comparison of the two main language model based WSD strategies, i.e., fine-tuning and feature extraction, finding that the latter approach is more robust with respect to sense bias and it can better exploit limited available training data. In fact, a simple feature extraction strategy based on the averaging of contextualized embeddings proves robust even using only three training sentences per word sense, with minimal improvements beyond this small number of examples .
after_revision: Transformer-based language models have taken many fields in NLP by storm. BERT and its derivatives dominate most of the existing evaluation benchmarks, including those for Word Sense Disambiguation (WSD), thanks to their ability in capturing context-sensitive semantic nuances. However, there is still little knowledge about their capabilities and potential limitations in encoding and recovering word senses. In this article, we provide an in-depth quantitative and qualitative analysis of the celebrated BERT model with respect to lexical ambiguity. One of the main conclusions of our analysis is that BERT can accurately capture high-level sense distinctions , even when a limited number of examples is available for each word sense. Our analysis also reveals that in some cases language models come close to solving coarse-grained noun disambiguation under ideal conditions in terms of availability of training data and computing resources. However, this scenario rarely occurs in real-world settings and, hence, many practical challenges remain even in the coarse-grained setting. We also perform an in-depth comparison of the two main language model based WSD strategies, i.e., fine-tuning and feature extraction, finding that the latter approach is more robust with respect to sense bias and it can better exploit limited available training data. In fact, the simple feature extraction strategy of averaging contextualized embeddings proves robust even using only three training sentences per word sense, with minimal improvements obtained by increasing the size of this training data .
edit_actions: R
edit_actions: for
edit_actions: in
edit_actions: fluency
edit_actions: fluency clarity fluency
edit_actions: R
edit_actions: captures
edit_actions: can accurately capture
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: D
edit_actions: accurately
edit_actions: clarity
edit_actions: style clarity clarity
edit_actions: R
edit_actions: a
edit_actions: the
edit_actions: clarity
edit_actions: fluency clarity clarity
edit_actions: R
edit_actions: based on the averaging of
edit_actions: of averaging
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: R
edit_actions: beyond this small number of examples
edit_actions: obtained by increasing the size of this training data
edit_actions: clarity
edit_actions: clarity clarity clarity