before_revision: For natural language processing ' text-to-text' tasks, the prevailing approaches heavily rely on pretraining large self-supervised models on massive external datasources, which incurs exceptional pretraining data requirements and a diminished ability to pretrain over small datasets. However, fundamental pretraining method capabilities like few to zero-shot learningor preserving minority concept ( long-tail ) prediction performance along with accordingly designed evaluation scenarios remain open challenges . We thus propose Contrastive Label-Embedding Self-Supervision (CLESS) pretraining , which enables pretraining from multiple magnitudes smaller, 'task internal' data only, while still strongly improving fully supervised, long-tail , few-shot and self-supervised zero-shot learning abilities. Accordingly, we analyse improvements in learning dynamics over baselines on a challenging long-tailed, low-resource, multi-label text classification scenario with noisy, highly sparse labels and many minority concepts . We find that long-tailed zero and few-shot learning markedly benefit from increasing 'dataset-internal' self-supervised pretraining signals, to help reduce the reliance on large external sources .
after_revision: For natural language processing ` text-to-text' tasks, the prevailing approaches heavily rely on pretraining large self-supervised models on increasingly larger `task-external' data. Transfer learning from high-resource pretraining works well, but research has focused on settings with very large data and compute requirements, while the potential of efficient low-resource learning, without large `task-external' pretraining, remains under-explored. In this work, we evaluate against three core challenges for resource efficient learning. Namely, we analyze: (1) pretraining data (X) efficiency; (2) zero to few-shot label (Y) efficiency; and (3) long-tail generalization, since long-tail preservation has been linked to algorithmic fairness and because data in the tail is limited by definition. To address these challenges, we propose a data and compute efficient self-supervised, contrastive text encoder, pretrained on 60MB of `task-internal' text data, and compare it to RoBERTa, which was pretrained on 160GB of `task-external' text . We find our method outperforms RoBERTa, while pretraining and fine-tuning in a 1/5th of RoBERTa's fine-tuning time .
edit_actions: R
edit_actions: '
edit_actions: `
edit_actions: fluency
edit_actions: fluency fluency fluency
edit_actions: R
edit_actions: massive external datasources, which incurs exceptional pretraining
edit_actions: increasingly larger `task-external' data. Transfer learning from high-resource pretraining works well, but research has focused on settings with very large data and compute requirements, while the potential of efficient low-resource learning, without large `task-external' pretraining, remains under-explored. In this work, we evaluate against three core challenges for resource efficient learning. Namely, we analyze: (1) pretraining
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed meaning-changed
edit_actions: R
edit_actions: requirements and a diminished ability to pretrain over small datasets. However, fundamental pretraining method capabilities like few to zero-shot learningor preserving minority concept (
edit_actions: (X) efficiency; (2) zero to few-shot label (Y) efficiency; and (3)
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed clarity
edit_actions: R
edit_actions: ) prediction performance along with accordingly designed evaluation scenarios remain open challenges . We thus propose Contrastive Label-Embedding Self-Supervision (CLESS) pretraining , which enables pretraining from multiple magnitudes smaller, 'task internal' data only, while still strongly improving fully supervised,
edit_actions: generalization, since
edit_actions: clarity
edit_actions: clarity clarity coherence
edit_actions: R
edit_actions: , few-shot and self-supervised zero-shot learning abilities. Accordingly, we analyse improvements in learning dynamics over baselines on a challenging long-tailed, low-resource, multi-label text classification scenario with noisy, highly sparse labels and many minority concepts
edit_actions: preservation has been linked to algorithmic fairness and because data in the tail is limited by definition. To address these challenges, we propose a data and compute efficient self-supervised, contrastive text encoder, pretrained on 60MB of `task-internal' text data, and compare it to RoBERTa, which was pretrained on 160GB of `task-external' text
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed style
edit_actions: R
edit_actions: that long-tailed zero and few-shot learning markedly benefit from increasing 'dataset-internal' self-supervised pretraining signals, to help reduce the reliance on large external sources
edit_actions: our method outperforms RoBERTa, while pretraining and fine-tuning in a 1/5th of RoBERTa's fine-tuning time
edit_actions: meaning-changed
edit_actions: meaning-changed clarity meaning-changed