before_revision: For natural language processing (NLP) taskssuch as sentiment or topic classification, currently prevailing approaches heavily rely on pretraining large self-supervised models on massive external data resources. However, this methodology is being critiqued for: exceptional compute and pretraining data requirements ; diminishing returns on both large and small datasets; and importantly, favourable evaluation settings that overestimate performance differences. The core belief behind current methodology, coined `the bitter lesson' by R. Sutton, is that `compute scale-up beats data and compute-efficient algorithms', neglecting that progress in compute hardware scale-up is based almost entirely on the miniaturisation of resource consumption. We thus approach pretrainingfrom a miniaturisation perspective, such as not to require massive external data sources and models, or learned translations from continuous input embeddings to discrete labels. To minimise overly favourable evaluation, we examine learning on a long-tailed, low-resource, multi-label text classification dataset with noisy, highly sparse labels and many rare concepts. To this end, we propose a novel `dataset-internal' contrastive autoencoding approach to self-supervised pretraining and demonstrate marked improvements in zero-shot, few-shot and solely supervised learning performance; even under an unfavorable low-resource scenario, and without defaulting to large-scale external datasets for self-supervision. We also find empirical evidence that zero and few-shot learning markedly benefit from adding more ` dataset-internal' , self-supervised training signals, which is of practical importance when retrieving or computing on large external sources of such signals is infeasible .
after_revision: For natural language processing 'text-to-text' tasks, the prevailing approaches heavily rely on pretraining large self-supervised models on massive external data sources, which incurs exceptional pretraining data requirements and a diminished ability to pretrain over small datasets. However, fundamental pretraining method capabilities like few to zero-shot learning or preserving minority concept (long-tail) prediction performance along with accordingly designed evaluation scenarios remain open challenges. We thus propose Contrastive Label-Embedding Self-Supervision (CLESS) pretraining, which enables pretraining from multiple magnitudes smaller, 'task internal' data only, while still strongly improving fully supervised, long-tail, few-shot and self-supervised zero-shot learning abilities. Accordingly, we analyse improvements in learning dynamics over baselines on a challenging long-tailed, low-resource, multi-label text classification scenario with noisy, highly sparse labels and many minority concepts. We find that long-tailed zero and few-shot learning markedly benefit from increasing ' dataset-internal' self-supervised pretraining signals, to help reduce the reliance on large external sources .
edit_actions: R
edit_actions: (NLP) taskssuch as sentiment or topic classification, currently
edit_actions: 'text-to-text' tasks, the
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: R
edit_actions: resources. However, this methodology is being critiqued for: exceptional compute and
edit_actions: sources, which incurs exceptional
edit_actions: clarity
edit_actions: clarity clarity coherence
edit_actions: R
edit_actions: ; diminishing returns on both large and small datasets; and importantly, favourable evaluation settings that overestimate performance differences. The core belief behind current methodology, coined `the bitter lesson' by R. Sutton, is that `compute scale-up beats data and compute-efficient algorithms', neglecting that progress in compute hardware scale-up is based almost entirely on the miniaturisation of resource consumption. We thus approach pretrainingfrom a miniaturisation perspective, such as not to require massive external data sources and models, or learned translations from continuous input embeddings to discrete labels. To minimise overly favourable evaluation, we examine learning on a
edit_actions: and a diminished ability to pretrain over small datasets. However, fundamental pretraining method capabilities like few to zero-shot learning or preserving minority concept (long-tail) prediction performance along with accordingly designed evaluation scenarios remain open challenges. We thus propose Contrastive Label-Embedding Self-Supervision (CLESS) pretraining, which enables pretraining from multiple magnitudes smaller, 'task internal' data only, while still strongly improving fully supervised, long-tail, few-shot and self-supervised zero-shot learning abilities. Accordingly, we analyse improvements in learning dynamics over baselines on a challenging
edit_actions: meaning-changed
edit_actions: meaning-changed meaning-changed clarity
edit_actions: R
edit_actions: dataset
edit_actions: scenario
edit_actions: clarity
edit_actions: style clarity clarity
edit_actions: R
edit_actions: rare concepts. To this end, we propose a novel `dataset-internal' contrastive autoencoding approach to self-supervised pretraining and demonstrate marked improvements in zero-shot, few-shot and solely supervised learning performance; even under an unfavorable low-resource scenario, and without defaulting to large-scale external datasets for self-supervision. We also find empirical evidence that
edit_actions: minority concepts. We find that long-tailed
edit_actions: coherence
edit_actions: coherence coherence clarity
edit_actions: R
edit_actions: adding more `
edit_actions: increasing '
edit_actions: clarity
edit_actions: style clarity clarity
edit_actions: D
edit_actions: ,
edit_actions: fluency
edit_actions: fluency fluency fluency
edit_actions: R
edit_actions: training signals, which is of practical importance when retrieving or computing
edit_actions: pretraining signals, to help reduce the reliance
edit_actions: clarity
edit_actions: clarity clarity clarity
edit_actions: D
edit_actions: of such signals is infeasible
edit_actions: clarity
edit_actions: clarity clarity coherence