|
|
|
FlauBERT |
|
|
|
Overview |
|
The FlauBERT model was proposed in the paper FlauBERT: Unsupervised Language Model Pre-training for French by Hang Le et al. It's a transformer model pretrained using a masked language |
|
modeling (MLM) objective (like BERT). |
|
The abstract from the paper is the following: |
|
Language models have become a key step to achieve state-of-the art results in many different Natural Language |
|
Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way |
|
to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their |
|
contextualization at the sentence level. This has been widely demonstrated for English using contextualized |
|
representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., |
|
2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and |
|
heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for |
|
Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text |
|
classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the |
|
time they outperform other pretraining approaches. Different versions of FlauBERT as well as a unified evaluation |
|
protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research |
|
community for further reproducible experiments in French NLP. |
|
This model was contributed by formiel. The original code can be found here. |
|
Tips: |
|
- Like RoBERTa, without the sentence ordering prediction (so just trained on the MLM objective). |
|
Resources |
|
|
|
Text classification task guide |
|
Token classification task guide |
|
Question answering task guide |
|
Masked language modeling task guide |
|
Multiple choice task guide |
|
|
|
FlaubertConfig |
|
[[autodoc]] FlaubertConfig |
|
FlaubertTokenizer |
|
[[autodoc]] FlaubertTokenizer |
|
|
|
FlaubertModel |
|
[[autodoc]] FlaubertModel |
|
- forward |
|
FlaubertWithLMHeadModel |
|
[[autodoc]] FlaubertWithLMHeadModel |
|
- forward |
|
FlaubertForSequenceClassification |
|
[[autodoc]] FlaubertForSequenceClassification |
|
- forward |
|
FlaubertForMultipleChoice |
|
[[autodoc]] FlaubertForMultipleChoice |
|
- forward |
|
FlaubertForTokenClassification |
|
[[autodoc]] FlaubertForTokenClassification |
|
- forward |
|
FlaubertForQuestionAnsweringSimple |
|
[[autodoc]] FlaubertForQuestionAnsweringSimple |
|
- forward |
|
FlaubertForQuestionAnswering |
|
[[autodoc]] FlaubertForQuestionAnswering |
|
- forward |
|
|
|
TFFlaubertModel |
|
[[autodoc]] TFFlaubertModel |
|
- call |
|
TFFlaubertWithLMHeadModel |
|
[[autodoc]] TFFlaubertWithLMHeadModel |
|
- call |
|
TFFlaubertForSequenceClassification |
|
[[autodoc]] TFFlaubertForSequenceClassification |
|
- call |
|
TFFlaubertForMultipleChoice |
|
[[autodoc]] TFFlaubertForMultipleChoice |
|
- call |
|
TFFlaubertForTokenClassification |
|
[[autodoc]] TFFlaubertForTokenClassification |
|
- call |
|
TFFlaubertForQuestionAnsweringSimple |
|
[[autodoc]] TFFlaubertForQuestionAnsweringSimple |
|
- call |
|
|
|
|