| FlauBERT | |
| Overview | |
| The FlauBERT model was proposed in the paper FlauBERT: Unsupervised Language Model Pre-training for French by Hang Le et al. It's a transformer model pretrained using a masked language | |
| modeling (MLM) objective (like BERT). | |
| The abstract from the paper is the following: | |
| Language models have become a key step to achieve state-of-the art results in many different Natural Language | |
| Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way | |
| to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their | |
| contextualization at the sentence level. This has been widely demonstrated for English using contextualized | |
| representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., | |
| 2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and | |
| heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for | |
| Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text | |
| classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the | |
| time they outperform other pretraining approaches. Different versions of FlauBERT as well as a unified evaluation | |
| protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research | |
| community for further reproducible experiments in French NLP. | |
| This model was contributed by formiel. The original code can be found here. | |
| Tips: | |
| - Like RoBERTa, without the sentence ordering prediction (so just trained on the MLM objective). | |
| Resources | |
| Text classification task guide | |
| Token classification task guide | |
| Question answering task guide | |
| Masked language modeling task guide | |
| Multiple choice task guide | |
| FlaubertConfig | |
| [[autodoc]] FlaubertConfig | |
| FlaubertTokenizer | |
| [[autodoc]] FlaubertTokenizer | |
| FlaubertModel | |
| [[autodoc]] FlaubertModel | |
| - forward | |
| FlaubertWithLMHeadModel | |
| [[autodoc]] FlaubertWithLMHeadModel | |
| - forward | |
| FlaubertForSequenceClassification | |
| [[autodoc]] FlaubertForSequenceClassification | |
| - forward | |
| FlaubertForMultipleChoice | |
| [[autodoc]] FlaubertForMultipleChoice | |
| - forward | |
| FlaubertForTokenClassification | |
| [[autodoc]] FlaubertForTokenClassification | |
| - forward | |
| FlaubertForQuestionAnsweringSimple | |
| [[autodoc]] FlaubertForQuestionAnsweringSimple | |
| - forward | |
| FlaubertForQuestionAnswering | |
| [[autodoc]] FlaubertForQuestionAnswering | |
| - forward | |
| TFFlaubertModel | |
| [[autodoc]] TFFlaubertModel | |
| - call | |
| TFFlaubertWithLMHeadModel | |
| [[autodoc]] TFFlaubertWithLMHeadModel | |
| - call | |
| TFFlaubertForSequenceClassification | |
| [[autodoc]] TFFlaubertForSequenceClassification | |
| - call | |
| TFFlaubertForMultipleChoice | |
| [[autodoc]] TFFlaubertForMultipleChoice | |
| - call | |
| TFFlaubertForTokenClassification | |
| [[autodoc]] TFFlaubertForTokenClassification | |
| - call | |
| TFFlaubertForQuestionAnsweringSimple | |
| [[autodoc]] TFFlaubertForQuestionAnsweringSimple | |
| - call | |