| XLNet | |
| Overview | |
| The XLNet model was proposed in XLNet: Generalized Autoregressive Pretraining for Language Understanding by Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, | |
| Quoc V. Le. XLnet is an extension of the Transformer-XL model pre-trained using an autoregressive method to learn | |
| bidirectional contexts by maximizing the expected likelihood over all permutations of the input sequence factorization | |
| order. | |
| The abstract from the paper is the following: | |
| With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves | |
| better performance than pretraining approaches based on autoregressive language modeling. However, relying on | |
| corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a | |
| pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive | |
| pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all | |
| permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive | |
| formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into | |
| pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large | |
| margin, including question answering, natural language inference, sentiment analysis, and document ranking. | |
| This model was contributed by thomwolf. The original code can be found here. | |
| Usage tips | |
| The specific attention pattern can be controlled at training and test time using the perm_mask input. | |
| Due to the difficulty of training a fully auto-regressive model over various factorization order, XLNet is pretrained | |
| using only a sub-set of the output tokens as target which are selected with the target_mapping input. | |
| To use XLNet for sequential decoding (i.e. not in fully bi-directional setting), use the perm_mask and | |
| target_mapping inputs to control the attention span and outputs (see examples in | |
| examples/pytorch/text-generation/run_generation.py) | |
| XLNet is one of the few models that has no sequence length limit. | |
| XLNet is not a traditional autoregressive model but uses a training strategy that builds on that. It permutes the tokens in the sentence, then allows the model to use the last n tokens to predict the token n+1. Since this is all done with a mask, the sentence is actually fed in the model in the right order, but instead of masking the first n tokens for n+1, XLNet uses a mask that hides the previous tokens in some given permutation of 1,…,sequence length. | |
| XLNet also uses the same recurrence mechanism as Transformer-XL to build long-term dependencies. | |
| Resources | |
| Text classification task guide | |
| Token classification task guide | |
| Question answering task guide | |
| Causal language modeling task guide | |
| Multiple choice task guide | |
| XLNetConfig | |
| [[autodoc]] XLNetConfig | |
| XLNetTokenizer | |
| [[autodoc]] XLNetTokenizer | |
| - build_inputs_with_special_tokens | |
| - get_special_tokens_mask | |
| - create_token_type_ids_from_sequences | |
| - save_vocabulary | |
| XLNetTokenizerFast | |
| [[autodoc]] XLNetTokenizerFast | |
| XLNet specific outputs | |
| [[autodoc]] models.xlnet.modeling_xlnet.XLNetModelOutput | |
| [[autodoc]] models.xlnet.modeling_xlnet.XLNetLMHeadModelOutput | |
| [[autodoc]] models.xlnet.modeling_xlnet.XLNetForSequenceClassificationOutput | |
| [[autodoc]] models.xlnet.modeling_xlnet.XLNetForMultipleChoiceOutput | |
| [[autodoc]] models.xlnet.modeling_xlnet.XLNetForTokenClassificationOutput | |
| [[autodoc]] models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringSimpleOutput | |
| [[autodoc]] models.xlnet.modeling_xlnet.XLNetForQuestionAnsweringOutput | |
| [[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetModelOutput | |
| [[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetLMHeadModelOutput | |
| [[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForSequenceClassificationOutput | |
| [[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForMultipleChoiceOutput | |
| [[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForTokenClassificationOutput | |
| [[autodoc]] models.xlnet.modeling_tf_xlnet.TFXLNetForQuestionAnsweringSimpleOutput | |
| XLNetModel | |
| [[autodoc]] XLNetModel | |
| - forward | |
| XLNetLMHeadModel | |
| [[autodoc]] XLNetLMHeadModel | |
| - forward | |
| XLNetForSequenceClassification | |
| [[autodoc]] XLNetForSequenceClassification | |
| - forward | |
| XLNetForMultipleChoice | |
| [[autodoc]] XLNetForMultipleChoice | |
| - forward | |
| XLNetForTokenClassification | |
| [[autodoc]] XLNetForTokenClassification | |
| - forward | |
| XLNetForQuestionAnsweringSimple | |
| [[autodoc]] XLNetForQuestionAnsweringSimple | |
| - forward | |
| XLNetForQuestionAnswering | |
| [[autodoc]] XLNetForQuestionAnswering | |
| - forward | |
| TFXLNetModel | |
| [[autodoc]] TFXLNetModel | |
| - call | |
| TFXLNetLMHeadModel | |
| [[autodoc]] TFXLNetLMHeadModel | |
| - call | |
| TFXLNetForSequenceClassification | |
| [[autodoc]] TFXLNetForSequenceClassification | |
| - call | |
| TFLNetForMultipleChoice | |
| [[autodoc]] TFXLNetForMultipleChoice | |
| - call | |
| TFXLNetForTokenClassification | |
| [[autodoc]] TFXLNetForTokenClassification | |
| - call | |
| TFXLNetForQuestionAnsweringSimple | |
| [[autodoc]] TFXLNetForQuestionAnsweringSimple | |
| - call | |