RAG / knowledge_base /model_doc_distilbert.txt
Ahmadzei's picture
update 1
57bdca5
DistilBERT
Overview
The DistilBERT model was proposed in the blog post Smaller, faster, cheaper, lighter: Introducing DistilBERT, a
distilled version of BERT, and the paper DistilBERT, a
distilled version of BERT: smaller, faster, cheaper and lighter. DistilBERT is a
small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than
google-bert/bert-base-uncased, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language
understanding benchmark.
The abstract from the paper is the following:
As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP),
operating these large models in on-the-edge and/or under constrained computational training or inference budgets
remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation
model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger
counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage
knowledge distillation during the pretraining phase and show that it is possible to reduce the size of a BERT model by
40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive
biases learned by larger models during pretraining, we introduce a triple loss combining language modeling,
distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we
demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device
study.
This model was contributed by victorsanh. This model jax version was
contributed by kamalkraj. The original code can be found here.
Usage tips
DistilBERT doesn't have token_type_ids, you don't need to indicate which token belongs to which segment. Just
separate your segments with the separation token tokenizer.sep_token (or [SEP]).
DistilBERT doesn't have options to select the input positions (position_ids input). This could be added if
necessary though, just let us know if you need this option.
Same as BERT but smaller. Trained by distillation of the pretrained BERT model, meaning it’s been trained to predict the same probabilities as the larger model. The actual objective is a combination of:
finding the same probabilities as the teacher model
predicting the masked tokens correctly (but no next-sentence objective)
a cosine similarity between the hidden states of the student and the teacher model
Resources
A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with DistilBERT. If you're interested in submitting a resource to be included here, please feel free to open a Pull Request and we'll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.
A blog post on Getting Started with Sentiment Analysis using Python with DistilBERT.
A blog post on how to train DistilBERT with Blurr for sequence classification.
A blog post on how to use Ray to tune DistilBERT hyperparameters.
A blog post on how to train DistilBERT with Hugging Face and Amazon SageMaker.
A notebook on how to finetune DistilBERT for multi-label classification. 🌎
A notebook on how to finetune DistilBERT for multiclass classification with PyTorch. 🌎
A notebook on how to finetune DistilBERT for text classification in TensorFlow. 🌎
[DistilBertForSequenceClassification] is supported by this example script and notebook.
[TFDistilBertForSequenceClassification] is supported by this example script and notebook.
[FlaxDistilBertForSequenceClassification] is supported by this example script and notebook.
Text classification task guide
[DistilBertForTokenClassification] is supported by this example script and notebook.
[TFDistilBertForTokenClassification] is supported by this example script and notebook.
[FlaxDistilBertForTokenClassification] is supported by this example script.
Token classification chapter of the 🤗 Hugging Face Course.
Token classification task guide
[DistilBertForMaskedLM] is supported by this example script and notebook.
[TFDistilBertForMaskedLM] is supported by this example script and notebook.
[FlaxDistilBertForMaskedLM] is supported by this example script and notebook.
Masked language modeling chapter of the 🤗 Hugging Face Course.
Masked language modeling task guide
[DistilBertForQuestionAnswering] is supported by this example script and notebook.
[TFDistilBertForQuestionAnswering] is supported by this example script and notebook.
[FlaxDistilBertForQuestionAnswering] is supported by this example script.
Question answering chapter of the 🤗 Hugging Face Course.
Question answering task guide
Multiple choice
- [DistilBertForMultipleChoice] is supported by this example script and notebook.
- [TFDistilBertForMultipleChoice] is supported by this example script and notebook.
- Multiple choice task guide
⚗️ Optimization
A blog post on how to quantize DistilBERT with 🤗 Optimum and Intel.
A blog post on how Optimizing Transformers for GPUs with 🤗 Optimum.
A blog post on Optimizing Transformers with Hugging Face Optimum.
⚡️ Inference
A blog post on how to Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia with DistilBERT.
A blog post on Serverless Inference with Hugging Face's Transformers, DistilBERT and Amazon SageMaker.
🚀 Deploy
A blog post on how to deploy DistilBERT on Google Cloud.
A blog post on how to deploy DistilBERT with Amazon SageMaker.
A blog post on how to Deploy BERT with Hugging Face Transformers, Amazon SageMaker and Terraform module.
Combining DistilBERT and Flash Attention 2
First, make sure to install the latest version of Flash Attention 2 to include the sliding window attention feature.
pip install -U flash-attn --no-build-isolation
Make also sure that you have a hardware that is compatible with Flash-Attention 2. Read more about it in the official documentation of flash-attn repository. Make also sure to load your model in half-precision (e.g. torch.float16)
To load and run a model using Flash Attention 2, refer to the snippet below:
thon
import torch
from transformers import AutoTokenizer, AutoModel
device = "cuda" # the device to load the model onto
tokenizer = AutoTokenizer.from_pretrained('distilbert/distilbert-base-uncased')
model = AutoModel.from_pretrained("distilbert/distilbert-base-uncased", torch_dtype=torch.float16, attn_implementation="flash_attention_2")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt').to(device)
model.to(device)
output = model(**encoded_input)
DistilBertConfig
[[autodoc]] DistilBertConfig
DistilBertTokenizer
[[autodoc]] DistilBertTokenizer
DistilBertTokenizerFast
[[autodoc]] DistilBertTokenizerFast
DistilBertModel
[[autodoc]] DistilBertModel
- forward
DistilBertForMaskedLM
[[autodoc]] DistilBertForMaskedLM
- forward
DistilBertForSequenceClassification
[[autodoc]] DistilBertForSequenceClassification
- forward
DistilBertForMultipleChoice
[[autodoc]] DistilBertForMultipleChoice
- forward
DistilBertForTokenClassification
[[autodoc]] DistilBertForTokenClassification
- forward
DistilBertForQuestionAnswering
[[autodoc]] DistilBertForQuestionAnswering
- forward
TFDistilBertModel
[[autodoc]] TFDistilBertModel
- call
TFDistilBertForMaskedLM
[[autodoc]] TFDistilBertForMaskedLM
- call
TFDistilBertForSequenceClassification
[[autodoc]] TFDistilBertForSequenceClassification
- call
TFDistilBertForMultipleChoice
[[autodoc]] TFDistilBertForMultipleChoice
- call
TFDistilBertForTokenClassification
[[autodoc]] TFDistilBertForTokenClassification
- call
TFDistilBertForQuestionAnswering
[[autodoc]] TFDistilBertForQuestionAnswering
- call
FlaxDistilBertModel
[[autodoc]] FlaxDistilBertModel
- call
FlaxDistilBertForMaskedLM
[[autodoc]] FlaxDistilBertForMaskedLM
- call
FlaxDistilBertForSequenceClassification
[[autodoc]] FlaxDistilBertForSequenceClassification
- call
FlaxDistilBertForMultipleChoice
[[autodoc]] FlaxDistilBertForMultipleChoice
- call
FlaxDistilBertForTokenClassification
[[autodoc]] FlaxDistilBertForTokenClassification
- call
FlaxDistilBertForQuestionAnswering
[[autodoc]] FlaxDistilBertForQuestionAnswering
- call