File size: 338 Bytes
5fa1a76 |
1 2 3 4 5 6 7 8 9 |
Here's an example using the BERT tokenizer, which is a WordPiece tokenizer: thon from transformers import BertTokenizer tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-cased") sequence = "A Titan RTX has 24GB of VRAM" The tokenizer takes care of splitting the sequence into tokens available in the tokenizer vocabulary. |