Preprocess

The next step is to load a DistilBERT tokenizer to preprocess the tokens field:

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")

As you saw in the example tokens field above, it looks like the input has already been tokenized.