Preprocess

The next step is to load a DistilGPT2 tokenizer to process the text subfield:

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("distilbert/distilgpt2")

You'll notice from the example above, the text field is actually nested inside answers.