Load DistilGPT2 with [AutoModelForCausalLM]:

from transformers import AutoModelForCausalLM, TrainingArguments, Trainer
model = AutoModelForCausalLM.from_pretrained("distilbert/distilgpt2")

At this point, only three steps remain:

Define your training hyperparameters in [TrainingArguments].