pip install deepspeed pip install transformers[deepspeed] Memory requirements Before you begin, it is a good idea to check whether you have enough GPU and CPU memory to fit your model.