| deepspeed --num_gpus=1 examples/pytorch/translation/run_translation.py \ | |
| --deepspeed tests/deepspeed/ds_config_zero2.json \ | |
| --model_name_or_path google-t5/t5-small --per_device_train_batch_size 1 \ | |
| --output_dir output_dir --overwrite_output_dir --fp16 \ | |
| --do_train --max_train_samples 500 --num_train_epochs 1 \ | |
| --dataset_name wmt16 --dataset_config "ro-en" \ | |
| --source_lang en --target_lang ro | |
| DeepSpeed is still useful with just 1 GPU because you can: | |
| Offload some computations and memory to the CPU to make more GPU resources available to your model to use a larger batch size or fit a very large model that normally won't fit. |