ZeRO works in several stages: ZeRO-1, optimizer state partioning across GPUs ZeRO-2, gradient partitioning across GPUs ZeRO-3, parameteter partitioning across GPUs In GPU-limited environments, ZeRO also enables offloading optimizer memory and computation from the GPU to the CPU to fit and train really large models on a single GPU.