ZeRO works in several stages:

ZeRO-1, optimizer state partioning across GPUs
ZeRO-2, gradient partitioning across GPUs
ZeRO-3, parameteter partitioning across GPUs

In GPU-limited environments, ZeRO also enables offloading optimizer memory and computation from the GPU to the CPU to fit and train really large models on a single GPU.