ZeRO-1 shards the optimizer states across GPUs, and you can expect a tiny speed up.