For inference, Transformers support ZeRO-3 and offloading since it allows loading huge models.