Note that the encoder of MAE is only used to encode the visual patches.