The pretraining includes both supervised and self-supervised training.