arxiv:2504.10188

Efficient Generative Model Training via Embedded Representation Warmup

Published on Apr 14

· Submitted by

SempraETY on Apr 16

Upvote

Authors:

Deyuan Liu ,

Abstract

Diffusion models excel at generating high-dimensional data but fall short in training efficiency and representation quality compared to self-supervised methods. We identify a key bottleneck: the underutilization of high-quality, semantically rich representations during training notably slows down convergence. Our systematic analysis reveals a critical representation processing region -- primarily in the early layers -- where semantic and structural pattern learning takes place before generation can occur. To address this, we propose Embedded Representation Warmup (ERW), a plug-and-play framework where in the first stage we get the ERW module serves as a warmup that initializes the early layers of the diffusion model with high-quality, pretrained representations. This warmup minimizes the burden of learning representations from scratch, thereby accelerating convergence and boosting performance. Our theoretical analysis demonstrates that ERW's efficacy depends on its precise integration into specific neural network layers -- termed the representation processing region -- where the model primarily processes and transforms feature representations for later generation. We further establish that ERW not only accelerates training convergence but also enhances representation quality: empirically, our method achieves a 40times acceleration in training speed compared to REPA, the current state-of-the-art methods. Code is available at https://github.com/LINs-lab/ERW.

View arXiv page View PDF Project page GitHub repository Add to collection

Community

SempraETY

Paper author Paper submitter 6 days ago

•

edited 6 days ago

Efficient Generative Model Training via Embedded Representation Warmup

Deyuan Liu^1* · Peng Sun^1,2* · Xufeng Li^1,3 · Tao Lin^1†
¹ Westlake University ² Zhejiang University ³ Nanjing University
* These authors contributed equally. † Corresponding author.

[arXiv] [Project Page]

Summary:
Diffusion models have made impressive progress in generating high-fidelity images. However, training them from scratch requires learning both robust semantic representations and the generative process simultaneously. Our work introduces Embedded Representation Warmup (ERW) – a plug-and-play two-phase training framework that:

Phase 1 – Warmup: Initializes the early layers of the diffusion model with high-quality, pretrained visual representations (e.g., from DINOv2 or other self-supervised encoders).
Phase 2 – Full Training: Continues with standard diffusion training while gradually reducing the alignment loss, so the model can focus on refining generation.

🔥 News

(🔥 New) [2025/4/15] 🔥ERW code & weights are released! 🎉 Include: Training & Inference code and Weights in HF are all released.

Acknowledgement

This code is mainly built upon REPA, LightningDiT, DiT, SiT, edm2, and RCG repositories.

BibTeX



@misc
	{liu2025efficientgenerativemodeltraining,
      title={Efficient Generative Model Training via Embedded Representation Warmup}, 
      author={Deyuan Liu and Peng Sun and Xufeng Li and Tao Lin},
      year={2025},
      eprint={2504.10188},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2504.10188}, 
}