TaiMingLu/GenEx-World-Explorer

GenEx-World-Initializer 🚀🌍

GenEx World Explorer is a video generation pipeline built on top of Stable Video Diffusion (SVD). . It takes a keyframe, and generates a temporally consistent video. This explorer version builds on SVD with a custom UNetSpatioTemporalConditionModel.

The diffuser generate a forward moving path of a panoramic input image, to explore a given scene.

📦 Usage

from diffusers import UNetSpatioTemporalConditionModel, StableVideoDiffusionPipeline
import torch

model_id = 'TaiMingLu/GenEx-World-Explorer'

# Load the custom UNet
unet = UNetSpatioTemporalConditionModel.from_pretrained(
    model_id,
    subfolder='unet',
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
)

# Load the full pipeline with custom UNet
pipe = StableVideoDiffusionPipeline.from_pretrained(
    model_id,
    unet=unet,
    low_cpu_mem_usage=True,
    torch_dtype=torch.float16,
    local_files_only=True,
).to('cuda')

🔧 Requirements

diffusers>=0.33.1
transformers
numpy
pillow

✨ BibTex

@misc{lu2025genexgeneratingexplorableworld,
      title={GenEx: Generating an Explorable World}, 
      author={Taiming Lu and Tianmin Shu and Junfei Xiao and Luoxin Ye and Jiahao Wang and Cheng Peng and Chen Wei and Daniel Khashabi and Rama Chellappa and Alan Yuille and Jieneng Chen},
      year={2025},
      eprint={2412.09624},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.09624}, 
}