Open-Sora 2.0: An Open-Source SOTA Video Generation Model by Colossal-AI 211

What is Open-Sora 2.0?

Open-Sora 2.0 is a brand-new open-source SOTA video generation AI model launched by HPC-AI Tech. It successfully trained a commercial-grade 11B parameter video generation large model with only $200,000 (224 GPUs), achieving quality close to SOTA (state-of-the-art), with performance directly comparable to HunyuanVideo and Step-Video with 30B parameters from StepFun, and approaching OpenAI Sora! Open-Sora 2.0 features efficient training and inference optimization, and can quickly generate smooth videos at 720p resolution, making it suitable for creative video production, social media content generation, and other fields.

Open-Sora 2.0 Features

High-Quality Video Generation: Supports 720p high resolution and 24FPS smooth video generation, capable of producing videos with rich details and natural scene transitions.

Controllable Motion Magnitude: The motion magnitude can be set as needed to better showcase the subtle movements of people or scenes.

High Performance and Low Cost:

• Open-Sora 2.0 adopts an 11B parameter scale, with performance comparable to or even surpassing mainstream closed-source large models (such as HunyuanVideo and StepFun's 30B parameter model), delivering outstanding results in VBench and human preference tests.

• Training cost is significantly reduced, requiring only $200,000 (224 GPUs) to complete training—a 5-10x cost reduction compared to traditional high-performance video generation models. Comparable to OpenAI Sora.

Multiple Generation Methods:

• Text-to-Video: Supports generating corresponding video content directly from text descriptions.

• Image-to-Video: Combines open-source image models to generate videos based on images.

Efficient Training and Optimization:

• Adopts a multi-stage, multi-level data filtering mechanism to ensure high-quality data input and improve training efficiency.

• Prioritizes training at low resolution to learn key dynamic features, then progressively increases resolution, significantly reducing computational overhead.

• Prioritizes training image-to-video tasks, using image features to guide video generation and accelerate model convergence.

• Combines ColossalAI and system-level optimizations to improve computing resource utilization.

• Introduces a high compression-ratio video autoencoder, reducing inference time to within 3 minutes on a single GPU, with inference speed improved by 10x.

Powerful Model Architecture:

• Based on 3D autoencoders, 3D full attention mechanism, and MMDiT architecture, significantly improving training efficiency and inference speed.

• Through a multi-bucket training mechanism, enables simultaneous training on videos of different lengths and resolutions.

Open-Sora 2.0 Technical Breakthroughs

• Model Architecture: Adopts a 3D full attention mechanism and MMDiT architecture to enhance spatiotemporal feature modeling capabilities; extends the diffusion Transformer (DiT) by introducing temporal attention layers to enable dynamic generation of video sequences.

• Efficient Training Strategy: Ensures high-quality training data input through a multi-level filtering mechanism, prioritizes learning motion features, and then improves image quality through image-to-video (T2I2V), saving 40x computational resources.

• Inference Acceleration Revolution: Adopts a high compression-ratio autoencoder (4×32×32 compression ratio), reducing 768px video generation time from 30 minutes to 3 minutes—a 10x speed improvement.

Open-Sora 2.0 Use Cases

• Creative Video Production: Can be used to create sci-fi short films, animated shorts, etc., meeting the needs of creative video production and content generation.

• Social Media Content Generation: Capable of generating high-quality video content suitable for social media sharing.

• Game Video Production: Provides convenience for the production of game videos.

GitHub Open-Source Repository:https://github.com/hpcaitech/Open-Sora

Technical Report:https://github.com/hpcaitech/Open-Sora-Demo/blob/main/paper/Open_Sora_2_tech_report.pdf