VideoWorld: Mastering Complex Tasks Through Vision Alone, Without Language Models 193

On February 10, 2025, the video generation model "VideoWorld" was jointly developed and open-sourced by the Doubao large model team in collaboration with Beijing Jiaotong University and the University of Science and Technology of China. The core innovation of the VideoWorld model lies in the fact that it does not rely on language models at all, and can learn and generate complex tasks solely through visual information.

I. Core Features

1. Pure Visual Knowledge Learning

• Autonomously masters complex rules and strategies through unlabeled video data (such as Go move logic and robot operation sequences), without requiring language instructions or reinforcement learning reward mechanisms

• Supports multi-step reasoning and long-term planning capabilities, such as predicting the opponent's next 5-9 moves in a Go game

2. Efficient Visual Representation Compression

• Introduces a Latent Dynamics Model (LDM) to encode inter-frame video variations into compact latent sequences, reducing redundant information interference

• Achieves joint modeling of long-term and short-term dynamic dependencies—focusing on fine-grained displacement in the short term (such as robotic arm fine-tuning) while capturing task-level patterns in the long term (such as Go tactical combinations)

3. Cross-Scenario Generalization Capability

Achieves performance close to the Oracle model on the CALVIN and RLBench robot test sets, and can adapt to different environmental parameters (such as lighting and object position)

Reaches a professional 5-dan level in Go play (Video-GoBench benchmark), surpassing traditional reinforcement learning agents.

II. VideoWorld Technical Architecture:

Base Architecture

• Based on VQ-VAE to discretize video frames into token sequences, using an autoregressive Transformer to achieve frame-level prediction

• Training data contains only videos of task execution processes (such as Go game evolution and robotic arm operation recordings)

Latent Dynamics Model (LDM)

• Encoding mechanism: compresses the visual changes of the next H frames into continuous latent codes, which are used as prediction targets after quantization

• Dual-stream prediction: jointly optimizes video frame generation and latent code prediction, enhancing the modeling of key decision nodes

• Interpretable analysis: UMAP visualization shows that latent space clustering is strongly correlated with task dynamics.

Efficiency Optimization Design

• Model parameter size is controlled at the 300-million level, reducing computational overhead by 80% compared to traditional video models

• Training data scale: Video-GoBench contains hundreds of thousands of Go game video frames

III. VideoWorld Application Scenarios

• Content creation: provides powerful support for short video production, movie scene conceptualization, and visual content generation in games.

• Education: helps students understand complex knowledge more vividly by generating dynamic videos.

• Advertising: generates personalized ad videos based on user preferences to improve marketing effectiveness.

IV. VideoWorld Download and Installation:

1. Clone the repository:

git clone https://github.com/bytedance/VideoWorld.git
cd VideoWorld

2. Install dependencies:

pip install -r requirements.txt

3. Run examples:Example code is typically located in the examples or scripts directory. Adjust the configuration file as needed and run the corresponding script.

V. VideoWorld Project Page:

Project homepage: https://maverickren.github.io/VideoWorld.github.io

GitHub address: https://github.com/bytedance/VideoWorld

Paper link: https://arxiv.org/abs/2501.09781