Matrix-Game: Kunlun Wanwei Open-Sources 17B+ Interactive World Foundation Model 80

小编 · 2025-05-14

Matrix-Game is a 17B+ interactive world foundation model officially open-sourced by Kunlun Wanwei on May 13, 2025, and the industry's first open-source 10B+ spatial intelligence large model.

Matrix-Game: Kunlun Wanwei Open-Sources 17B+ Interactive World Foundation Model

What is Matrix-Game?

Matrix-Game is a 17B+ interactive world foundation model officially open-sourced by Kunlun Wanwei on May 13, 2025, and is also the first open-source 10B+ spatial intelligence large model in the industry.

screenshot (8).webp

Core Features of Matrix-Game

Interactive World Generation: Matrix-Game is an interactive world foundation model designed for game world modeling. Users can freely explore, manipulate, and even create virtual worlds with rich details and reasonable physics rules through simple keyboard commands, mouse movements, and other operations.
High-Precision Control: In terms of control performance, Matrix-Game can achieve an accuracy of over 90% for actions such as "movement" and "attack", and maintains high-precision response even under fine-grained viewpoint control.
Multi-Scenario Generalization Capability: Matrix-Game outperforms across 8 typical Minecraft scenarios, demonstrating exceptional environmental adaptability and generalization capability, with the potential to generalize to non-Minecraft game environments.
Autoregressive Long Video Generation: Supports autoregressive long video generation, achieving seamless transitions between actions and viewpoints, ensuring temporal consistency and environmental adaptability.
Systematic Evaluation: Proposes a unified GameWorldScore standard to comprehensively quantify model performance across four dimensions: visual quality, temporal quality, action controllability, and physics rule understanding of videos.

Core Components

Matrix-Game-MC Dataset: A self-built large-scale interactive world dataset containing large-scale unlabeled Minecraft game videos and Minecraft and Unreal controllable video data with keyboard and mouse control signals, featuring fine-grained action annotations.
Matrix-Game Main Model: Built on advanced diffusion model technology, it constructs an image-to-world generation framework capable of generating coherent and controllable interactive videos based on user input.
GameWorldScore Evaluation System: Comprehensively quantifies model performance from four dimensions—visual quality, temporal quality, action controllability, and physics rule understanding of videos—filling the gap of lacking systematic evaluation benchmarks in this field.

Technical Principles of Matrix-Game

Two-Stage Training Strategy: The first stage uses large-scale unlabeled Minecraft game video data for pre-training, allowing the model to learn the basic characteristics and dynamic rules of the environment; the second stage uses Minecraft and Unreal controllable video data with keyboard and mouse control signals for fine-grained controllable training, enabling the model to generate corresponding interactive videos based on user input.
Image-to-World Modeling: Uses a single reference image as the starting point for generating interactive videos, without relying on language prompts, modeling spatial geometry, object motion, and their physical interactions based on visual signals.
Autoregressive Video Generation: Supports extending the generation length in an autoregressive manner, using the last few frames of each previous video segment as motion context to generate segment by segment, ensuring temporal coherence. Random perturbation, random deletion, and Classifier-free guidance strategies are introduced during training to mitigate temporal drift and error accumulation.
Controllable Interaction Design: Keyboard actions are represented by discrete tokens, and viewpoint movement actions are represented by continuous tokens. Based on the GameFactory control module, a multimodal Diffusion Transformer architecture is integrated, and the Classifier-free guidance strategy is used to enhance robust response capability to control signals.

Matrix-Game技术原理.jpg

Application Scenarios of Matrix-Game

Rapid Virtual Game World Construction: Through the model's controllable generation capability, diverse and structurally reasonable game maps and interactive environments can be created at low cost and high efficiency, enhancing the freedom of level design and task construction.
Film & Television and Metaverse Content Production: Supports high-fidelity, physically consistent dynamic scene synthesis, providing a universal world modeling foundation for immersive experience development and creative content generation.
Embodied Intelligent Agent Training and Data Generation: Although the current model is not directly used for embodied intelligence, Matrix-Game has the capability to generate large-scale interactive videos and holds the potential to extend to embodied intelligent agent training and evaluation. It can create diverse and complex virtual environments for agents, helping improve their task execution and reasoning capabilities.

Matrix-Game Model Advantages

Fine-Grained User Interaction Control: Supports a variety of detailed operations and can accurately and naturally respond to user input.
High-Fidelity Visual and Physical Consistency: The generated results are visually coherent and follow physical laws.
Multi-Scenario Generalization Capability: Has generalization ability across multiple Minecraft game scenarios and the potential to generalize to non-Minecraft game environments.
Systematic Evaluation System: The GameWorld Score standard provides strong support for objective evaluation and continuous optimization of the model.