Goku: ByteDance's Multimodal AI Video Generation Model 171

Goku is a multimodal video generation model jointly developed by the University of Hong Kong and ByteDance. It is based on a Rectified Flow Transformer architecture and is specifically optimized for joint image and video generation. The model officially released its technical report in February 2025, with its core goal of revolutionizing digital content creation workflows through low-cost, high-efficiency generative capabilities, demonstrating disruptive potential particularly in advertising, marketing, and e-commerce display fields.

Goku Features:

Multimodal Generation Capabilities

• Text-to-Video: Directly generates dynamic videos through natural language descriptions, supporting complex scenes and character interactions—for example, generating a complete short film featuring character actions and product close-ups based on the prompt "summer beach beverage advertisement."

• Image-to-Video: Takes a single product image or scene image as input to generate dynamic showcase videos—for instance, transforming a static clothing image into a coherent sequence of a model trying on the outfit.

• Text-to-Image: Generates high-resolution images with support for fine-grained control such as adjusting lighting and texture parameters.

Technical Architecture Innovations

• Rectified Flow Framework: Goku employs a Rectified Flow Transformer, combined with an image-video joint VAE (Variational Autoencoder) and a full-attention Transformer. This enables the model to handle image and video generation tasks in a unified manner, reducing the iterative steps of traditional diffusion models by optimizing the continuity of the generation path, thereby improving generation efficiency.

• Multi-stage Training Strategy: Through image-text semantic alignment pre-training, joint image-video training, and modality-specific fine-tuning, the model's capabilities are progressively enhanced in stages—beginning with a focus on learning fundamental features, followed by strengthening temporal coherence and detail precision.

• Large-scale Dataset Support: The researchers constructed a large-scale dataset of approximately 36 million videos and 160 million images, and employed various data filtering and augmentation techniques to enhance data quality.

Cost and Performance Advantages

• Goku+ Commercial Version: As a foundational model dedicated to video advertising, its generation cost is only 1/100 of traditional production methods, and it supports personalized customization (such as brand logo embedding, multi-language dubbing, etc.).

• SOTA Performance: In benchmarks such as MSR-VTT and UCF-101, it set new industry records in text-to-video generation tasks, leading comparable models by more than 20% in video coherence and semantic alignment.

How to Use Goku:

Model Selection and Deployment

• Three model scales are offered: a lightweight version (for mobile adaptation), a standard version (for general scenarios), and an enterprise version (supporting 4K resolution and long video generation).

• Supports cloud-based API calls and on-premise deployment. Enterprise users can access it through the ByteDance Volcengine platform.

Input and Parameter Configuration

• Input Format: Text prompts should follow a structured description (such as "subject-action-scene-style"); image input supports PNG/JPG and transparent background formats.

• Control Parameters: Includes video duration (default 5 seconds, extendable up to 60 seconds), frame rate (24/30/60fps), resolution (up to 4096×2160), and stylized filters (such as "realistic style" or "cartoon rendering").

Interactive Editing

• Supports frame-by-frame fine-tuning after generation, such as modifying local actions, replacing background elements, or appending effects via text commands (e.g., "add raindrop effect").

Goku Product Applications:

E-commerce and Advertising Marketing

• Dynamic Product Showcase: With just a single product image, it can generate advertising videos featuring virtual host presentations and multi-angle displays, suitable for product detail pages on platforms such as Taobao and Douyin.

• Low-cost Localized Advertising: Rapidly generates advertising content adapted to different regional markets' languages and cultural backgrounds, significantly reducing cross-border marketing costs.

Social Media Content Creation

• Mukbang/Tutorial Video Generation: Input ingredient lists and step descriptions to automatically generate cooking process videos, with support for adding interactive subtitles and background music.

• UGC Assistance Tools: Provides creators with a "copywriting-to-video" feature, automatically converting text and image content into short videos to boost content production efficiency.

Film, Television, and Game Pre-production

• Storyboard Preview: Quickly generates storyline storyboards through text, helping directors and screenwriters visualize narrative pacing.

• NPC Action Library Construction: Provides batch character action generation for game development, reducing manual modeling workload.

The release of Goku marks the leap of AI video generation from an "auxiliary tool" to a "production-grade application." Its low-cost nature has already attracted several international FMCG brands to conduct pilot collaborations, with estimates suggesting it can reduce advertising production cycles by 80%. In the future, combined with real-time rendering and 3D modeling technology, Goku is expected to further expand into virtual reality, metaverse, and other scenarios. Current technical limitations are primarily reflected in plot logic control for long videos, but the team has indicated they will continue to optimize by introducing reinforcement learning mechanisms.

GitHub: https://github.com/Saiyan-World/goku

Project Page: https://saiyan-world.github.io/goku/

Paper: https://arxiv.org/abs/2502.04896