VACE: A Unified Model for Video Generation and Editing by Alibaba Tongyi Wan Team 187

What is VACE?

VACE (Video Creation and Editing) is a one-stop unified ai video generation and editing model launched by Alibaba Tongyi Lab. It can perform a variety of video creation and editing tasks through a unified model architecture. It can generate videos from a piece of text or an image, and can also perform style transfer, restoration, or extension on existing videos. It integrates multiple functions into a single model, making it easy to operate and suitable for creating creative videos or restoring old videos.

VACE Core Features

Video Generation:

• Text-to-Video Generation: Generate videos based on text prompts.

• Reference-to-Video Generation: Generate videos by combining text and reference images.

• Video Expansion: Generate new openings or endings based on existing video clips.

Video Editing:

• Video-to-Video Editing: Perform overall style transfer on input videos (such as colorization and stylization).

• Masked Video Editing: Perform editing in specified areas, such as Inpainting and Outpainting.

• Subject Removal and Reconstruction: Remove specific subjects from the video and fill in the background.

Task Combination:

• Supports flexible combinations of multiple tasks, such as reference generation + subject replacement, pose control + video expansion.

VACE Technical Principles

• Video Condition Unit (VCU): Integrates text, image, video, mask, and other multimodal inputs into a unified conditioning unit, supporting flexible combinations of multiple tasks.

• Context Adapter Structure: Injects the concepts of different tasks into the model through formalized representations in temporal and spatial dimensions, enabling it to adapt to different task requirements.

• Diffusion Model: Built on top of a diffusion model (such as the Diffusion Transformer), it generates high-quality video content through step-by-step denoising.

VACE Advantages

• Unified Framework: Integrates video generation and editing tasks into a single model, reducing the complexity of switching between tasks.

• High Efficiency: Achieves performance comparable to task-specific models across a variety of tasks.

• Diversity: Supports multiple task combinations, providing broader application possibilities for video content creation.

VACE Application Scenarios

• Creative Video Generation: Quickly generate creative video content such as advertisements and animations based on text or images.

• Video Restoration and Enhancement: Restore old videos, fill in missing parts of the picture, or enhance video styles.

• Efficient Video Editing: Perform complex editing tasks such as subject replacement and animation addition.

• Video Expansion: Generate new clips for short videos to extend the video content.

• Interactive Video Creation: Generate personalized videos based on user input (such as poses or sketches).

Project Resources

Project Website:https://ali-vilab.github.io/VACE-Page/

GitHub Repository:https://github.com/ali-vilab/VACE

Technical Paper:https://arxiv.org/pdf/2503.07598