MAGREF: ByteDance's Multi-Subject Video Generation Tool 71

MAGREF is a multi-subject video generation tool launched by ByteDance. It allows users to generate highly consistent videos from a single reference image, handling both single-subject and multi-subject scenes with ease. By employing a mask-guided and channel concatenation mechanism, it addresses common issues in traditional video generation, such as identity confusion and uncoordinated actions. MAGREF also supports various combinations of people, objects, and backgrounds, ensuring identity stability and structural consistency in the generated videos.

Core Features

• Multi-Subject Consistency Generation: With just one reference image, it can generate videos of people or objects with high consistency. It supports multiple people in the same frame without any "face mixing" issues. For example, you can use it to generate a video of Einstein riding a motorcycle, or Newton writing and painting. The characters' faces and clothing remain consistent across every frame, without becoming increasingly distorted.

• Diverse Reference Image Processing: Through a mask-guided and channel concatenation mechanism, it can handle multiple reference images without increasing model complexity. Whether it's a single-person performance, multi-person interaction, or scenes featuring people together with objects and backgrounds, it can generate stable and coherent videos.

• Multimodal Input and Generation: Based on different text prompts, it can achieve "one image, a thousand faces." Regardless of who is in the input, which era they come from, or whether the style is abstract, it can accurately restore their features and generate different actions, environments, and lighting effects according to text instructions. For example, you can input a portrait, an object image, a background image, plus a descriptive text, and it will generate a complete video containing these three types of elements. The interactions between people and objects look natural, and the scenes feel perfectly seamless.

Technical Implementation

• Data Processing Pipeline: It adopts a three-stage data processing pipeline to construct high-quality, structurally clear video training samples. The first stage involves segmenting semantically consistent clips from raw videos, filtering out low-quality samples, and generating structured text for each segment. The second stage identifies critical objects in the video through label extraction and semantic segmentation, followed by post-processing to obtain precise masks. The third stage detects and assigns identities to people in the video, screening high-quality facial images for reference image construction, ensuring identity consistency during the training process.

Key Mechanisms

• Region-Aware Dynamic Masking: Based on deep learning attention mechanisms, the model gains "dynamic visual perception" capabilities. Through semantic segmentation algorithms, it accurately identifies key regions in the video and dynamically adjusts mask weights for different subjects based on input prompts. This way, even when the number and order of reference images vary, the system maintains structural consistency, keeps identities distinct, and ensures clear relationships.

• Pixel-Level Channel Concatenation: In the underlying technology of image generation, it performs information fusion along the channel dimension (RGB channels + feature channels). A convolutional neural network is used to extract appearance features from reference images, which are then concatenated at the channel level with the feature maps of the generated scene, ensuring that details such as skin tone and hairstyle are highly consistent with the reference image. At the same time, normalization operations are applied to balance lighting parameters across different scenes, preventing a sense of disconnection between subjects and backgrounds due to lighting differences. It also supports the simultaneous input of portraits, object images, and environmental photos, generating logically coherent cross-modal scenes through channel-level fusion.

Application Scenarios

• Film and Television Production: It can reshape the pre-production and visual-effects processes of filmmaking. For example, during the script stage, it can be used to generate dynamic storyboards, allowing directors to adjust scene color tones and character actions in real time. It can also replace some green screen shooting needs. For instance, in historical dramas, inputting reference images of a few soldiers allows for the batch generation of group marching videos that conform to ancient clothing and formations.

• Game Development: It accelerates the iteration of game content and enhances immersion. Based on game character modeling images, it can quickly generate videos of characters' expressions and actions at different story nodes; it can also automatically generate NPCs' daily activities in towns without the need for manual, frame-by-frame animation debugging.

• Advertising and Marketing: It acts as a "mass production engine" for personalized content, capable of generating hundreds of use-case videos for the same product to meet the fragmented distribution needs of social media. It can also generate multi-version advertising materials based on celebrity portraits, avoiding the scheduling constraints and copyright disputes associated with real-life shoots.

Project Links

• Project Homepage:https://magref-video.github.io/magref.github.io/

• Github:https://github.com/MAGREF-Video/MAGREF

• Paper:https://arxiv.org/pdf/2505.23742