AnchorCrafter: AI Virtual Streamer Tool Co-Built by CAS and Tencent 124

AnchorCrafter is a powerful AI virtual anchor livestream selling tool jointly developed by the Chinese Academy of Sciences and Tencent. It leverages advanced AI technology to generate natural and smooth product promotion videos, greatly enhancing the quality and efficiency of e-commerce video production.

What is AnchorCrafter?

AnchorCrafter is a diffusion-based system that automatically creates high-fidelity anchor-style product promotion videos by integrating Human-Object Interaction (HOI) into pose-guided human video generation.

AnchorCrafter Technical Highlights:

• Human-Object Interaction Modeling: AnchorCrafter not only drives virtual models to showcase products, but also accurately captures and controls the interaction details between humans and products, such as hand-to-product contact, thereby generating more natural and realistic video content.

• High-Quality Visual Effects: The system uses HOI-appearance perception technology to integrate multi-view product image features, ensuring that the appearance details of products in videos are accurately presented. At the same time, it employs a decoupled network structure to distinguish and optimize the appearance of humans and objects, enhancing visual fidelity.

• Efficient Use of Training Data: Despite a relatively small training dataset (44 web videos and 307 self-collected videos), AnchorCrafter leverages efficient training strategies to produce satisfactory results under limited resources (such as 4 A100 GPUs, 40,000 training steps, and 6 frames per second length), demonstrating its efficiency and robustness.

• HOI-region Reweighting Loss: To strengthen the model's learning of hand-object interactions, the system adopts a specific loss function that increases the weight of the loss for hand and object regions, ensuring the accuracy and naturalness of interaction parts.

AnchorCrafter's Technical Principles

• Video Diffusion Model: Uses a diffusion Unet and a Variational Autoencoder (VAE) to process video frames, encoding video sequences into latent space to generate high-quality outputs.

• HOI-Appearance Perception: Extracts appearance features of humans and objects and integrates them into the backbone network to achieve appearance control, using multi-view object reference images to enhance the recognition of object appearance.

• HOI-Motion Injection: Controls object motion by injecting human motion sequences, and addresses issues of object trajectory conditioning and occlusion management, using depth maps as input to maintain motion clarity and consistency.

• HOI Region Reweighting Loss: By applying higher training weights to interaction regions, the model's learning of object details is enhanced, improving the quality and realism of the generated videos.

AnchorCrafter's Main Uses

• E-commerce Video Production: AnchorCrafter can generate natural and smooth product promotion videos based on character images and products to be promoted provided by users, helping e-commerce businesses effortlessly produce high-quality marketing videos.

• Product Promotion: By integrating people with products, it creates attractive product promotion videos that enhance the market competitiveness of products.

AnchorCrafter's Advantages

• Image Quality and Motion Naturalness: The virtual anchor livestream videos generated by AnchorCrafter achieve industry-leading levels in image quality and motion naturalness, even comparable to those of real-life anchors.

• Efficiency: Users only need to provide simple images and product information to quickly generate high-quality product promotion videos, significantly saving time and costs.

Steps to Use AnchorCrafter:

• Prepare Materials: Provide a photo of the anchor. Provide photos of the products to be showcased.

• Select an Anchor: Choose your favorite anchor from a wide range of web anchors, and use AI technology to "clone" their image into the virtual world.

• Generate Video: AnchorCrafter will automatically generate an excellent livestream selling video, including the anchor's movements, expressions, and interactions with the products.

• Precise Control: Design various interactive actions according to your needs, allowing the anchor and products to blend perfectly.

• Export Video: Once generation is complete, export the high-quality video for use in e-commerce platform promotions.

AnchorCrafter is particularly suitable for e-commerce platforms, as it can quickly generate product promotion videos, enhance the attractiveness of product displays, increase sales potential, and reduce reliance on real-life anchors while lowering production costs.