ChatAnyone: Alibaba's Portrait Video Generator from Audio with Rich Expressions and Upper-Body Motion 147

小编 · 2025-03-28

ChatAnyone is a real-time stylized portrait video generation framework developed by Alibaba's Tongyi Lab. It upgrades from "talking heads" to high-fidelity upper-body interactive animation and supports diverse facial expressions and style control. Ideal for real-time video chat, virtual streamers, AI assistants, and more.

ChatAnyone: Alibaba's Portrait Video Generator from Audio with Rich Expressions and Upper-Body Motion

What is ChatAnyone?

ChatAnyone is a real-time stylized portrait video generation framework developed by Alibaba Group's Tongyi Lab, achieving high-fidelity animation generation from "talking heads" to upper-body interactions, supporting diverse facial expressions and style control. It is suitable for real-time video chat, virtual streamers, AI assistants, and other scenarios.

ChatAnyone: Portrait video with rich expressions and upper-body movements generated through audio input, released by Alibaba.webp

ChatAnyone Technical Approach

Efficient Hierarchical Motion Diffusion Model

Input: audio signal.
Output: control signals for face and body, considering explicit and implicit motion signals.
Function: generates diverse facial expressions and synchronizes head and body movements.
Fine-grained expression control: supports expression variations of different intensities, as well as transferring stylized expressions from reference videos.

Hybrid Control Fusion Generative Model

Facial expression generation: combines explicit landmarks and implicit offsets to produce realistic facial expressions.
Hand motion control: injects explicit hand control signals to generate more accurate and realistic hand movements.
Face refinement module: enhances facial realism, ensuring the generated portrait video is highly expressive and lifelike.

Scalable Real-Time Generation Framework

Flexibility: supports animation from head-driven to upper-body generation with gestures.
Real-time performance: generates upper-body portrait videos in real time on a 4090 GPU at up to 512×768 resolution and 30fps.

ChatAnyone Experimental Results

Audio-driven upper-body animation: generates highly expressive upper-body digital human videos, supporting both hand-present and hand-free scenarios.
Audio-driven head animation: achieves high-precision lip-sync, generating natural head poses and rich facial expressions.
Audio-driven stylized animation: supports the generation of stylized character animations, and is also suitable for generating expressive singing videos.
Dual-host AI podcast demonstration: capable of generating dual-host podcasts, supporting AI-driven conversations.
Real-time interactive demonstration: achieves 30fps real-time generation on a 4090 GPU, supporting practical interactive video chat applications.

ChatAnyone: Portrait video with rich expressions and upper-body movements generated through audio input, released by Alibaba.webp

ChatAnyone Advantages

High fidelity and naturalness: the generated portrait videos feature rich expressions and natural upper-body movements.
Real-time performance: supports real-time interaction, suitable for application scenarios such as video chat.
Stylized control: expression styles can be adjusted as needed to achieve personalized animation generation.

Application Scenarios of ChatAnyone

Virtual Streamers and Video Conferencing

Virtual streamers for news broadcasting and live commerce.
Virtual avatars in video conferencing to enhance interaction.

Content Creation and Entertainment

Animation production: generating stylized animated characters.
Virtual concerts: generating real-time animations of virtual singers or performers.
AI podcasts: generating virtual avatars for dual-host podcasts.

Education and Training