FantasyTalking: Alibaba's Tool for Generating Realistic Talking Portraits from a Single Static Image 58

What is FantasyTalking?

FantasyTalking is a virtual character animation generation tool developed by Alibaba, capable of generating realistic dynamic talking portraits from a single static portrait. It adopts a two-stage audio-visual alignment strategy to ensure precise lip-sync with speech, while generating natural and diverse animations through a face-focused cross-attention module and a motion intensity modulation module. When applied to gaming, film and television, and virtual reality, it can quickly generate high-quality virtual character animations.

FantasyTalking Technical Principles

• Two-stage audio-visual alignment strategy: The first stage uses a clip-level training scheme to align audio-driven dynamics across the entire scene, including the reference portrait, contextual objects, and background, establishing coherent global motion; the second stage refines lip movements at the frame level through lip-tracking masks to ensure precise synchronization with the audio signal.

• Face-focused cross-attention module: Replaces the commonly used reference network, effectively maintaining facial consistency in the video.

• Motion intensity modulation module: Explicitly controls the intensity of expressions and body movements, so that the portrait's motion is not limited to lip movements, enabling more natural and diverse animations.

FantasyTalking Main Features

• Lip-sync: Accurately identifies and synchronizes the virtual character's lip movements with the input speech, making the character's mouth movements fully consistent with the spoken content.

• Facial motion generation: Generates rich facial actions such as blinking, frowning, and smiling based on speech content and emotional information.

• Full-body motion generation: Generates full-body actions such as walking, running, and jumping based on scene and plot requirements.

FantasyTalking Core Advantages

• Highly realistic: Leveraging advanced artificial intelligence algorithms and deep learning techniques, the generated virtual character animations are indistinguishable from real humans in appearance, motion, and expression.

• Flexible and easy to use: Provides a clean and intuitive user interface along with rich API interfaces, supporting multiple input and output formats, making it easy for users to get started and integrate into their own projects.

• Efficient and stable: Uses optimized algorithms and an efficient computing architecture to generate high-quality virtual character animations in a short time, while ensuring system stability and reliability.

FantasyTalking Application Scenarios

• Game development: Can be used to generate dialogue animations, battle animations, and more for game characters, enhancing the game's visual effects and interactivity.

• Film and television production: Helps production teams quickly generate high-quality virtual character performance animations, special effect animations, and more, reducing production costs and time.

• Virtual reality and augmented reality: Generates interactive animations and guidance animations for virtual characters for VR and AR applications, enhancing user immersion and experience.

Project website:https://fantasy-amap.github.io/fantasy-talking/

Github repository:https://github.com/Fantasy-AMAP/fantasy-talking

arXiv paper:https://arxiv.org/pdf/2504.04842