Emote Portrait Alive: Generate Singing Videos from a Single Image 130

Emote Portrait Alive is EMO, an audio-driven AI portrait video generation framework released by Alibaba. Given a single reference image and a vocal audio—such as speech or singing—Emote Portrait Alive can generate an audio-driven portrait video with expressive facial expressions and a variety of head poses, allowing the person in the photo to naturally open their mouth to speak or sing in sync with the audio content, with highly realistic facial expressions and head movements. At the same time, it can generate videos of any duration based on the length of the input audio.

Emote Portrait Alive Features:

• Audio-driven video generation: EMO is able to directly generate videos based on input audio (such as speech or singing), without relying on pre-recorded video clips or 3D face models.

• High expressiveness and realism: The generated videos are highly expressive, capturing and reproducing the subtleties of human facial expressions—including micro-expressions—as well as head movements that match the rhythm of the audio.

• Seamless frame transitions: Ensures smooth and natural transitions between video frames, avoiding issues such as facial distortion or inter-frame jitter, thereby improving the overall quality of the video.

• Identity preservation: Through the Frame Encoding module, EMO is able to maintain character identity consistency throughout the video generation process, ensuring that the character's appearance stays consistent with the input reference image.

• Stable control mechanism: Adopts stable control mechanisms such as a speed controller and a face region controller to enhance stability during the video generation process and avoid issues such as video collapse.

• Flexible video duration: Can generate videos of any length based on the length of the input audio, providing users with flexible creative space.

• Cross-language and cross-style: The training dataset covers multiple languages and styles, including Chinese and English, as well as realism, anime, and 3D styles, enabling EMO to adapt to different cultural and artistic styles.

Emote Portrait Alive Use Cases:

• Singing video generation: Input a portrait photo and a singing audio, and EMO can generate an expressive singing video synchronized with the audio.

• Speaking video generation: Not limited to singing, it can also handle spoken audio to generate speaking videos synchronized with the voice.

• Multi-language support: Supports audio input in multiple languages and can generate portrait videos in different languages.

• Cross-style applications: Portraits in different styles (such as anime or 3D models) can be combined with audio to generate video content with a specific style.

Emote Portrait Alive is not limited to handling singing audio input—it can also accommodate spoken audio in various languages. In addition, Emote Portrait Alive is able to turn old photos, paintings, as well as 3D models and AI-generated content, into videos, bringing them to life with realistic motion and authenticity.