Alibaba EMO: Turn a Single Photo and Audio into Talking or Singing Videos 288

Alibaba Group's Institute for Intelligent Computing proudly presents EMO: Emote Portrait Alive, a groundbreaking AI that generates expressive portrait videos from just an image and audio! The length of the generated video matches your audio length, bringing talking and singing videos to life, supporting any voice, any speech speed, any image...

Unlike traditional methods, EMO uses a direct audio-to-video approach, bypassing the need for intermediate 3D models or facial landmarks. This ensures seamless frame transitions and consistent identity preservation throughout the video, resulting in highly expressive and lifelike animations, meaning smoother transitions and more natural expressions.

EMO even excels at creating singing videos in various styles! Imagine making a music video using just one image of your favorite artist!

EMO Research Paper: https://arxiv.org/pdf/2402.17485.pdf

EMO Open Source Address: https://humanaigc.github.io/emote-portrait-alive/

EMO Main Features:

1. Audio-driven portrait video generation: Using a single reference image and audio input (such as talking or singing), EMO can generate virtual portrait videos with expressive changes and head dynamics. This means users can create dynamic talking or singing videos by providing a static image and the corresponding audio file. Regardless of the facial expression changes or head movements in the video, the underlying features all come from this reference image.

2. Expressive dynamic rendering and head pose support: EMO particularly emphasizes generating natural and expressive facial movements in videos. It can capture subtle facial expressions and head motions, creating lifelike talking and singing videos, thereby producing facial animations that look natural and vivid, adding dynamism and realism to the video.

3. Support for multiple languages and portrait styles: This technology is not limited to specific languages or music styles. It can process audio input in multiple languages and supports diverse portrait styles, including historical figures, paintings, 3D models, and AI-generated content.

4. Fast-paced rhythm synchronization: EMO can handle fast-paced audio, such as rapid lyrics or speech, ensuring that the virtual portrait's actions stay synchronized with the audio rhythm.

5. Cross-actor performance transfer: EMO can achieve performance transfer between different actors, allowing one actor's virtual avatar to mimic the specific performance of another actor or voice, expanding the diversity and application scenarios of character portrayal.

EMO Application Scenarios:

• Create instant dubbing

• Create realistic and expressive characters for movies and video games

• Develop engaging interactive educational materials

• Give virtual assistants a more human-like look and feel

• Make video conferencing more engaging and personalized

• Produce more engaging and memorable marketing and advertising videos

Experimental results show that EMO can not only produce convincing spoken videos but also create singing videos in various styles, significantly outperforming existing state-of-the-art methods in terms of expressiveness and realism. It sets a new standard for video generation and is of great significance for the future.