Follow-Your-Emoji: A Portrait Animation Project 236

Follow-Your-Emoji, jointly developed by HKUST, Tencent Hunyuan, and Tsinghua University based on a diffusion model, applies target expressions or motions to a reference portrait while preserving the portrait's identity features, motion consistency, and realism. It generates new facial animations by extracting facial features—especially fine-grained facial details and pupil dynamics—from a video subject.

You only need to provide a single photo, and Follow-Your-Emoji can transform it into a vivid facial animation, while capturing subtle expression changes.

Follow-Your-Emoji can use a single driving video or motion sequence to drive multiple different reference avatars to generate animations. Each reference avatar will exhibit the same motions and expression changes according to the same motion sequence.It can also use a single reference avatar combined with multiple different driving video generation inputs to produce multiple animation segments. Each motion sequence will make the same reference avatar exhibit different motions and expression changes.

Follow-Your-Emoji Features:

• Identity Preservation: During the animation generation process, it ensures that the identity features of the reference avatar are not lost, keeping your image consistent throughout.

• Rich Expressions: Generates a wide range of exaggerated and subtle expressions, such as large-scale pupil movements—whether it's fine eye dynamics or dramatic expression changes.

• Diverse Styles: Supports avatar animations in multiple styles, including cartoon, realistic characters, sculptures, animal-style animations, and more.

• Stable and Long-Lasting Animations: Uses a progressive generation strategy to ensure stability and consistency in long-duration animations.

The main technical challenge lies in consistently preserving the reference avatar's identity features throughout the animation process while transferring the target expressions, and ensuring temporal coherence and realism.

To address this challenge, Follow-Your-Emoji employs two key techniques. First, it uses a new explicit motion signal—expression-aware keypoints—to guide the animation process. These keypoints ensure accurate alignment between the reference avatar and the target motion during inference, and they enhance the ability to express exaggerated expressions (such as large-scale pupil movements) while avoiding leakage of identity information. Second, it proposes a facial fine-grained loss function that improves the model's ability to capture subtle expression changes and reconstruct the reference avatar's appearance by using expression and facial masks.

Follow-Your-Emoji is ideal for professionals and enthusiasts engaged in portrait animation production, including animation designers, video creators, game developers, and more.

Paper: https://arxiv.org/html/2406.01900v1

Follow-Your-Emoji Project Page: https://follow-your-emoji.github.io/