Media2Face_云技派,专注ai人工智能! 79

In the field of artificial intelligence, creating accurate facial lip movements from speech has been an ongoing challenge. However, a breakthrough has arrived — Media2Face.

Media2Face is a co-speech facial animation generation tool guided by audio, text, and image multimodal inputs. It leverages guidance from audio, text, and image inputs to naturally mimic human speech, introducing the Generalized Neural Parametric Facial Assets (GNPFA). This is an efficient latent space that decouples expression from identity. Then, we use a variational autoencoder built upon GNPFA, extracted from a large amount of video, to map facial geometry and images into a highly generalized expression latent space, yielding high-quality expressions and accurate head poses.

Due to the limited availability of high-quality 4D facial data, synthesizing 3D facial animation from speech presents challenges. However, with Media2Face, it is now possible to seamlessly generate realistic and expressive facial animations from any audio, image, or text input.

Media2Face is capable of generating expressive 3D facial animations synchronized with speech based on sound. At the same time, it allows users to perform more granular personalized adjustments to the generated facial animations, such as emotion adjustments like "happy" or "sad." Media2Face can also understand multiple types of input information (audio, text, image) and use them as guidance for generating facial animations.

Media2Face Practical Applications:

• Creating dialogue scenes: Based on the script you write, the computer can generate animation scenes of character dialogues.

• Producing stylized facial animations: You can give the computer an emoji, and it can create an animation based on that emoji.

• Emotional singing: The computer can also sing in different languages, expressing corresponding emotions.

• Personalized animation: Most amazingly, this project can create personalized facial animations that match different ethnicities, ages, and genders.

• Film and television production companies use Media2Face to generate facial animations for virtual characters in films.

• Virtual host platforms leverage Media2Face to generate facial expressions for virtual hosts.

• Game development companies apply Media2Face in virtual character design for facial animation generation.

How does Media2Face work?

The working principle of the Media2Face project involves several key technologies and steps, enabling it to synthesize 3D facial animations with rich expressions and emotions from speech. The following is the main workflow of the project:

1. Generalized Neural Parametric Facial Assets (GNPFA):

First, the research team created a special tool (called GNPFA), which acts like a large facial expression database. No matter what expression you want, this tool can help you find it, and it also ensures that each person's facial animation is unique and won't be confused with others.

This process achieves the decoupling of expression and identity, meaning it can transfer the same expression across different identities.

Then, they used this tool to process a large amount of video to extract high-quality expressions and head movements. This created a massive dataset containing various facial animations and their corresponding emotion and style labels.

2. Multimodal-Guided Animation Generation:

Media2Face employs a diffusion model to perform animation generation in the latent space of GNPFA. This model can accept multimodal guidance from audio, text, and images.

The model uses audio features and CLIP latent codes as conditions to denoise, together with the noisy version of the expression latent code sequence and head motion codes (i.e., head poses). The conditions are randomly masked and processed through cross-attention with the noisy head motion codes.

3. Expression and Head Pose Generation:

Using GNPFA, high-quality expressions and accurate head poses are extracted from a large amount of video. This presents the M2F-D dataset, a large, diverse, and scan-level co-speech 3D facial animation dataset with well-annotated emotion and style labels.

4. Expression and Style Fine-Tuning:

Key frame expression latent codes are extracted through the expression encoder, and style prompts for each frame, such as "happy" or "sad," are provided through CLIP, allowing users to adjust the intensity and control range of the animation.

Through the above technical steps, Media2Face is capable of generating expressive 3D facial animations synchronized with speech, supporting complex emotional expressions and style variations, providing a powerful tool for creating virtual characters and enhancing the interactive experience of AI digital humans. Extensive experiments show that Media2Face not only achieves high fidelity in facial animation synthesis but also broadens the expressiveness and style adaptability of 3D facial animation.