JoyGen: Audio-Driven 3D Talking Face Video Generation Framework 122

JoyGen is an audio-driven 3D talking face video generation framework jointly developed by JD Tech and the University of Hong Kong. It is a novel two-stage framework that drives a 3D depth-aware model through audio input to provide precise lip-audio synchronization, accurately simulating the speaker's lip movements and facial expressions, making the generated videos more realistic. It is mainly used in the fields of video editing and virtual interaction.

JoyGen is trained on a Chinese talking face dataset containing 130 hours of high-quality videos. This dataset, combined with the open-source HDTF (High-Definition Talking Face Dataset), supports both Chinese and English input.

In addition, JoyGen also takes into account the emotional characteristics of the audio, and is able to naturally portray the character's emotional changes in the generated animation, such as smiling or frowning, which is extremely powerful.

Core Features of JoyGen:

• Multilingual Support: JoyGen is capable of handling audio input in multiple languages, including Chinese and English.

• High-Quality Visual Effects: Through meticulous handling of facial expressions and lip details, the generated videos are extremely realistic.

• Precise Lip Synchronization: Through audio feature analysis and facial depth map techniques, the lip movements of characters in the video perfectly match the audio content.

• Video Editing Optimization: It can not only generate new videos, but also edit the lip movements of existing videos, without the need to re-create the entire video sequence from scratch.

• Efficient Technical Architecture: It adopts a single-step UNet architecture, which makes the video editing process simpler.

Technical Principles:

1. Audio-Driven Lip Motion Generation:

• Use a 3D reconstruction model to extract identity features from facial images, defining the character's unique facial structure.

• Through an audio-to-motion model, convert audio signals into expression coefficients that control lip movements.

• Combine identity and expression coefficients, and use differentiable rendering techniques to generate facial depth maps, preparing for subsequent video synthesis.

2. Visual Appearance Synthesis:

• A single-step UNet network is used to integrate audio features with depth maps and directly generate video frames containing precise lip movements.

• A cross-attention mechanism is introduced to ensure that lip movements are highly consistent with audio signals, enhancing synchronization.

• The optimization process (such as the L1 loss function) ensures video quality, balancing pixel-level and latent space accuracy.

How to Use JoyGen:

• Environment Setup: Users need to create a specific conda environment and install the necessary dependency packages, including specific libraries such as Nvdiffrast.

• Download Pretrained Models: Obtain JoyGen's pretrained models, including the 3D model, audio-to-motion model, etc. These resources are usually available on the project's GitHub page.

• Run Inference: By executing specific scripts and parameters, users can convert audio files into 3D talking face videos with realistic lip synchronization.

Applications of JoyGen:

• Virtual Meetings: Enhance facial expressions in virtual meetings.

• Film and Television Production: Create special effects in movies and television.

• Education and Training: Used to produce vivid educational videos.

• AI Assistants: Increase the anthropomorphic degree of AI assistants in human-machine interaction.

JoyGen greatly simplifies the complex task of human face video generation. It plays a very significant role in digital human content creation, virtual meetings, entertainment, and other areas.

GitHub: https://github.com/JOY-MM/JoyGen