Audio2Face: Generate 3D Facial Animation from Voice 128

Audio2Face, an application powered by AI, generates expressive 3D facial animation from voice. Audio2Face simplifies the animation of 3D characters to match any voiceover track, whether you're animating characters for games, films, real-time digital assistants, or just for fun. You can use this application for interactive real-time applications or as a traditional facial animation creation tool.

How does Audio2Face work?

The Omniverse Audio2Face application is based on an original paper from NVIDIA Research. Audio2Face comes preloaded with "Digital Mark"—a 3D character model that can be animated with your audio track, so getting started is simple. Just select your audio and upload it into the application. The technology takes the audio input and feeds it into a pre-trained deep neural network, whose output drives the character's facial animation in real time. Users can choose to edit various post-processing parameters to edit the character's performance. The network's output then drives the 3D vertices of the character mesh to create facial animation. Most of the results you see on this page are Audio2Face's raw output, with almost no post-processing parameters edited.

Real-world applications of Audio2Face:

Language

Audio2Face will be able to handle any language with ease. We will regularly add more and more languages. Check out these tests in English, French, Italian, and Russian.

Misty Animated Chat Robot

Shown at GTC Spring 2020, Misty is an interactive weather robot driven by Audio2Face at runtime. We demonstrated retargeting from a realistic human mesh to stylized character meshes for use as interactive service agents.

Omniverse Machinima

Audio2Face debuted at the GeForce 30 Series launch event, appearing in the Omniverse Machinima demo. As we all know, facial animation is extremely complex and costly. Audio2Face automates detailed facial animation, democratizing the 3D content creation process.

Face

Audio2Face can use any conversation-driven audio track and ultimately retarget to any 3D face, whether realistic or stylized. Watch this test where we retarget from Digital Mark to Rain.

Experimental Characters

Creatures and Aliens

This feature allows you to retarget and drive facial animation for fantastical creatures and aliens. Here, we have Digital Mark driving the performance of an alien.