Audio2Photoreal: Generate Realistic Virtual Characters from Audio 216

Generate fully photorealistic virtual avatars from audio. It produces lifelike facial expressions, full-body gestures, and hand motions aligned with multi-speaker conversations.

#Ai工具箱 #Ai开源项目

Visit Audio2Photoreal: Generate Realistic Virtual Characters from Audio

Audio2Photoreal: Generate Realistic Virtual Characters from AudioIntroduction

Meta ai recently released a groundbreaking technology: Audio2PhotoReal, which generates full-body photorealistic virtual avatars from audio.It can generate realistic facial expressions, complete body movements, and gestures that correspond to multi-person conversations from their speech.

These generated virtual avatars are not only visually realistic, but also accurately reflect the gesture and expression details in the conversation, such as pointing, wrist flicks, shrugs, smiles, smirks, and more.

How Audio2PhotoReal Works:

Audio2PhotoReal combines the advantages of vector-quantized sample diversity with high-frequency details obtained through diffusion to generate more dynamic and expressive motions.

1. Dataset Capture: First, a rich dataset of two-person conversations is captured, which enables realistic reconstruction.

2. Motion Model Construction: The project builds a composite motion model that includes a facial motion model, a guiding pose predictor, and a body motion model.

3. Facial Motion Generation: A pre-trained lip regressor is used to process the audio and extract features related to facial motion.

A conditional diffusion model is used to generate facial motions based on these features.

4. Body Motion Generation: Taking audio as input, it autoregressively outputs vector-quantized (VQ) guiding poses at 1 frame per second. The audio and guiding poses are then fed together into a diffusion model to generate high-frequency body motion at 30 frames per second.

5. Avatar Rendering: The generated facial and body motions are passed into a trained avatar renderer to generate photorealistic virtual avatars.

6. Results Display: The final output is a full-body photorealistic virtual avatar generated from audio, capable of exhibiting subtle expressions and gestures from the conversation.