StableAvatar: Generate Singing or Talking Videos Directly from Audio 135

What is StableAvatar?

StableAvatar is a tool developed by researchers from Fudan University, Microsoft Research Asia, and Xi'an Jiaotong University. It can directly generate singing or speaking videos from audio, with no limit on video length and high visual quality.

StableAvatar Core Technologies

• Timestep-aware Audio Adapter: This technology mainly prevents audio information from going wrong during the conversion process. It first performs "cross-attention" processing on the initial audio information and the image's initial information (simply put, it makes the two correlate better with each other), and then makes adjustments using "timestep embedding" (similar to fine-tuning the information at different time points), finally obtaining more accurate audio information and avoiding situations where the picture and audio don't match or the picture becomes distorted when generating the video later.

• Audio-Native Guidance Mechanism: In the final stage of video generation (that is, the inference stage), it uses the audio and picture information predicted by the model itself, treating this dynamically changing information as a "guidance signal" to make the generated video and audio more synchronized, avoiding situations where the sound and lip movements don't match.

• Dynamic Weighted Sliding-Window Denoising Strategy: When generating long videos, the picture tends to lag or become incoherent. This technology fuses the latent variable information of the picture along the time dimension (which can be understood as the hidden features of the picture), making the transitions between frames in the long video smoother, with no sudden jumps or distortions.

How StableAvatar Works

• Audio Processing: First, the input audio is passed through the Wav2Vec model to extract audio features (i.e., audio embeddings), and then these features are optimized using the previously mentioned "Audio Adapter" to make the audio information more accurate.

• Reference Image Processing (such as the person's initial photo): The reference image is fed into the video generation model through two paths:

• First Path: The reference image and zero-padded frames (equivalent to blank frames) are concatenated along the time axis, then passed through a fixed 3D VAE encoder to be converted into latent variable encoding (similar to the hidden information of the picture). This latent variable encoding is concatenated with the compressed video frames and a binary mask (used to mark which parts need to be focused on) along the channel axis, providing the basis for subsequent picture generation.

• Second Path: The reference image is converted into image embeddings through a CLIP encoder (a model capable of extracting key image features), and then these features are fed into each "image-audio cross-attention module" of the denoising module (DiT), with the purpose of keeping the appearance of the person in the generated video consistent with the reference image.

• Video Generation: During the inference stage, the real video frames that should have been input are replaced with random noise (this is a standard operation for this type of model), while other inputs (such as the optimized audio and the processed reference image information) remain unchanged, and finally continuous video frames are generated through model computation to form the complete video.

Advantages of StableAvatar

• Can Generate Extra-Long Videos: Previous tools could mostly only generate short videos of less than 15 seconds, while StableAvatar can generate videos of any length while still maintaining high visual quality (high fidelity).

• Consistent Character Appearance: Even when generating very long videos, the face and body movements of the person in the video remain consistent, without issues such as face distortion, weird body proportions, or mismatched appearance between the beginning and end.

• No Additional Processing Required: The generated videos can be used directly, with no need to fix the face using other tools (such as the common face-swapping tool FaceFusion or the restoration models GFP-GAN, CodeFormer, etc.), saving a lot of trouble.

Application Scenarios of StableAvatar

1. Film and Video Production

• High-Difficulty Action Scenes: No need for actors to personally perform risky actions (such as jumping off buildings or explosions); using it to generate videos can reduce costs and risks.

• Character Animation: Add realistic movements and expressions to characters in films or videos, making them more vivid.

• Old Film Restoration: Repair characters in old film footage to make them clearer, bringing classic characters back to life.

2. Game Development

• Character Animation: Character movements and expressions in games become more realistic, giving players a greater sense of immersion.

• Cutscenes: The transitional videos for game storylines become more exciting, attracting players to keep playing.

• Virtual Character Interaction: The movements and expressions of NPCs (non-player characters) in games become more natural, making the game world more realistic.

3. Virtual Reality (VR) and Augmented Reality (AR)

• Creating Virtual Characters: Generate realistic characters in VR/AR applications, making users feel as if they are interacting with real people.

• VR Socializing: Users can chat and interact with the generated virtual characters, adding fun to social interaction.

• VR Tourism: Allows users to "personally" experience scenery from around the world through VR devices, as if they were really traveling.

4. Social Media and Live Streaming

• Virtual Streamers: The generated virtual streamer images remain consistent, with more realistic animation effects during live streams.

• Creative Content: Short video creators and digital artists can use it to design various interesting dynamic images and produce more creative content.

5. Education and Customer Service

• Virtual Assistants: In education or customer service scenarios, generate virtual teachers or customer service representatives with consistent appearances, providing more vivid and personalized services (for example, a virtual teacher always keeps the same appearance when giving lessons to students).

6. Advertising and Digital Marketing

• Cinematic-Quality Ads: It can generate complex character movements and camera effects, suitable for producing ads, short films, or animated promotions, with visual quality reaching a cinematic level.