Vid2Avatar-Pro: Generate Realistic 3D Human Avatars from Monocular Video 189

What is Vid2Avatar-Pro?

Vid2Avatar-Pro is a technology that can create realistic and animatable 3D human avatars from ordinary monocular natural video. As an upgraded version of Vid2Avatar, it enhances the realism and generalization capabilities of the avatars by introducing a Universal Prior Model (UPM) and an optimized 3D representation method.

Vid2Avatar-Pro Features

• High Realism and Animatability: Vid2Avatar-Pro can generate realistic 3D avatars from ordinary monocular video and supports complex motion animations.

• Multi-View Rendering: The generated avatars can be rendered from any viewpoint, addressing the limited-view issue of traditional methods.

• Diverse Motion Support: The avatars can perform a variety of complex motions, such as playing badminton, playing drums, and single-leg jumping.

• Universal Prior Model (UPM): A universal prior model pretrained on large-scale multi-view data, significantly improving generalization to new poses and viewpoints.

Vid2Avatar-Pro Use Cases

• Film and TV Production: Reconstruct 3D models of actors from ordinary video for special effects production.

• Virtual Reality (VR) and Augmented Reality (AR): Create realistic virtual characters to enhance immersion.

• Fashion and E-commerce: Reconstruct user 3D models from video to enable virtual try-on.

• Sports Analysis: Reconstruct athlete models from game video to analyze motion details.

• Medical Rehabilitation: Record and analyze patient movements during the rehabilitation process.

Vid2Avatar-Pro Technical Principles

Universal Prior Model (UPM)

• Pretraining Stage: UPM is trained using large-scale multi-view, dynamic data of clothed human performances, enabling it to capture general features of human appearance and pose.

• Finetuning Stage: On a given monocular in-the-wild video, the pretrained UPM is finetuned via inverse rendering techniques to recover the details of a specific person.

3D Gaussian Representation

• Use 3D Gaussian representation with canonical front and back maps to construct the avatars; these maps are shared across different identities.

• Synthesize training pose/shape parameters via forward Linear Blend Skinning (LBS) and rasterization techniques.

Self-Supervised Scene Decomposition

• Separates the human body and background directly in 3D without relying on external segmentation modules.

• Parameterizes the person and background via two independent neural fields, addressing the scene decomposition and surface reconstruction tasks.

Optimization Strategy

• Introduces a coarse-to-fine sampling strategy and novel objective functions to clearly separate the dynamic human body from the static background.

• Optimizes the background model, normalized human shape and texture, as well as per-frame human pose parameters.

How to Use Vid2Avatar-Pro

1. Data Preparation:

• Use ROMP to obtain the initial SMPL shape and pose.

• Use OpenPose to extract keypoints.

2. Model Training:

• Jointly optimize the background model, human shape and texture, and pose parameters.

• Finetune using the pretrained UPM model.

3. Generation and Rendering:

• Use the trained model to generate the final 3D avatar.

• Achieve multi-view rendering through inverse rendering techniques.

4. Animation Generation:

• Bind the avatar to new motion sequences to produce diverse animations.

Paper Link:https://moygcc.github.io/vid2avatar-pro/static/CVPR2025_Vid2Avatar_Pro.pdf

Project Page:https://moygcc.github.io/vid2avatar-pro/