HRAvatar: High-Quality 3D Avatar Generation from Monocular Video 76

小编 · 2025-05-23

With the rapid growth of digital humans, virtual streamers, and AR/VR, generating realistic, animatable, and relightable 3D avatars has become a critical need. However, creating highly realistic animatable avatars from monocular video remains challenging. Existing 3D Gaussian Splatting (3DGS) methods face limitations in reconstruction quality, such as insufficient geometric deformation flexibility, inaccurate expression tracking, and the inability to achieve realistic relighting.

HRAvatar: High-Quality 3D Avatar Generation from Monocular Video

Against the backdrop of rapid development in digital humans, virtual streamers, and AR/VR, efficiently generating realistic, animatable, and relightable 3D avatars has become a key demand. However, generating highly realistic and animatable avatars from monocular video still faces many challenges. Existing methods based on 3D Gaussian Splatting (3DGS) have some limitations in reconstruction quality, such as insufficient flexibility for geometric deformation, inaccurate expression tracking, and the inability to achieve realistic relighting.

Tsinghua University, in collaboration with the IDEA team, has developed HRAvatar—a new technique that uses monocular video to create high-quality, relightable 3D avatars. It leverages learnable deformation bases and linear skinning to achieve personalized geometric deformation, while being equipped with a precise expression encoder that reduces tracking errors and improves reconstruction quality. HRAvatar excels in real-time performance and detail rendering, and has been accepted at the CVPR2025 conference. Experiments show that it outperforms existing techniques across multiple evaluation metrics, achieves a real-time performance of approximately 155 FPS, and also supports relighting under new lighting environments.

HRAvatar：单目生成高质量3D头像方法.jpg

Core Technology of HRAvatar

Learnable Deformation Bases and Linear Skinning: By adaptively learning the deformation basis and blend skinning weights for each point, it achieves flexible deformation of Gaussian points from a standard pose to various expressions and poses, solving the problem that generic parametric models struggle to accurately capture personalized expression changes.
Precise Expression Encoder: An end-to-end expression encoder is introduced and jointly optimized with the 3D avatar reconstruction, supervised by Gaussian reconstruction loss to reduce tracking errors and improve reconstruction quality.
Appearance Modeling and Physical Shading: The avatar's appearance is decomposed into material attributes such as albedo, roughness, and Fresnel reflection. Albedo pseudo-priors are introduced to better decouple material information, a simplified BRDF physical model is used for shading, and the SplitSum approximation technique is introduced to precompute the environment lighting map, achieving high-quality, relightable real-time rendering.

Technical Advantages of HRAvatar

High Reconstruction Quality: It outperforms existing methods across all evaluation metrics, particularly in LPIPS, where the reconstructed avatars feature richer details.
Excellent Real-Time Performance: The rendering speed reaches approximately 155 FPS, meeting real-time requirements.
Avatars reconstructed by HRAvatar can not only be driven but also be relit under new environment light maps or undergo simple material editing, making them applicable to fields such as film, gaming, immersive conferencing, and AR/VR.