DiffPortrait360: Generate Consistent 360° Head Views from a Single Portrait Image 123

What is DiffPortrait360?

In game, film, and animation production, CG characters are at the core of content creation, requiring a balance between realism and stylization. Traditional multi-view stereo systems and 3D scanners are often used to create realistic human characters, but stylized characters still need to be modeled from scratch. The emergence of DiffPortrait360 provides a new approach for efficiently generating stylized characters.

DiffPortrait360 is a 3D head view generation technique capable of producing 360-degree full-view head images from a single portrait photo. By combining a pre-trained latent diffusion model with several auxiliary modules, this technique achieves high-quality, stylized 3D head view synthesis. DiffPortrait360 supports various complex inputs, such as different hairstyles, poses, and facial expressions, and the generated models can be widely applied to immersive remote interaction, personalized content creation, and game and animation production.

Method Principle

• Architecture: DiffPortrait360 uses a pre-trained latent diffusion model (LDM) as its rendering backbone and introduces three auxiliary trainable modules for controlling dual appearance R, camera control C, and view-consistent U-Nets. Among them, R extracts appearance information from front and back images, while C derives camera poses and uses an off-the-shelf 3D GAN for rendering.

• Key Innovation: This method is based on the DiffPortrait3D framework, incorporating a customized ControlNet for generating back-of-head details, along with a dual appearance module to ensure global front-back consistency. By training on continuous view sequences and integrating back-view reference images, robust local continuous view synthesis is achieved.

• Training Strategy: A continuous sampling training strategy is adopted to better maintain the continuity of camera trajectories and enhance attention to continuity between frames, ensuring that appearance information remains consistent as rotation angles change.

Advantages and Features

• Wide Applicability: Capable of generating 360-degree head views in human, stylized, and anthropomorphic forms, such as those with accessories like glasses and hats.

• High Quality: Can generate high-quality Neural Radiance Fields (NeRFs) for real-time free-viewpoint rendering, outperforming existing state-of-the-art methods in object synthesis and 360-degree head generation.

• Strong Robustness: Handles various complex inputs, including complex hairstyles, different head poses, rich facial expressions, and detailed elements such as tongues.

Application Prospects

• Immersive Telepresence Applications: Provides users with realistic experiences by generating high-quality 360-degree head views.

• Personalized Content Creation: Quickly generates 3D head models with different viewpoints based on a single portrait image provided by the user.

• Game and Animation Production: Efficiently generates stylized character head models, improving the efficiency and quality of game and animation production.

Github Address:https://github.com/FreedomGu/DiffPortrait360