Hi3DGen: A Framework for Generating High-Fidelity 3D Geometry from 2D Images 131

小编 · 2025-04-16

Hi3DGen is a framework for generating high-fidelity 3D geometry from 2D images. Using normal maps as an intermediate representation, it combines an image-to-normal estimator and a normal-to-geometry learner to improve generation fidelity. It also builds a high-quality synthetic dataset, DetailVerse, to support training. Experiments show that Hi3DGen excels at producing rich geometric details, making it suitable for VR, game development, and more.

Hi3DGen: A Framework for Generating High-Fidelity 3D Geometry from 2D Images

What is Hi3DGen?

Hi3DGen is a framework for generating high-fidelity 3D geometric models from 2D images. It uses normal maps as an intermediate representation, combining an image-to-normal estimator and a normal-to-geometry learning approach to improve generation fidelity. In addition, it builds a high-quality synthetic dataset called DetailVerse to support training. Experiments show that Hi3DGen performs very well in generating rich geometric details, making it suitable for virtual reality, game development, and other scenarios.

Hi3DGen：A Framework for Generating High-Fidelity 3D Geometric Models from 2D Images.webp

Hi3DGen Features

High-Fidelity 3D Generation: By using normal maps as an intermediate representation and combining advanced estimators and learning methods, it can generate high-fidelity, detail-rich 3D geometric models from 2D images.
Strong Generalization Capability: A dual-stream training strategy is adopted to decouple the low-frequency and high-frequency modes of images, enabling the model to stably output high-quality results under different input conditions.
Efficient Geometric Supervision: Through the normal-regularized latent diffusion learning method, clear supervision signals are provided for 3D geometry generation, significantly improving generation accuracy.
High-Quality Data Support: Equipped with the DetailVerse synthetic dataset, which contains complex geometric structures and rich details, providing sufficient data support for model training.

Hi3DGen Framework Composition

Image-to-Normal Estimator (NiRNE): By injecting noise and using dual-stream training, it decouples the low-frequency and high-frequency modes of images, thereby achieving highly generalizable, stable, and clear normal estimation.
Normal-to-Geometry Learning Method (NoRLD): Adopts a normal-regularized latent diffusion learning method that provides clear 3D geometry supervision during training, significantly improving generation fidelity.
3D Data Synthesis Pipeline: A high-quality synthetic 3D dataset named DetailVerse is built to support framework training.

Hi3DGen：A Framework for Generating High-Fidelity 3D Geometric Models from 2D Images.webp

Hi3DGen Technical Advantages

Using Normal Maps as an Intermediate Representation: Normal maps encode surface orientation information, which can effectively bridge the domain gap between synthetic training data and real-world applications, and provide clearer geometric cues for geometry learning.
Dual-Stream Training Strategy: By decoupling the learning of low-frequency and high-frequency image modes, NiRNE can generate clearer normal maps while maintaining stability.
High-Quality Synthetic Dataset: The DetailVerse dataset contains complex geometric structures and rich surface details, helping to train more accurate models.

Hi3DGen Application Scenarios

Virtual Reality (VR) and Augmented Reality (AR): Quickly generate realistic 3D scenes and object models from 2D images.
Game Development: Used to quickly generate 3D models in games from concept art or existing 2D assets.
Architectural Design and Visualization: Generate 3D building models from architectural drawings or photos, helping designers quickly produce design visualizations.
Film Special Effects and Animation Production: Quickly generate complex 3D scenes and prop models, saving modeling time.
Cultural Heritage Preservation: Generate high-fidelity 3D models from photos of cultural relics for digital preservation and display.