LucidDreamer: Domain-Free 3D Gaussian Splatting Scene Generation 122

With the widespread use of VR devices and content, the demand for 3D scene generation technology is becoming increasingly common. However, existing 3D scene generation models restrict the target scene to a specific domain, primarily because the 3D scanning datasets they use are far from the real world. To address this limitation, we propose LucidDreamer, a domain-free scene generation pipeline that fully leverages the powerful capabilities of existing large-scale diffusion-based generative models.

LucidDreamer project page:https://luciddreamer-cvlab.github.io/

Our LucidDreamer has two alternating steps: dreaming and alignment.

First, to generate multi-view consistent images from input, we set the point cloud as the geometric reference each time an image is generated. Specifically, we project a portion of the point cloud to the desired view and provide the projection as guidance for inpainting using the generative model. The painted images are lifted into 3D space using the estimated depth map, thereby composing a new point.

Second, to aggregate new points into the 3D scene, we propose an alignment algorithm that harmoniously integrates the various parts of the newly generated 3D scene. The resulting 3D scene serves as the initial point for optimizing Gaussian Splatting. Compared to previous 3D scene generation methods, the Gaussian Splatting generated by LucidDreamer is highly detailed and unrestricted in the domain of the target scene.

LucidDreamer can accept a series of text prompts to generate scenes, enabling fine-grained control.

Quantitative comparison of scenes generated by images produced with CLIP-based Stable Diffusion. We use CLIP-Score and CLIP-IQA with RGBD2 for quantitative comparison of results. For CLIP-IQA, we use the criteria of quality, color vividness, and clarity. LucidDreamer shows dominant results across all metrics.

Metrics for reconstructing Gaussian Splatting based on the source of the initial SfM points. We use COLMAP and compare the reconstruction results. Using our point cloud consistently shows better reconstruction metrics.