Zero123: High-Quality 3D from a Single Image 139

Today, we are releasing the stable version of Zero123, our internally trained model for generating view-conditioned images. Compared to the previous state-of-the-art Zero123-XL, the stable Zero123 produces significantly improved results. This was achieved through 3 key innovations:

1. An improved training dataset, heavily filtered from Objaverse to retain only high-quality 3D objects, with the objects we render being more realistic than previous methods.

2. During both training and inference, we provide the model with estimated camera angles. This elevation conditioning enables it to make more informed and higher-quality predictions.

3. A precomputed dataset (precomputed latent variables) and an improved data loader that supports higher batch sizes, which, combined with the first innovation, results in a 40x improvement in training efficiency compared to Zero123-XL.

Zero123 Features:

The stable Zero123 can generate novel views of objects, demonstrating a 3D understanding of object appearance from various angles. Thanks to improvements in the training dataset and elevation conditioning, its quality is significantly improved over Zero1-to-3 or Zero123-XL.

The model is based on Stable Diffusion 1.5 and consumes the same amount of VRAM as SD1.5 to generate 1 new view. Generating 3D objects with Stable Zero123 requires more time and memory (24GB VRAM is recommended).

To enable open research in 3D object generation, we have improved the open-source code of Threestudio to support Zero123 and Stable Zero123. A simplified version of the Stable 3D pipeline is currently in private preview. From a technical perspective, this uses Score Distillation Sampling (SDS) to optimize a NeRF using the stable Zero123 model, from which we can later create a textured 3D mesh. This process can be adapted for text-to-3D generation by first using SDXL to generate a single image, and then using Stable Zero123 to generate a 3D object.

Zero123 Project: https://github.com/cvlab-columbia/zero123

Zero123 Demo: https://huggingface.co/spaces/cvlab/zero123-live

Zero123 Paper: https://arxiv.org/abs/2303.11328

The model is now available on Hugging Face, and researchers and non-commercial users can download and experiment with it.