SceneSplat: An Open-Source 3D Scene Understanding Tool by Stanford and Google 72

小编 · 2025-09-11

SceneSplat is an open-source 3D scene understanding tool jointly developed by Stanford University and Google. It enables 3D Gaussian models to follow natural language instructions, turning text prompts directly into 3D scenes.

SceneSplat: An Open-Source 3D Scene Understanding Tool by Stanford and Google

SceneSplat is an open-source 3D scene understanding tool jointly developed by Stanford University and Google. It mainly enables 3D Gaussian models to understand human language instructions, achieving conversion from text to 3D scenes.

SceneSplat: An open-source 3D scene understanding tool jointly developed by Stanford University and Google

SceneSplat Technical Principles

End-to-end vision-language pre-training architecture: By simultaneously learning language descriptions and their corresponding 3D scenes, the model can understand natural language instructions and convert them into 3D scenes. For example, given the input "place the bookshelf against the wall by the window," the model can automatically identify key information and accurately position it in 3D space.
Cross-modal feature alignment: Combines 3D Gaussian representations with vision-language models. Through multi-scale feature extraction and cross-modal attention mechanisms, it enhances the ability to understand complex language instructions.
Self-supervised learning scheme: Capable of learning more 3D features from unlabeled scenes.

SceneSplat Datasets

SceneSplat - 7K: The first large-scale 3DGS dataset for indoor scenes, containing 7,916 scenes sourced from seven existing datasets including ScanNet and Matterport3D. Generating this dataset required approximately 150 days of computation on an L4 GPU.
SceneSplat - 49K: Contains indoor and outdoor scenes from multiple sources, and is currently the most comprehensive open-source dataset, featuring complex and high-quality complete scene-level 3DGS reconstructions that can advance the development of generalizable 3DGS scene understanding.

SceneSplat Advantages and Features

Integrated semantic understanding: The first to implement semantic understanding applications in 3D Gaussian models, breaking through the limitations of traditional 3D graphics systems in processing semantics.
Natively runs on 3DGS: The first large-scale 3D indoor scene understanding method that can run directly on 3DGS, avoiding the information loss and computational latency caused by data conversion in traditional methods.
Efficient generalization capability: Achieves state-of-the-art results in open-vocabulary and semantic segmentation tests.

SceneSplat Application Scenarios

AR/VR: Capable of processing 3D point clouds in real time, answering questions in natural language with precise localization, making AR/VR interactions more natural.
Autonomous driving: Helps autonomous driving systems better understand complex 3D environments, improving decision-making accuracy and safety.
Intelligent robots: Enables robots to understand human instructions and complete tasks with greater precision.
3D game development: Rapidly generates complex 3D environments through natural language instructions, enhancing development efficiency and the dynamic generation capabilities of game worlds.