PrimitiveAnything: A New 3D Shape Generation Framework from Tencent AIPD & Tsinghua University 80

What is PrimitiveAnything?

PrimitiveAnything is a novel 3D shape generation framework jointly developed by Tencent's Artificial Intelligence Platform Department (AIPD) and Tsinghua University. It breaks down complex 3D shapes into simple parts and reassembles them into complete shapes. This tool generates high-quality models, supports multiple ways of creation, and is convenient for storage and editing. It employs a clear parametric approach and an autoregressive architecture, ensuring a stable and accurate training process. It supports generating 3D content from text or images, and can flexibly incorporate new part types to accommodate various representations.

Main Features of PrimitiveAnything

• High-quality 3D primitive assembly generation: Capable of generating high-quality 3D primitive assemblies that are geometrically faithful to the original models and align with human intuition about shapes.

• Diverse 3D content creation: Supports generating 3D content from text or image conditions, offering users flexible creation methods.

• Efficient storage and editing: Thanks to the primitive representation, the generated 3D models are more storage-efficient and easy to edit and adjust.

• Autoregressive transformer architecture: Generates 3D primitives frame by frame through an autoregressive transformer, capable of handling primitive sequences of varying lengths and easily extensible to new primitive types.

• Unambiguous parametric scheme: By eliminating ambiguity in parameterization, it ensures stability and accuracy throughout the training and generation processes.

• Geometric fidelity and semantic consistency: Maintains high geometric fidelity during generation, producing semantic decomposition results that align with human cognition.

• Modular design: The framework's modular design supports seamless integration of new primitive types without architectural modifications, adapting to different primitive representations.

Technical Principles

• Unambiguous parametric scheme: Uses multiple types of primitives (such as cuboids, ellipsoidal cylinders, and ellipsoids) to jointly represent 3D shapes under a unified parametric scheme. To address the inherent ambiguity in parameterization, the research team developed a comprehensive set of rules. By analyzing the symmetry of primitives, they select the representation with the smallest L1 norm of rotation parameters as the unique representation, ensuring stability and accuracy throughout the training process.

• Autoregressive transformer architecture: Adopts a decoder-based Transformer architecture that generates variable-length primitive sequences based on shape features. First, a point cloud encoder extracts the feature representation of the 3D shape, then the autoregressive Transformer model receives these shape features along with previously generated primitive information to predict the features of the next primitive. To model dependencies between primitive attributes, a cascaded decoder is used to sequentially predict the type, position, rotation, and scale of each primitive, capturing the natural correlations between primitive attributes—consistent with human assembly logic.

• Autoregressive generation pipeline: Reimagines the entire primitive abstraction process as a sequence generation task. The model takes a point cloud as input condition, then autoregressively generates a primitive sequence until an end token is predicted. During training, cross-entropy loss, Chamfer distance (for reconstruction accuracy), and Gumbel-Softmax (for differentiable sampling) are combined until the sequence end token is generated.

Performance

On a test set covering 12 categories including furniture, vehicles, and architecture, PrimitiveAnything demonstrates remarkable performance. For example, on the Chamfer Distance (CD) metric, traditional methods score 0.82, the existing SOTA scores 0.45, while PrimitiveAnything achieves 0.21. On the voxel IoU metric, traditional methods score 68%, the existing SOTA scores 75%, and PrimitiveAnything reaches 89%. On the Segmentation Consistency (SC) metric, traditional methods score 0.63, the existing SOTA scores 0.71, and PrimitiveAnything scores 0.92. On storage compression ratio, traditional methods achieve 1x, the existing SOTA achieves 3x, and PrimitiveAnything reaches 15x. Even more noteworthy is its cross-modal generation capability. In text-to-3D generation tasks, the system can accurately understand complex descriptions such as "a wooden dining chair with an arched backrest" and generate semantically consistent primitive combinations.

PrimitiveAnything Application Scenarios

• 3D modeling and design: Quickly generates the "geometric skeleton" of complex 3D models, allowing designers to focus on detail refinement, thereby improving work efficiency and design speed.

• Game asset generation: Game designers can use it to rapidly generate various game scenes and character models, boosting development efficiency. Players can also create new characters or items by simply assembling geometric blocks, with AI automatically optimizing and adjusting them for seamless integration into the physics engine.

• User-Generated Content (UGC): The framework supports generating 3D content from text or image input, allowing users to easily edit the generated results, opening up new possibilities for in-game UGC.

• Virtual Reality (VR) and Augmented Reality (AR): In VR and AR environments, realistic 3D objects can be quickly generated to enhance the user experience.

Project Links

Project Website: https://primitiveanything.github.io/

Github Repository: https://github.com/PrimitiveAnything/PrimitiveAnything-

HuggingFace Model Library: https://huggingface.co/hyz317/PrimitiveAnything

arXiv Technical Paper: https://arxiv.org/pdf/2505.04622