HSMR: A New Method for Reconstructing 3D Human Skeleton and Mesh Models from a Single Image 147

小编 · 2025-04-01

HSMR is a computer vision and deep learning technique that recovers human skeleton structure and 3D mesh models from monocular or multi-view images. This technology has broad applications in virtual reality, augmented reality, game development, and motion analysis.

HSMR: A New Method for Reconstructing 3D Human Skeleton and Mesh Models from a Single Image

What is HSMR?

HSMR is a computer vision and deep learning technique that can recover the human body's skeletal structure and 3D mesh model from monocular or multi-view images. This technology has a wide range of applications in virtual reality, augmented reality, game development, and motion analysis.

What is HSMR?.webp

Method

Model Architecture: HSMR adopts a Transformer-based architecture. The input is a single human body image, and the output is the pose parameters q and shape parameters β of the SKEL model.
Data Generation and Training:
Due to the lack of sufficient training data, the authors constructed a pipeline for generating pseudo-realistic data and progressively improved these pseudo-labels through iterative optimization.
During training, the estimation results of HSMR are optimized to align with real 2D keypoints (SKELify), and the optimized parameters are used as supervision targets for subsequent training.
Biomechanical Accuracy: The SKEL model has biomechanically reasonable degrees of freedom, enabling more realistic estimation of joint rotations and avoiding the violation of joint angle limits commonly seen in traditional methods.

HSMR: A New Method for Reconstructing Human 3D Skeleton and Mesh Model from a Single Image.webp

Features

End-to-End Single Image Reconstruction: HSMR can directly recover human body skeleton parameters from a single image, making it the first method to achieve such functionality.
Biomechanical Accuracy: It adopts the SKEL parametric human body model, which is based on a biomechanically accurate skeleton design. This allows it to provide joint rotation estimates that better conform to human physiological structure, avoiding the unnatural rotations and biomechanical inaccuracies commonly seen in traditional methods.
Pseudo-Realistic Data Generation and Iterative Optimization Training: To address the issue of insufficient training data, HSMR constructs a pipeline for generating pseudo-realistic data and progressively improves these pseudo-labels through iterative optimization to enhance the model's training accuracy.
Transformer Architecture: It uses a Transformer-based architecture to process input images and estimate the pose parameters and shape parameters of the SKEL model, enabling better capture of complex features and human pose information in images.

Application Scenarios

Virtual Reality and Augmented Reality: Provides more realistic interactive experiences for games, training, and simulation training.
Sports Analysis and Biomechanics: Analyzes athletes' movements to optimize training methods and equipment design.
Medical Diagnosis and Rehabilitation: Assists doctors in posture assessment and disease diagnosis, and helps patients with rehabilitation training.

Experimental Results

Performance Comparison: HSMR is competitive with state-of-the-art methods on standard benchmarks and significantly outperforms other methods under extreme 3D pose and viewpoint settings.
Visualization Results: The reconstruction effects of HSMR are demonstrated through visualizations from camera viewpoints and side viewpoints, proving its robustness under different viewpoints.