TaoAvatar: Alibaba's High-Fidelity, Lightweight Full-Body Interactive Digital Human Project 207

What is TaoAvatar?

TaoAvatar is a high-fidelity, lightweight full-body interactive digital human technology developed based on 3D Gaussian Splatting (3DGS) technology. It creates personalized parametric clothed human templates and binds Gaussian points to the template to represent appearance, enabling realistic full-body digital human avatar generation. The core advantage of TaoAvatar lies in its ability to run in real-time on mobile devices and AR devices while maintaining high-resolution rendering and low storage requirements.

TaoAvatar Features

• High-Fidelity Rendering: TaoAvatar can generate realistic 3D full-body digital human avatars, supporting high-resolution rendering while maintaining low storage requirements.

• Real-Time Interaction: The technology can run in real-time on various mobile and AR devices, supporting high frame rate rendering at 90 FPS, suitable for high-definition stereoscopic devices such as Apple Vision Pro.

• Multi-Signal Driven: TaoAvatar can be driven by multiple signals, including facial expressions, gestures, and body poses, supporting rich interactive experiences.

• Lightweight Architecture: By "baking" complex non-rigid deformations into lightweight MLP networks and combining blend shapes to compensate for details, TaoAvatar significantly improves runtime efficiency while maintaining high-quality rendering.

TaoAvatar Applications

• E-commerce Livestreaming: TaoAvatar can be used to create realistic virtual streamers.

• Holographic Communication: In remote communication, TaoAvatar can generate realistic virtual avatars.

• Virtual Entertainment: In gaming and virtual reality applications, TaoAvatar can create personalized virtual characters.

• Mobile and AR Devices: TaoAvatar supports running on various mobile devices and AR devices, such as Android, MacBook, and Apple Vision Pro.

How to Use TaoAvatar

1. Data Preparation: Requires multi-view RGB video sequences and corresponding SMPLX registration information.

2. Template Creation: Create personalized parametric clothed human templates by binding Gaussian points to the SMPLX++ template.

3. Training and Optimization: Pre-train using the StyleUnet network, then bake non-rigid deformations into lightweight MLP networks through knowledge distillation techniques.

4. Deployment and Running: Deploy the optimized model to the target device, supporting real-time rendering and interaction.

Related Links

Project: https://pixelai-team.github.io/TaoAvatar/

Paper: https://arxiv.org/pdf/2503.17032

Model: https://huggingface.co/datasets/PixelAI-Team/TalkBody4D