Mnn3dAvatar: Alibaba's Open-Source 3D Digital Human Framework — A New Tool for Livestream Selling 67

What is Mnn3DAvatar?

Mnn3dAvatar is an open-source 3D digital human architecture released by Alibaba, built on the lightweight MNN framework. It can capture facial expressions in real time and map them onto 3D virtual characters, making the characters more vivid and lifelike. Even without complex modeling experience, developers can quickly create stunning 3D digital humans.

Technical Principles of Mnn3dAvatar

Mnn3dAvatar is built on the MNN framework, combining 3D Gaussian reconstruction with AI voiceprint-driven technologies. AI enables high-precision facial motion capture and real-time rendering, driving multiple modalities such as voice, expressions, and gestures.

Technical Highlights

• Real-time Facial Capture: Deep learning can accurately capture user expressions and quickly map them onto 3D characters, suitable for live streaming, virtual meetings, and other scenarios.

• Lightweight Deployment: Optimized by the MNN framework, Mnn3dAvatar does not require high-end hardware to run efficiently across numerous devices, greatly lowering the development barrier.

• Multimodal Support: In addition to facial capture, it also supports tasks such as text generation and image generation, providing developers with flexible interfaces.

• Open-Source Ecosystem: As part of Alibaba's open-source ecosystem, it includes rich APIs and tools, and can be seamlessly embedded into applications on platforms such as Android and iOS.

Application Scenarios of Mnn3dAvatar

• Live Streaming and Selling: Streamers can broadcast live as 3D virtual characters, adding fun and interactivity while reducing on-camera appearances, ideal for e-commerce platforms to create differentiated content.

• Virtual Display and Entertainment: Enables immersive visual experiences for virtual idols, game character animations, and virtual meeting scenarios.

• Education and Training: Educational institutions can use it to create virtual instructors, enhancing the interactivity and appeal of online teaching.

The release of Mnn3dAvatar has enriched the MNN ecosystem. Since going open source in 2019, MNN has been widely applied within Alibaba, covering more than 70 business scenarios including Taobao, Tmall, and DingTalk. At the same time, Alibaba has also open-sourced another 3D digital human generation model called LAM (Live Avatar Model), which supports the rapid generation of interactive 3D digital humans from a single image.

Open-Source Address:

https://github.com/alibaba/MNN/blob/master/apps/Android/Mnn3dAvatar/README_CN.md