MimicMotion: Motion Video Generation Project 130

A joint project by Tencent and Shanghai Jiao Tong University that generates high-quality human motion videos with confidence-aware pose guidance. MimicMotion is a framework capable of producing videos of any length guided by arbitrary motion sequences.

#Ai工具箱 #Ai开源项目

Visit MimicMotion: Motion Video Generation Project

MimicMotion: Motion Video Generation ProjectIntroduction

Made by Tencent and Shanghai Jiao Tong University, MimicMotion is a video generation framework that can generate high-quality videos of any length with any motion guidance. It can produce detail-rich, lifelike videos of human motion, generate video content of any length, and achieve precise control over the video content, including motion, pose, and video style.

Similar to Alibaba's Tongyi Dance King, but the results appear to be much better than Alibaba's Tongyi Dance King. MimicMotion supports facial features and lip-sync, so it can be used not only for dance videos but also for creating digital humans.

In recent years, generative artificial intelligence has made significant progress in the field of image generation, giving rise to a variety of applications. However, video generation still faces considerable challenges in terms of controllability, video length, detail richness, and other aspects, which hinders the application and popularization of the technology. In this work, we propose a controllable video generation framework called MimicMotion, which can generate high-quality videos of any length under any motion guidance. Compared with previous methods, our approach

has several highlights.

First, through confidence-aware pose guidance, temporal smoothness can be achieved, which allows the model's coherence and fluidity to be enhanced through large-scale training data.

Second, a regional loss based on pose confidence significantly reduces image distortion, effectively minimizing warping and deformation in generated images.

Finally, to generate smooth long videos, a progressive latent fusion strategy is proposed. In this way, videos of any length can be generated while keeping computational resource consumption within an acceptable range.

Through extensive experiments and user studies, MimicMotion has achieved significant improvements over Alibaba's Tongyi Dance King method in multiple aspects.