X-UniMotion: A Video Model That Reproduces Detailed Hand Movements 89

X-UniMotion is a video model capable of reproducing hand movements. Given a reference person and an action video, it enables the reference person to accurately perform the same movements, making it suitable for complex hand manipulations. Its technical advantage lies in highly restoring hand details with accurate and fluid motions.

Features

• End-to-End Training: X-UniMotion is trained end-to-end to learn latent representations of full-body motion and leverages a DiT network to generate realistic videos.

• Fine Motion Capture: Accurately captures detailed local movements such as those of the hands and face.

• Identity-Agnostic Representation: By employing spatial and color augmentation, it separates identity cues from motion latents, achieving an identity-agnostic motion representation.

• High-Quality Animation Generation: The generated animations are realistic and motion-consistent, highly similar to the reference image.

X-UniMotion Technology

• Image Encoder: Extracts latent motion descriptors of full-body joints as well as local descriptors of the hands and face from the reference image.

• ViT Decoder: Re-positions the motion descriptors onto the body structure of the reference subject, outputting spatial motion guidance.

• DiT Network: Combines spatial motion guidance, noisy video latents, and reference image latents to generate the final animation.

• Cross-Attention Layer: Injects facial motion latents into the DiT network to achieve expression control.

Application Scenarios

• Film and TV Production: Used to generate character motion animations, saving costs and improving special effects quality.

• Animation Production: Quickly generates high-quality animation clips, improving production efficiency.

• Game Development: Makes in-game character movements more natural and realistic, enhancing immersion.

• Online Education: Generates fine motion demonstration videos to aid in teaching courses such as handicrafts and instrument performance.

• Virtual Laboratories: Simulates experimental operations to assist students in learning.

• Language Learning: Generates teaching videos of language gestures such as sign language to help students learn.

• Rehabilitation Training: Generates rehabilitation motion videos to assist patients in hand and facial rehabilitation training.

• Surgical Simulation: Generates detailed surgical operation videos to help doctors master surgical techniques.

• Psychotherapy: Generates expression and motion videos to guide patients in learning and expressing emotions.

• Virtual Assistants: Generates motion animations for virtual assistants to enhance the naturalness of interactions.

• VR/AR: Generates motion animations for virtual characters to enhance the interactive experience in virtual environments.

• Gesture Recognition: Generates fine hand motion samples used to train gesture recognition models.

How to Use

1. Input Data: A reference image and an action video are required.

2. Extract Motion Descriptors: Extract latent motion descriptors of full-body joints as well as local descriptors of the hands and face from the reference image.

3. Motion Re-positioning: Re-position the motion descriptors onto the body structure of the reference subject.

4. Animation Generation: Combine motion guidance, noisy video latents, and reference image latents, and input them into the DiT model to generate the animation.

5. Expression Control: Inject facial motion latents into the DiT network to achieve expression control.