Tora: AI Video Generation Framework by Alibaba 144

Alibaba's Tora video is an AI video generation framework launched by Alibaba. Based on trajectory-oriented Diffusion Transformer (DiT) technology, it integrates text, visual, and trajectory conditions to generate high-quality dynamic videos that conform to the physical world.

Main features of Tora:

Trajectory-oriented video generation: Tora is capable of generating videos based on given trajectories while maintaining consistency and accuracy in motion. The Trajectory Extractor (TE) converts input trajectories into hierarchical spatiotemporal motion patches that align with the latent space of the video content.

Multimodal input integration: Tora supports simultaneous input of text, image, and trajectory conditions, enabling multidimensional control over video content. This means users can generate videos through textual descriptions, images, or object movement paths.

High-resolution video output: Tora can generate videos up to 204 frames long at a resolution of 720p, ensuring video quality. This ensures that the generated videos are not only visually clear but also feature smooth motion.

Physical world dynamic simulation: The motion effects in videos generated by Tora closely resemble real-world physical dynamics, such as gravity and kinetics. Through the Spatial-Temporal DiT, Tora processes video data, enabling the model to understand and generate videos with coherent motion.

Flexible content creation: Tora supports video generation with varying durations, aspect ratios, and resolutions to meet diverse creative needs. Whether it's short videos or long videos, standard aspect ratios or custom aspect ratios, Tora can adapt.

Motion-guidance Fuser (MGF): This component is responsible for integrating the spatiotemporal motion patches generated by the Trajectory Extractor into the DiT model, ensuring that the generated video content follows the predetermined trajectory and dynamics. MGF integrates motion patches into DiT blocks through adaptive normalization layers, providing precise control over video content dynamics.

Two-stage training strategy: To better understand and generate motion, Tora learns through a two-stage training process. First, training is performed using dense optical flow, followed by fine-tuning with sparse trajectories to improve the model's flexibility and accuracy in motion control.

Data preprocessing: Before training, Tora needs to process video data, such as splitting long videos into short clips based on scene detection, and then selecting video clips suitable for training based on aesthetic scores and motion segmentation results.

Main functional modules of Tora

Trajectory Extractor (TE): Converts input trajectories into hierarchical spatiotemporal motion patches that align with the latent space of the video content.

Spatial-Temporal DiT: Combines spatial and temporal self-attention mechanisms to process video data, enabling the model to understand and generate videos with coherent motion.

Motion-guidance Fuser (MGF): Responsible for integrating the spatiotemporal motion patches generated by the Trajectory Extractor into the DiT model, ensuring that the generated video content follows the predetermined trajectory and dynamics.

Tora application scenarios:

Film and television production: In film and television production, Tora can be used to generate special effects scenes in movies, TV series, or short films. By using trajectory control to generate complex dynamic visuals, it reduces actual filming costs and time.

Animation creation: Automatically generate animation sequences based on scripts, providing animators with preliminary dynamic sketches to accelerate the creative process.

Virtual Reality (VR) and Augmented Reality (AR): Generate dynamic environments that interact with users, providing realistic visual effects for VR and AR applications.

Game development: Quickly generate game environments and character animations, improving the efficiency of game design.

How to access Tora:

Tora project official website: https://ali-videoai.github.io/tora_video/

Tora GitHub repository: https://github.com/ali-videoai/Tora

Tora arXiv technical paper: https://arxiv.org/pdf/2407.21705

Overall, the quality of videos generated by Tora is extremely high, capable of producing high-resolution videos ranging from 10 to 60 seconds, and performing excellently across different resolutions and aspect ratios. Tora's videos are not only smooth but also capable of finely simulating motion in the physical world. Tora's trajectory control features have also received high praise from users.