SketchVideo: Controllable Video Generation and Editing from Sketches 76

What is SketchVideo?

SketchVideo is an open-source project for sketch-based controllable video generation and editing, proposed by researchers from the University of Chinese Academy of Sciences, the Hong Kong University of Science and Technology, and the Kling team at Kuaishou. It relies on three core technologies—intelligent motion prediction, multimodal control, and real-time rendering—to make animation creation effortless. The project has great value in fields such as marketing, education, and personal creative expression. It can capture audience attention, create resonance, and represents an emerging form of video.

SketchVideo Features

• Video Generation: Users only need to provide one or two sketch frames along with a text prompt, and SketchVideo can generate high-quality videos. For example, by drawing a simple animal outline sketch and adding the text description "running in the forest," a video of the animal running through the forest can be generated.

• Video Editing: Given a real video, users can specify editing regions and draw sketches to make local modifications. For instance, in a video of a person, modifying the hairstyle will cause the newly generated hairstyle to change naturally in sync with the person's head movements.

• Dynamic Control: Supports motion interpolation and extrapolation, allowing users to control the motion state of an object by drawing sketches at different keyframes.

• Detail Preservation: When editing videos, the details of unmodified regions are preserved, ensuring that the edited video looks visually natural and coherent.

• Efficient Generation: Employs memory optimization techniques to quickly produce high-quality videos. For example, the SketchGen model takes about 95 seconds to generate a single-frame video on an A100 GPU.

SketchVideo Technical Principles

• Sketch Conditioning Network: Built on a DiT video generation model, with a dedicated sketch conditioning network. This network contains several sketch control blocks that predict the residual features of skipped DiT blocks and inject control signals into features at different levels.

• Inter-Frame Attention Mechanism: By computing the relationships between the hidden features of all frames and the hidden features of the control frames, sketch conditions on keyframes are propagated to all video frames, achieving spatiotemporal propagation of sketch features.

• Video Insertion Module: In video editing tasks, this module analyzes the relationship between the input sketch and the original video, generating new content that is spatially and temporally aligned with the original video.

• Latent Fusion Technique: During inference, noise latent codes for the input video are generated based on DDIM inversion. These latent codes are replaced in unedited regions to preserve the details of the original video.

• Hybrid Training Strategy: Combines image and video data for training. Image and video data are first used to accelerate convergence and address the issue of limited video data; then video data is used to further optimize temporal coherence.

SketchVideo Advantages

• Lower Creative Barriers: Enables ordinary users to create videos easily through sketches and text, with no specialized knowledge or skills required, unleashing everyone's creative potential.

• Precise Detail Control: By predicting and injecting control signals into features at different levels through the sketch conditioning network, every detail—from object shape and position to the overall layout of the scene—can be carefully depicted and presented.

• Efficient and Convenient: Built on a DiT-based video generation model, it innovatively incorporates sketch control blocks and an inter-frame attention mechanism, ensuring high-quality output while achieving efficient memory optimization, rapidly generating videos and saving creation time.

SketchVideo Application Scenarios

• Film and Video Production: Can be used to quickly generate animated videos, or to add visual effects and modify scenes in film footage.

• Short Video Synthesis: Creators can effortlessly produce personalized short video content, such as hand-drawn animated stories and creative advertisements.

• Virtual Simulation: In virtual scene construction, dynamic virtual environments can be quickly generated through sketches for use in game development, virtual training, and other fields.

Project Information

Paper:https://arxiv.org/abs/2503.23284

Project Website:http://geometrylearning.com/SketchVideo

GitHub Repository:https://github.com/IGLICT/SketchVideo