LiveSketch_云技派,专注ai人工智能! 160

LiveSketch is an AI video generation tool that transforms static sketches into animations through text prompts, providing a simple and intuitive way to bring sketches to life. Whether used for storytelling, illustration, website design, or presentations, LiveSketch can breathe life into static sketch drawings, making them richer and more engaging.

With just a brief text prompt, LiveSketch can turn a few scribbled strokes into vivid animations, outputting lively SVG animation clips that are easy to edit and adjust, with motion and amplitude controllable through text.

Sketches are one of the most intuitive and versatile tools humans use to convey their ideas. Animated sketches open up another dimension for expressing ideas and are widely used for various purposes by designers.

Sketch animation is a laborious process that requires extensive experience and professional design skills. In this work, we propose an approach for LiveSketch that automatically adds motion to single-subject sketches by simply providing a text prompt indicating the desired motion (hence, "bringing it to life"). The output is a short animation provided in vector representation, making it easy to edit in LiveSketch.

Our approach does not require extensive training; instead, it leverages the motion priors of large pre-trained text-to-video diffusion models, using score distillation loss to guide the position of strokes. To facilitate natural and fluid movement, and to better preserve the appearance of the sketch, we model the learned motion through two components: the first controls small local deformations, and the second controls global affine transformations.

Surprisingly, we found that even models that struggle to generate sketch videos on their own can still serve as useful backbones for animating abstract representations through LiveSketch.

LiveSketch has significant limitations. First, currently, it can only control the generated results to a certain extent by simply modifying the motion-describing prompts. These modifications and controls are limited to the small motions the model can create, and the text prompts need to be semantically consistent with the initial sketch. When the subject contains multiple objects, it cannot distinguish between them.

In the future, technologies like LiveSketch may shine in fields such as artistic creation and children's education, significantly lowering the barrier to artistic creation and enhancing the fun and interactivity of education.