StoryMem: ByteDance's Open-Source AI Multi-Shot Long Video Generator 95

An AI long-video framework open-sourced by ByteDance and NTU. It turns existing single-shot video diffusion models into a tool for multi-shot storytelling, using a visual memory bank to produce coherent 40–60 second stories across 8–12 shots with consistent characters and scene styles.

#Ai工具箱 #Ai开源项目

Visit StoryMem: ByteDance's Open-Source AI Multi-Shot Long Video Generator

StoryMem: ByteDance's Open-Source AI Multi-Shot Long Video GeneratorIntroduction

StoryMem is an open-source AI multi-shot long-video generation tool from ByteDance and Nanyang Technological University, primarily addressing the problem of inconsistent character appearance and visual style across multiple shots in multi-shot videos. It transforms the current single-shot video diffusion model into a tool capable of generating multi-shot story films, relying on a visual memory bank to produce continuous stories of 40 to 60 seconds with 8 to 12 shots, where character appearances and scene styles remain consistent.

In terms of technical implementation, StoryMem builds upon the Wan2.2 model and adds memory processing capabilities through fine-tuning. During generation, the first shot is created as the initial memory anchor, and each subsequent shot references the previous memory content while continuously updating the memory bank, thereby maintaining overall video coherence.

StoryMem Features

Memory to Video (M2V): Uses LoRA to inject previous key frames into the current noise, and with simple tuning, ensures different shots look consistent.
Dynamic Memory Bank: Retains up to 10 key frames, first selecting them by meaning using CLIP, then picking the visually appealing ones using HPSv3, automatically discarding poor-quality frames.
Three Extension Methods:

MI2V: Uses the first-frame image as a condition for smoother transitions;
MM2V: Uses the motion from the first five frames as a condition for more natural movement;
MR2V: Allows uploading a custom image as the initial memory to lock in the protagonist's face.

Output Performance: Compared with current methods, consistency is improved by 29%, while the original model's high image quality and prompt adherence are preserved.

StoryMem Use Cases

Marketing Ads: Input a script and quickly generate multiple dynamic storyboards for A/B testing.
Film and TV Pre-production: Turn text-based storyboards directly into visual scenes, saving costs on early-stage ideation.
Short Videos / Independent Creation: Anyone can create 1-minute cinematic-quality short films with no barrier to entry.
Education and Corporate Training: Historical recreations, science explanations, and product demos can all be generated with a single click.

How to Use StoryMem

Clone the repository:
git clone https://github.com/Kevin-thu/StoryMem
Install dependencies:
pip install -r requirements.txt
Prepare the script: Write multi-shot prompts according to the examples, or use the 300 story templates provided by ST-Bench.
Start generation:
python inference.py --prompt "Your story script" --length 60 --memory_size 10
Optional features:
Add --mi2v for seamless transitions
Add --ref_image path/to/face.jpg --mr2v to customize the protagonist