MMStoryAgent: An Open-Source Multimodal AI Story Generation System from SJTU and Alibaba 189

What is MM-StoryAgent?

MM-StoryAgent is a multimodal story generation system open-sourced by Shanghai Jiao Tong University and Alibaba. It adopts a multi-agent paradigm and can generate immersive audio storybook videos. It integrates multiple elements such as text, images, and audio, and leverages large language models and some specialized tools to automatically generate story videos containing text, images, and multi-channel audio (narration, sound effects, background music), which can be used for rapid creation of children's stories.

Features of MM-StoryAgent

• High-quality story creation: Creates stories through a multi-agent, multi-stage process based on the input story settings. It adopts a top-down approach to outline drafts and a bottom-up approach to generate materials, solving issues such as manual intervention, scene arrangement, and narrative consistency.

• Multimodal content generation: Combines text, images, speech, music, and other modalities to generate immersive story videos. Tasks are decomposed and assigned to different specialized "agents," such as those responsible for story design, storyboard generation, and video creation, simulating a professional production workflow.

• Support for customized workflows: Provides reliable and customizable workflows. Users can customize agent behavior through configuration files and other methods, and define their own specialized tools to improve the generation quality of each component.

• Enhanced character consistency: The generation process emphasizes consistency of characters and themes. It adopts a customized image-to-video method and a new storyboard generation process to ensure subject consistency across shots.

• Automated and flexible: Leverages large language models and specialized tools to automatically convert simple prompts into multimodal digital stories, generating coherent content without the need for reference videos.

• Easy evaluation and optimization: Provides a list of story themes and evaluation criteria, making it convenient for users to assess the quality of generated stories and make improvements.

Core Technologies of MM-StoryAgent

• Multi-agent framework operating mechanism: Creates stories through a multi-stage writing process and multi-agent collaboration, integrating visual, auditory, and narrative elements to deliver a full sensory experience.

• Customized image-to-video method: Introduces LoRA-BE, a customized image-to-video method that enhances temporal consistency within shots and improves video visual quality.

How to Use MM-StoryAgent

1. Obtain the source code: Visit the MM_StoryAgent GitHub repository (see the link below) to download the source code.

2. Install dependencies: Follow the project documentation to install necessary software packages such as Python libraries.

3. Configure the environment: Set up the development environment, ensure that dependencies are installed correctly, and configure some environment variables if necessary.

4. Prepare input data: Prepare data such as story settings, character descriptions, and scene information for story creation.

5. Run the generation process: Use the provided scripts or command-line tools to start the generation process. The story can be customized by specifying input data and parameters.

6. View evaluation results: Once the story is generated in video form, view and evaluate the quality and effects, and make adjustments as needed.

7. Optimize and iterate: Adjust input data or parameters based on the evaluation results, and re-run the generation process to optimize story content and expressiveness.

GitHub Repository:https://github.com/X-PLUG/MM_StoryAgent