TaleStreamAI: Automated Conversion of Novel Text to Short Videos, Ideal for Novel Promotion 102

What is TaleStreamai?

TaleStreamAI is an open-source project that enables the automated conversion of novel text into short videos.

TaleStreamAI Technical Highlights

• Smart Storyboarding: Leverages the Gemini model to break down novel plots, splitting long chapters into multiple sections to avoid processing excessive content. It plans the text and shot script for each storyboard, making the story presentation more coherent.

• Prompt Optimization: Uses the DeepSeek-V3 model to refine drawing prompts, ensuring the text descriptions in the storyboards more precisely match AI drawing requirements and improving the quality and relevance of generated images.

• Precise Subtitles: Applies silence detection and voice analysis technology in combination with OpenAI's Whisper model for speech recognition, generating accurate SRT subtitles and ensuring perfect synchronization between subtitles and audio.

• Video Effects Processing: Utilizes image movement algorithms developed with the MoviePy library to achieve effects like panning, automatically add subtitles, and supports custom video resolutions and aspect ratios, producing professional-grade videos.

• Resource Management Optimization: Ensures long-term stable program operation through garbage collection, temporary file cleanup, memory optimization, and other techniques, making it less prone to errors even when handling large volumes of video tasks. At the same time, the modular architecture, multi-threading, and error recovery mechanisms make the system more flexible and efficient.

TaleStreamAI Workflow

• Content Scraping: Fetches specified novels from Qidian website, parses web pages using the BeautifulSoup library, extracts chapter content, and saves them as individual text files.

• Storyboard Creation: The Gemini large language model breaks down novel content into multiple scenes, with each storyboard containing content, Chinese and English shot descriptions, and other information.

• Prompt Refinement: The DeepSeek-V3 model further optimizes storyboard prompts into precise instructions suitable for AI drawing.

• Image Generation: Sends optimized prompts to the Stable Diffusion API (free) to obtain high-quality images.

• Audio Synthesis: Uses the CosyVoice2 speech synthesis model to convert storyboard text into natural-sounding voice narration, supporting multi-threading and API key rotation, outputting audio in MP3 format.

• Subtitle Generation: Performs speech recognition on audio using the Whisper model and generates precise SRT subtitle files through enhanced segmentation techniques.

• Single-Shot Composition: Integrates images, audio, and subtitles, adding video effects using the MoviePy library to create individual video clips.

• Final Production: Uses FFmpeg's concat feature to merge all video clips in order while optimizing memory usage, outputting the complete video.

TaleStreamAI Application Scenarios

• Novel Promotion: Authors can turn excerpts of their works into videos to attract more reader attention.

• Content Monetization: Convert public domain or original novels into short videos and publish them across platforms to earn traffic revenue.

• Educational Support: Transform literary works into visual videos to help students understand classic literature.

• Media Creation: Add visuals to radio dramas and audiobooks to enrich content presentation formats.

Github:https://github.com/Mubashir-414/TaleStreamAI