Linly-Dubbing: AI-Powered Multilingual Video Dubbing & Translation Tool 124

Linly-Dubbing is an open-source multilingual aidubbing and videotranslationtool. Linly-Dubbing can automatically translate videos into other languages and generate subtitles, clone the voice of the speaker in the video and perform automatic dubbing, and carry out lip-syncing.

Main Features of Linly-Dubbing:

• Automatic video download: supports downloading videos from websites such as YouTube

• Multilingual support: supports dubbing and subtitle translation in Chinese and a variety of languages.

• AI speech recognition: accurate speech recognition, voice-to-text conversion, and speaker identification.

• LLM translation: combined with leading large language models (such as GPT), it can translate quickly and accurately, ensuring professionalism and naturalness in translation.

• Voice cloning: through voice cloning technology, it generates a voice highly similar to the original video's dubbing, maintaining consistency in emotion and tone.

• Lip-syncing: by maintaining lip-syncing, the dubbing can be highly consistent with the video picture, improving the video's realism and interactivity.

• Flexible upload and translation: users can upload videos and select translation languages and standards to ensure personalization and flexibility.

Technical Details of Linly-Dubbing

Speech Recognition

WhisperX: an extension of OpenAI Whisper's speech recognition system, which can transcribe speech content into text, align precisely with video frames, generate subtitle files with timestamps, and support multi-speaker identification.

FunASR: a comprehensive speech recognition toolkit that provides speech recognition, voice activity detection, punctuation restoration, and other functions, with special optimization for Chinese speech.

Speech Synthesis

It integrates various advanced speech synthesis tools such as Edge TTS, XTTS, and CosyVoice.

Edge TTS: a high-quality text-to-speech conversion service provided by Microsoft, supporting multiple languages and voice styles, generating natural and smooth speech output.

XTTS: an advanced deep learning text-to-speech toolkit provided by Coqui, focusing on voice cloning and multilingual speech synthesis, which can achieve voice cloning through short audio clips and generate realistic speech output.

CosyVoice: a multilingual speech understanding and synthesis model developed by Alibaba's Tongyi Lab, supporting high-quality speech synthesis and cross-lingual voice cloning in multiple languages.

Subtitle Translation

It uses the OpenAI API and Qwen model for multilingual subtitle translation.

OpenAI API: uses OpenAI's GPT-4 and GPT-3.5-turbo for high-quality subtitle translation. These models are known for their natural language understanding and text generation capabilities, suitable for dialogue generation and text analysis.

Qwen: an open-source localized large-scale language model that supports multilingual translation and can process text in multiple languages economically and efficiently.

Google Translate: integrated as a supplement to the translation function, providing extensive language support and good translation quality.

Sound Source Separation

It uses Demucs and UVR5 technology to separate vocals from accompaniment.

|Demucs: a sound source separation model developed by Facebook's research team, which can separate different sound sources in mixed audio, including instruments, vocals, and background sounds. Widely used in music production and film and television post-production.

UVR5 (Ultimate Vocal Remover): an efficient vocal-accompaniment separation tool that can extract accompaniment close to the original stereo, with performance superior to other similar tools such as RX9, RipX, and SpectraLayers 9.

Lip-Syncing

Drawing on Linly-Talker, we focus on digital human lip-syncing technology, combining computer vision and speech recognition technology to precisely match the lip-syncing of virtual characters with dubbing, achieving highly natural synchronization effects. This technology is suitable for various scenarios such as animated characters, virtual anchors, and narration in educational videos.

Video Processing

Linly-Dubbing provides functions such as adding subtitles, inserting background music, adjusting volume and playback speed, allowing users to customize video content to make it more attractive and personalized.

Integration of yt-dlp: yt-dlp is a powerful open-source command-line tool designed to download videos and audio from YouTube and other websites. This tool has extensive parameter options, allowing users to fine-tune downloading behavior according to their needs. Whether selecting specific formats, resolutions, or extracting audio, yt-dlp provides a flexible solution.