Script Planner Assistant▸ 
AI Apps 
AI News MegaTTS 3: ByteDance's Zero-Shot Text-to-Speech Synthesis System
小编 ·
MegaTTS 3: ByteDance's Zero-Shot Text-to-Speech Synthesis System
What is MegaTTS 3?
MegaTTS 3 is a zero-shot text-to-speech synthesis system launched by ByteDance. It uses a lightweight diffusion model to decompose speech into attributes such as content, timbre, and prosody for modeling. It supports ultra-high-quality voice cloning, bilingual synthesis, accent intensity control, and other features, and can be applied to scenarios such as speech synthesis and speech editing.
Core Architecture and Technology
Lightweight and efficient diffusion model: The core of MegaTTS 3 is the TTS Diffusion Transformer, which has only 0.45B parameters, ensuring efficiency while generating high-quality speech.
Speech decomposition and modeling: Speech is decomposed into different attributes such as content, timbre, and prosody, with appropriate modules designed for each attribute for modeling. For example, global vectors are used to model timbre, as timbre is a global attribute that changes slowly over time; a VQGAN-based acoustic model is used to generate spectrograms, and a latent code-based language model is used to fit the distribution of prosody, as prosody changes rapidly within a sentence, and the language model can capture local and long-range dependencies.
Vocoder: A GAN-based vocoder is used to properly construct the phase without requiring a language model to model the phase.
MegaTTS 3 Features
Ultra-high-quality voice cloning: Capable of generating speech highly similar to the target speaker.
Bilingual support: Supports Chinese and English, as well as Chinese-English mixed speech synthesis.
Controllability: Supports accent intensity control, and will support more fine-grained pronunciation/duration adjustments in the future.
Zero-shot speech synthesis: Capable of generating speech for a target speaker without specific voice data from them.
MegaTTS 3 Application Scenarios
Speech synthesis: Can be used in various scenarios that require converting text to speech, such as audiobooks and voice broadcasts.
Speech editing: Capable of editing and modifying speech to meet different needs.
Cross-lingual speech synthesis: Supports cross-lingual speech synthesis tasks.
MegaTTS 3 Tutorial
Command-line tool: A command-line interface is provided, allowing users to generate synthesized speech by specifying the input voice prompt file and text content. For example:
Standard speech synthesis command:
CUDA_VISIBLE_DEVICES=0 python tts/infer_cli.py --input_wav 'assets/Chinese_prompt.wav' --input_text "另一边的桌上，一位读书人嗤之以鼻道，'佛子三藏，神子燕小鱼是什么样的人物，李家的那个李子夜如何与他们相提并论？'" --output_dir ./gen
Speech synthesis command with accent control:
CUDA_VISIBLE_DEVICES=0 python tts/infer_cli.py --input_wav 'assets/English_prompt.wav' --input_text '这是一条有口音的音频。' --output_dir ./gen --p_w 1.0 --t_w 3.0
Web UI: Speech synthesis via Web UI is also supported.
MegaTTS 3 Installation
Environment requirements: Python 3.9 environment is required.
Model download: Pretrained model checkpoints can be downloaded from Google Drive or Hugging Face and placed in the specified directory.
Installation steps:
# Create a Python 3.9 conda environment conda create -n megatts3-env python=3.9 conda activate megatts3-env # Install dependencies pip install -r requirements.txt
github project: https://github.com/bytedance/MegaTTS3
