Seed-TTS: A High-Quality Text-to-Speech Model by ByteDance 362

Recently, the TTS field has exploded. Just after the open-sourcing of chatTTS, Byte released seed-tts, which performs very well.

Seed-TTS is a high-quality text-to-speech (TTS) model developed by ByteDance. The Seed-TTS model is capable of generating speech that is nearly indistinguishable from human speech.

As a foundational model for speech generation, Seed-TTS excels at in-context learning for speech, and its performance in terms of speaker similarity and naturalness matches that of real human speech in both objective and subjective evaluations.

Without any training, Seed-TTS only needs a short speech clip to clone and generate highly natural and expressive speech, making it suitable for tasks such as reading novels and dubbing.

Seed-TTS also provides advanced control over various speech attributes, including but not limited to emotion, intonation, speaking style, etc. Additionally, the generated speech can be edited by editing the text.

Seed-TTS can modify the content in audio while keeping the voice timbre, pitch, and emotion unchanged.

Seed-TTS supports speech rate adjustment, allowing free control over audio speed.

Seed-TTS provides excellent controllability over various speech attributes (such as emotion) and is capable of generating highly expressive and diverse speech for in-the-wild speakers.

Furthermore, we propose a self-distillation method for speech factorization, as well as a reinforcement learning approach to enhance the model's robustness, speaker similarity, and controllability. We also introduce a non-autoregressive (NAR) variant of the Seed-TTS model, named Seed-TTS DiT, which adopts a fully diffusion-based architecture. Unlike previous NAR-based TTS systems, Seed-TTS DiT does not rely on pre-estimated phoneme durations and performs speech generation in an end-to-end manner. We demonstrate that this variant achieves performance comparable to the language model-based variant in both objective and subjective evaluations, and showcase its effectiveness in speech editing.

Paper: https://arxiv.org/abs/2406.02430