Llasa: A Llama-based TTS with Natural, Expressive Voice Generation 212

What is Llasa?

Llasa is an open-source text-to-speech (TTS) system developed by the Audio Lab at the Hong Kong University of Science and Technology. Built on the LLaMA architecture and combined with XCodec2 speech token technology, it can generate high-quality, natural, fluent, and expressive speech, improving the naturalness, prosodic accuracy, and emotional expressiveness of speech synthesis, with relatively natural generation results.

Core Features of Llasa

• High-quality speech synthesis: Capable of generating natural and fluent speech, supporting both Chinese and English, with excellent speech naturalness and emotional expression.

• Emotional expression: Can synthesize speech with emotional tones such as happiness, sadness, and anger, making the speech more natural and vivid.

• Voice cloning: With only a 15-second voice sample, it can highly accurately clone a voice, preserving the original voice timbre and emotional characteristics.

• Zero-shot learning: Can perform speech synthesis for unseen speakers or emotions without the need for additional fine-tuning.

• Long text support: Capable of processing long text input and generating coherent speech output, suitable for scenarios such as audiobooks and voice broadcasting.

Technical Principles of Llasa

• Architecture: Based on a single-layer vector quantization (VQ) codec and a single Transformer architecture, fully aligned with the standard LLaMA model.

• Speech tokenizer: Uses XCodec2 to encode speech waveforms into discrete tokens, preserving semantic and acoustic information.

• Training and inference scaling: Increasing model size (such as 1B, 3B, 8B parameters) or the amount of training data (such as 250,000 hours of speech data) to improve speech naturalness and emotional expression capabilities.

• Autoregressive generation: Generates speech tokens one by one, ensuring that the speech is consistent with the input text in terms of semantics and prosody.

Llasa Model Versions

Llasa has 1B, 3B, and 8B parameter versions, supporting Chinese and English, which can meet the needs of different hardware resources and application scenarios.

Application Scenarios of Llasa

It is suitable for fields such as virtual reality, gaming, accessibility services, customer service, and education, as well as for creating audiobooks or voice assistants.

• Intelligent voice assistants: Llasa TTS enables smart devices or software to have natural and smooth voice interaction functions, providing a better user experience.

• Audiobooks and online education: It can convert text content into vivid speech, used for audiobook production and online education, making it convenient for students to learn through listening.

• Voice broadcasting and customer service: Llasa TTS can be used for news broadcasting, traffic information announcements, and customer service systems, making information delivery more efficient.

• Gaming and entertainment: It can provide personalized voices for characters or virtual avatars, enhancing immersion.

• Advertising and marketing: Enterprises can use Llasa TTS to generate advertising voiceovers, enhancing brand promotion effects.

• Voice cloning and content creation: Llasa TTS supports zero-shot voice cloning. With only about 15 seconds of audio sample, it can clone the timbre and emotion of a specific voice, and can be used for advertising dubbing, video production, or personalized voice content creation.

GitHub Repository:https://github.com/zhenye234/LLaSA_training

HuggingFace Model Library:https://huggingface.co/collections/HKUSTAudio/llasa

Online Demo:https://huggingface.co/spaces/srinivasbilla/llasa-3b-tts