Moshi: Open-Source Real-Time Voice Converter 211

Moshi is a multi-stream real-time speech generation Transformer model that supports full-duplex voice conversations. Its main features are simultaneous speech input and output (full-duplex), as well as the ability to handle complex conversational scenarios, including overlapping speech, interruptions, and non-verbal information such as emotional expressions.

This means it can listen and speak at the same time, and is designed to address some of the issues in traditional dialogue systems, such as latency, loss of non-verbal information (e.g., emotion), and rigid structures of conversational turn-taking.

Full-duplex communication: Traditional dialogue systems are turn-based (one person finishes speaking before the other begins). Moshi breaks through this limitation and supports full-duplex communication. This means Moshi can generate speech responses while the user is speaking, without being limited by turn-taking, and can handle complex conversational dynamics such as overlapping speech, interruptions, and rapid feedback.

Multi-stream processing: Moshi achieves simultaneous listening and speech generation by processing multiple audio streams. This multi-stream architecture enables it to flexibly handle voice interactions between the user and the system without interrupting the natural flow of conversation.

Compared with traditional voice dialogue systems, Moshi has several notable advantages:

• Real-time response: Moshi's response speed is extremely fast, with a latency of only 160-200 milliseconds, which is close to the reaction speed in natural conversations, thus providing a smoother conversational experience.

• Speech-to-speech processing: Traditional systems typically rely on a speech-to-text-to-speech process, while Moshi can directly process speech input and generate speech output, preserving non-verbal information such as tone and emotion.

• Full-duplex dialogue: Moshi does not rely on strict conversational turn-taking, but can simultaneously process user and system speech, which means it can handle overlapping speech and interruptions, coming closer to the natural form of human conversation.

Main features of Moshi:

• Real-time voice dialogue: Moshi directly generates audio output from audio input, rather than relying on the traditional speech-to-text-to-speech process. By processing speech data directly, Moshi preserves non-verbal information such as tone, emotion, overlapping speech, and interruptions, ensuring more natural and fluid conversations.

• Full-duplex communication: Moshi is able to listen and speak simultaneously, which means it can generate speech responses while the user is speaking, without the need for strict conversational turn-taking. It can handle complex conversational scenarios, such as overlapping speech and non-interrupting feedback (such as "hmm" or "I see") that can be inserted at any time.

• Low latency: Moshi is designed with very low latency, theoretically only 160 milliseconds, and in practice about 200 milliseconds. This means Moshi can respond to user input in near real-time, providing a smoother conversational experience.

• Inner monologue method: Moshi predicts text tokens before generating speech, which significantly improves the language quality and consistency of the generated speech. This not only makes the generated speech clearer, but also enhances the system's speech recognition and text-to-speech capabilities in streaming environments. By introducing an "inner monologue" mechanism, Moshi achieves streaming speech recognition (ASR) and text-to-speech (TTS) capabilities, supporting the simultaneous processing of language and audio in continuous conversational streams.

• Parallel processing of multiple audio streams: Moshi is able to simultaneously process the audio streams of both the user and the system. This multi-stream processing capability allows Moshi not only to generate its own speech, but also to understand and respond to the user's speech in real time.

• Emotion and speech dynamics processing: By processing speech directly rather than intermediate text, Moshi is able to understand and generate emotionally rich speech, and handle complex conversational dynamics such as emotional expression, vocal variation, and more.

• Support for complex conversational dynamics: Moshi is able to handle the complex dynamics of natural conversations, such as interruptions, interleaving, interjections, and responses. Traditional systems rely on clear conversational turn-taking (one person speaks before the other's turn), but Moshi removes this limitation, making conversations more natural.

Moshi's model architecture

Moshi consists of three main components: Helium, a 7B language model trained on 2.1 trillion tokens; Mimi, a neural audio codec that models both semantic and acoustic information; and a new multi-stream architecture that can model the audio of the user and Moshi separately.

Working together, these modules enable smooth full-duplex dialogue, emotional expression, and the handling of complex conversational dynamics.

Helium text language model

Helium is the core of Moshi. It is a text language model with 7 billion parameters based on the Transformer architecture (similar to GPT). Helium provides Moshi with powerful language understanding and generation capabilities, able to handle complex text reasoning and dialogue tasks.

Its training data includes 2.1 trillion English words, giving it broad knowledge and linguistic capabilities.

Mimi neural audio codec:Mimi is Moshi's audio processing component. It is a neural network audio codec responsible for converting audio into discrete speech tokens, and is able to reversely generate high-quality speech output.

Mimi uses Residual Vector Quantization (RVQ) technology to encode speech data into discrete speech and semantic tokens, ensuring high speech fidelity and linguistic consistency.

By combining semantic and acoustic tokens, Mimi not only generates natural speech, but also handles complex speech contexts and emotional information.

Inner monologue method:The inner monologue method is the key technology behind Moshi's speech generation. It allows the model to predict text tokens synchronized with audio before generating speech. This approach not only improves the language quality of generated speech, but also enables Moshi to achieve speech recognition and text-to-speech conversion in streaming environments.

Simultaneous generation of text and speech: Before generating audio, Moshi generates a text stream corresponding to its speech output. This text stream serves as the basis for speech generation, making speech generation more accurate and helping handle complex conversational scenarios.

Streaming compatibility: This approach allows Moshi to process speech while still achieving efficient speech recognition and text-to-speech (TTS) in streaming environments.

The model architecture is designed to handle multiple parallel audio streams and generate speech and text in real time. Moshi can generate system speech while processing user speech, enabling it to support uninterrupted natural conversations.

Moshi's detailed technical approach

1. Speech-to-speech generation architecture

Moshi's core innovation lies in treating voice dialogue as a speech-to-speech generation task, rather than the traditional multi-component process of text-to-speech and then speech-to-speech. Traditional voice dialogue systems include multiple independent modules such as voice activity detection (VAD), automatic speech recognition (ASR), natural language understanding (NLU), natural language generation (NLG), and text-to-speech (TTS).

Moshi directly generates speech tokens, so that speech does not rely on intermediate text representations during understanding and generation, thereby avoiding the loss of information such as emotion, tone, and non-verbal sounds.

2. Helium text language model

Moshi is based on the Helium text language model, a large text generation model with 7B parameters. Helium has been pre-trained on 2.1 trillion English data tokens, giving it strong language understanding, reasoning, and generation capabilities. It serves as the foundation of Moshi's semantic understanding, supporting complex natural language processing functions, including open-domain dialogue and question answering.

Main features of Helium:

Autoregressive Transformer architecture: Moshi is based on Helium, a text language model built on the Transformer architecture. Similar to the classic Transformer, Helium uses multi-layer attention mechanisms and autoregressive modeling methods to process text input and generate output. The model has 7B parameters, sufficient to support learning from large-scale corpora.

RMS normalization: RMS normalization is applied in the attention module, feed-forward module, and output layer to improve the training stability of the model.

Rotary Position Embedding (RoPE): Used to handle longer context windows (4096 tokens), ensuring that the model can capture long-range dependencies in conversations.

Efficient FlashAttention: Through optimized attention computation, model inference under long sequence input is more efficient.

3. Mimi neural audio codec

Mimi is the neural audio codec used in Moshi for speech processing. Its task is to discretize continuous speech signals into audio tokens. These discrete audio tokens, similar to text tokens, can represent detailed information in speech. Mimi employs Residual Vector Quantization (RVQ) technology to preserve high-quality audio at a low bitrate, supporting real-time speech generation and processing.

Key technologies of Mimi:

Residual Vector Quantization (RVQ): Mimi uses multi-level residual vector quantization to discretize complex audio signals into multiple levels of audio tokens. This approach allows each time step to efficiently encode the semantic and acoustic information of speech, while ensuring the quality of audio reconstruction.

Combination of semantic and acoustic tokens: The audio tokens used by Mimi include both semantic and acoustic information. Semantic tokens preserve the content of speech (such as the specific words spoken), while acoustic tokens describe the audio characteristics of speech, such as timbre, emotion, and intonation.

Streaming encoding and decoding: Mimi supports streaming, enabling continuous speech generation and recognition in real-time conversations. This makes Moshi's response speed very close to that of natural conversation.

Moshi adopts a multi-stream hierarchical generation architecture that can process multiple audio streams in parallel. By simultaneously modeling the user's speech stream and the system's own speech stream, Moshi enables flexible interaction in conversations, allowing complex conversational dynamics such as interleaving, interruptions, and interjections between speakers.

This is an architecture previously proposed for discrete image generation, and it can model the hierarchy of semantic and acoustic tokens without increasing the Helium sequence length. This means that every second of audio only needs to pass through the 7B backbone model 12.5 times, enabling real-time performance on an L4 or M3 MacBook Pro! Combined with MusicGen's token latency, this delivers state-of-the-art performance for audio language modeling.

Hierarchical Autoregressive Modeling: Moshi uses an RQ-Transformer (Residual Quantizer Transformer) to decompose audio tokens into multiple levels, and generates audio through hierarchical autoregressive modeling. Specifically, the model first uses a larger Temporal Transformer to process the time series, and then uses a smaller Depth Transformer to handle multiple sub-sequences at each time step. This design greatly improves the efficiency of generating long audio sequences.

Multimodal Sequence Generation: The model simultaneously generates multiple sequences (including text, semantic tokens, and audio tokens), and ensures their precise temporal alignment through an inner monologue mechanism. The content generated at each time step includes not only the current speech but also the corresponding text prefix, making the generated speech more semantically logical.

Architecture of RQ-Transformer

5. The "Inner Monologue" Mechanism

Moshi's "Inner Monologue" mechanism is one of its key innovations in speech generation. Through this mechanism, Moshi predicts the corresponding time-aligned text tokens before generating audio. This not only improves the linguistic consistency of generated speech, but also supports real-time automatic speech recognition (ASR) and text-to-speech (TTS) conversion.

Features of the "Inner Monologue" mechanism:

Aligned Text and Audio Generation: Moshi first predicts the text, then generates audio, making the produced speech more accurate and fluent in grammar and content.

Delay Mechanism: By introducing a delay between text and audio, Moshi can perform ASR and TTS tasks separately. For example, if text is generated first and then audio, the model operates in TTS mode; otherwise, it is in ASR mode. Moshi can seamlessly switch between these two modes, ensuring the model can both generate and recognize speech.

Moshi: Delay mechanism

Moshi: TTS mode

6. Multi-Stream Modeling

Moshi's architecture allows simultaneous processing of multiple audio streams, capable of both monitoring the user's speech and generating the system's own speech. During a conversation, Moshi can dynamically handle overlapping parts of audio (such as interruptions and interleaving) without requiring an explicit pre-defined speaker turn-taking. This technology makes conversations more natural.

Synchronous Generation of Semantic and Acoustic Tokens: Moshi uses a parallel semantic and audio token generation mechanism, and optimizes the dependencies between these tokens by introducing temporal delays. Through precise modeling of the user's and the system's audio streams, Moshi can flexibly handle complex conversational scenarios.

Dual-Stream Audio Processing: Moshi simultaneously processes the user's and the system's speech streams, and achieves full-duplex conversation through parallel modeling of two autoregressive audio streams. This design enables the model to handle overlapping speech and interruptions in natural conversations.

Delay Alignment of Semantics and Audio: By introducing a delay between semantic tokens and audio tokens, the generated speech content is ensured to be coherent and efficient. The delay may be 1 to 2 frames, depending on the conversational dynamics.

Moshi: Multi-stream modeling

7. Model Training and Fine-Tuning

Large-Scale Pretraining: Moshi's text language model (Helium) has rich language understanding and generation capabilities through pretraining on over 2.1 trillion English tokens. The model is trained on large-scale text and speech data and can handle various complex conversational scenarios.

Unsupervised and Supervised Multi-Stage Training: Moshi first pretrains on large-scale unsupervised speech data, then post-trains on multi-stream data containing natural conversations, and finally performs instruction fine-tuning to improve its performance in real conversations.

Helium Pretraining: First, pretrain the Helium text language model on a large-scale text dataset to improve its language understanding and reasoning capabilities.

Moshi Pretraining: Train the multi-stream audio model on unlabeled audio datasets to learn to handle speech generation and semantic understanding.

Multi-Stream Fine-Tuning: Use the Fisher dataset (containing two-way speech conversation data) to fine-tune the model, improving its ability to handle multi-stream speech input.

Instruction Fine-Tuning: Finally, use generated instruction dialogue data for fine-tuning to enhance the model's performance in natural conversation scenarios.

Data Augmentation: During training, Moshi uses data augmentation techniques, such as adding background noise and simulating user echoes, enabling the model to perform stably across different acoustic environments and enhancing its robustness.

Performance Evaluation of Moshi

1. Quality and Consistency of Speech Generation

Speech Clarity: Moshi performs excellently in speech generation, and experiments show it can produce high-quality, easily understandable speech. It can maintain speech coherence throughout the generation process, especially in long conversations, which is an important performance indicator for conversational models in complex contexts.

Naturalness and Consistency of Speech: By using the Mimi neural audio codec, Moshi can generate high-fidelity speech and maintain consistency in the system's voice. In addition, the model is capable of generating appropriate emotional tones based on different conversational contexts, enhancing the naturalness of the user experience.

2. Real-Time Response Performance

Low Latency: Moshi's theoretical latency is 160 milliseconds, with actual testing at approximately 200 milliseconds. This means Moshi can respond to user input in near real-time, significantly improving interaction smoothness and the user's conversational experience.

Full-Duplex Communication Capability: Moshi has demonstrated in tests its ability to simultaneously receive and generate speech. This full-duplex functionality allows it to handle overlapping speech and conversation interruptions, showing a response speed close to natural human conversation.

3. Speech Recognition and Conversational Understanding

Automatic Speech Recognition (ASR): Through the inner monologue approach, Moshi combines text and speech streams, significantly improving speech recognition accuracy. The model not only captures the user's speech input but also enhances the system's response accuracy by first generating text predictions.

Conversational Understanding and Reasoning: Moshi uses the Helium language model for text understanding and reasoning, which enables it to perform well in handling complex questions, open-ended dialogues, and knowledge Q&A. Experimental results show that Moshi can effectively understand context and provide reasonable answers.

4. Robustness of Multi-Stream Speech Processing

Overlapping Speech Handling: Moshi is able to handle complex conversational scenarios in evaluations, such as conversations with overlapping speech across multiple streams. This is very important for multitasking in real-world applications, as natural conversations often involve interruptions and overlapping speech.

Multi-Context Conversation Handling: Trained on multiple data streams, Moshi performs well in different conversational scenarios, whether it's a single user's speech stream or simultaneous conversations with multiple users.

5. Question Answering and Knowledge Acquisition

Moshi outperforms other current speech conversation systems in question answering and knowledge acquisition tasks. With powerful text understanding capabilities and real-time speech generation, Moshi can handle multi-turn Q&A, accurately extracting and responding to user questions.

Language Reasoning and Commonsense Q&A: The model is capable of handling complex reasoning tasks and performs well in various standard natural language processing (NLP) evaluations, such as commonsense Q&A, reading comprehension, and open-ended question answering.

6. Speech Emotion and Personalized Generation

Emotional Speech Generation: Moshi has demonstrated in evaluations its ability to generate emotional speech. It can generate speech outputs with different emotions, such as anger, happiness, or sadness, based on the context of the conversation.

Personalized Voice Styles: Through instruction fine-tuning during training, Moshi can generate speech in different styles or specific characters according to user needs. This personalized capability makes it perform more diversely in specific conversational scenarios.

7. Safety and Reliability

Safe Conversation Evaluation: Moshi shows good safety when handling conversations containing sensitive or inappropriate content. It can effectively identify and avoid generating inappropriate content, ensuring the safety and ethics of conversations.

Robustness and Adaptation to Noisy Environments: In evaluations involving noisy and complex environments, Moshi demonstrates good robustness. Through data augmentation techniques (such as noise addition and echo processing), the model can cope with different acoustic environments and ensure high-quality output in noisy conditions.

8. Comprehensive Test Results

Moshi's comprehensive performance tests show that it has achieved leading results in speech generation, conversational understanding, real-time response, and complex conversation handling. In particular, Moshi's performance in handling overlapping dialogues, speech interruptions, and emotional generation far exceeds traditional conversation systems.

Technical Report: https://kyutai.org/Moshi.pdf

GitHub: https://github.com/kyutai-labs/moshi

Model Download: https ://huggingface.co/collections/kyutai/moshi-v01-release-66eaeaf3302bef6bd9ad7acd

Try it Online: https://moshi.chat/

Related News:

MoshiVis: A visual speech model that can hear, see, and discuss image content with you in fluent speech