SpeechGPT 2.0-preview: The First Voice Dialogue Model for Interactive and Contextual Intelligence 128

Fudan University's OpenMOSS AI Open Lab has launched its first open-source speech dialogue model targeting real-time interaction and situational intelligence: SpeechGPT 2.0-preview. SpeechGPT 2.0-preview is trained on millions of hours of Chinese speech data and uses an end-to-end architecture to achieve deep fusion of speech and text modalities. It can generate speech with multiple emotions and styles based on instructions, including mimicking the emotions of specific characters.

I. Core Technical Breakthroughs

1. End-to-End Speech Modeling

The model uses semantic-acoustic joint modeling to directly process speech input and generate speech or text output, eliminating the need for traditional cascaded ASR (Automatic Speech Recognition) and TTS (Text-to-Speech) modules. Its proprietary ultra-low-bitrate streaming speech Codec (750bps) compresses speech to 75 tokens per second. Combined with Codec Patchify technology, it aggregates speech tokens from adjacent time steps into unified vectors, effectively mitigating cross-modal modeling conflicts. It supports streaming input and output, achieving real-time interaction with latency under 200ms.

2. Speech-Text Hybrid Modeling

The model adopts a multi-stage training process:

• Modality-adaptive pre-training: Uses unlabeled speech data to learn speech modality features;

• Cross-modal instruction fine-tuning: Enhances instruction comprehension through a large-scale cross-modal instruction dataset (SpeechInstruct);

• Chained modality fine-tuning: Optimizes the precision of cross-modal conversion, e.g., receiving a speech instruction, thinking in text, and then responding in speech.

3. Multimodal Capability Alignment

Through speech-text alignment pre-training, the model "emerges" with generalization capabilities that were not explicitly trained, such as controlling speech rate without speech rate adjustment data, or mimicking the tone and style of unseen characters.

II. Features and Performance

1. Human-like Interaction Experience

Emotion and style control: Supports precise control of multiple emotions (e.g., weak, cheerful), multiple timbres (male/female switching), and multiple styles (poetry recitation, dialect imitation), with outstanding role-playing capabilities;

2. Real-time Interrupt Interaction: Hundred-millisecond response speed supports natural interruption and continuation in conversations.

Intelligence and Practicality

• Text capability integration: Built upon the expressive speech foundation, it retains the intelligence of the text model, supporting tool calling, web search, and external knowledge base integration;

• Multi-task compatibility: Capable of handling long-document parsing, multi-turn dialogues, and other scenarios, with performance on short-text tasks not degraded by the long-context capability.

3. Open-Source Ecosystem Support

Model code, weights, and technical report are fully open-sourced (Apache 2.0 license), with Hugging Face pretrained models and a Gradio demo interface provided, supporting local deployment;

Integrated with the vLLM framework to optimize inference efficiency, the sparse attention mechanism reduces memory usage by 96.7% and boosts processing speed by 3-7x.

III. Application Scenarios

• Intelligent assistants: Customer service, education, or medical assistants that support natural speech interaction, such as real-time oral practice and emotional companionship;

• Content creation: Automatically generates audiobooks, poetry recitations, or dialect content, enriching multimedia creation forms;

• Accessible communication: Provides real-time speech-to-text and synthesis services for the hearing-impaired or speech-impaired.

Through its end-to-end architecture and cross-modal alignment technology, SpeechGPT 2.0-preview achieves low-latency, highly realistic, and highly intelligent speech interaction, representing a significant breakthrough for the open-source community in the field of large speech models. Its open-source ecosystem and multi-scenario adaptability not only provide developers with efficient tools, but also drive human-computer interaction toward emotional and natural deep evolution.

GitHub Repository:https://github.com/OpenMOSS/SpeechGPT-2.0-preview

Online Demo:https://sp2.open-moss.com (Tested it, it's a bit laggy, breaks up when laughing, but still quite natural, though it answers irrelevantly)