Hibiki: A High-Fidelity Real-Time Speech Translation Model by Kyutai Labs 97

Hibiki is a high-fidelity real-time speech translation model developed by Kyutai Labs that supports French and English. It preserves the characteristics of the original voice and performs real-time translation, listening and translating simultaneously. Like simultaneous interpretation, it outputs in real time, generating both speech and text translations.

Hibiki Features:

• Multi-stream Language Model Architecture: Hibiki adopts a multi-stream architecture similar to Moshi, enabling it to process source and target language speech simultaneously and achieve seamless conversion between speech-to-speech (S2ST) and speech-to-text (S2TT).

• Preservation of Voice Characteristics: One of the highlights of the Hibiki model is that the generated speech sounds natural and smooth, close to normal human speaking patterns.

• Automatic Speech Rate Adjustment: Hibiki intelligently adjusts the speech rate of the translated output based on the semantic content of the source language, ensuring fluency and accuracy of translation.

• Real-time Translation: It can complete speech-to-speech or speech-to-text translation in a short time. For example, during cross-border travel conversations, when one party finishes speaking, the translated result is almost immediately available and played back—very convenient, isn't it?

• On-device Compact Model: Hibiki-M is a lightweight version of Hibiki, with 1.7 billion parameters, capable of running in real time on mobile devices such as iPhones without relying on cloud processing, showcasing the advancement of model optimization and compression techniques.

• Adaptive Translation: Hibiki can accurately recognize speech input with different accents. For example, whether it's standard British English, American English, or regionally accented Indian English.

• Multilingual Support: In addition to French and English, Hibiki's multi-stream audio-text language model can be extended to other languages, increasing the flexibility and adaptability of its applications.

Hibiki Technology:

Hibiki is a decoding model that uses precise synchronization algorithms to quickly translate speech from one language into speech or text in another language. The core of this technology lies in its multi-stream language model, which can generate audio and text output in real time during speech recognition and translation. Additionally, Hibiki supports voice characteristic transfer, preserving the speaker's tone and emotion during translation, making the translation effect more natural and authentic.

However, because it requires transmission of voice data and cloud computing processing, it has high network requirements. If the network signal is poor, recognition errors or translation delays may occur.

Hibiki Application Scenarios:

• Business Meetings: For business professionals engaged in cross-border activities, Hibiki can help attendees with instant translation in meetings or business dinners, eliminating language barriers.

• Online Education: In educational platforms, Hibiki can provide real-time speech translation, helping students from different language backgrounds learn and communicate better.

• Travel Translation: When traveling abroad, it helps tourists communicate simply with locals, such as asking for directions or ordering food. Tourists can speak their needs directly into their phone and receive translated voice or text responses.

• Doctor-Patient Communication: In places such as hospitals, Hibiki can assist communication between doctors and patients.

GitHub: https://github.com/kyutai-labs/hibiki

HuggingFace: https://huggingface.co/collections/kyutai/hibiki-fr-en-67a48835a3d50ee55d37c2b5

arXiv Paper: https://arxiv.org/pdf/2502.03382