StreamSpeech: Real-Time Translation Model for Streaming Speech Input 192

StreamSpeech is a real-time translation model that enables streaming voice input for offline and simultaneous speech recognition, speech translation, and speech synthesis. It can instantly translate speech into another language during real-time communication while simultaneously outputting the corresponding target speech.

It not only translates speech into another language, but also transcribes the speech content into text in real time. Users can simultaneously obtain translation results in both voice and text formats, and the translation process is synchronous—there is no need to wait for the entire voice input to finish, thereby achieving low-latency real-time translation.

StreamSpeech can also display real-time speech recognition results during the translation process, helping users keep track of translation progress instantly.

StreamSpeech can also be seamlessly integrated into various applications and devices, such as translation earphones, conference systems, live streaming platforms, etc., providing users with convenient translation services. Whether used on personal devices or deployed in large-scale conference systems, Simul-S2ST delivers stable performance.

StreamSpeech Application Scenarios:

• Use StreamSpeech for simultaneous interpretation in international conferences.

• Multinational companies use StreamSpeech for remote meetings, enabling real-time multilingual communication.

• Language learners use StreamSpeech to practice listening and speaking in different languages.

StreamSpeech Highlights:

• StreamSpeech achieves SOTA performance in both offline and simultaneous speech-to-speech translation.

• StreamSpeech performs streaming ASR, simultaneous speech-to-text translation, and simultaneous speech-to-speech translation seamlessly through an "all-in-one" model.

• StreamSpeech can present intermediate results (i.e., ASR or translation results) during simultaneous interpretation, providing a more comprehensive low-latency communication experience.

StreamSpeech is ideal for professionals who need real-time cross-language communication. By reducing translation latency, it enables people from different language backgrounds to engage in real-time conversations without barriers.

Official Website:https://ictnlp.github.io/StreamSpeech-site/

Github:https://github.com/ictnlp/StreamSpeech

Paper:https://arxiv.org/abs/2406.03049