SenseVoice: Alibaba's Multilingual Audio Model 118

SenseVoice, a multilingual audio foundation model under the FunAudioLLM framework released by Alibaba's Tongyi Lab. SenseVoice focuses on high-precision multilingual speech recognition, speech emotion recognition, and audio event detection, with multilingual, code-switched, timbre, and emotion control capabilities, and demonstrates excellent performance in zero-shot speech generation, cross-lingual voice cloning, and instruction following.

FunAudioLLM combines advanced speech understanding and generation technologies to handle complex speech tasks and enable natural interaction across multiple language environments.

SenseVoice can be applied to speech translation, emotional voice chat, interactive podcasts, and expressive audiobook narration.

SenseVoice provides audio understanding capabilities, including automatic speech recognition (ASR), language identification (LID), speech emotion recognition (SER), and acoustic event classification (AEC) or acoustic event detection (AED).

SenseVoice Performance:

• Multilingual Speech Recognition: Trained on more than 400,000 hours of data, it supports over 50 languages, with recognition performance surpassing the Whisper model.

• Rich Transcription:

  Excellent emotion recognition capability, achieving and surpassing the performance of the current best emotion recognition models on test data.

  Provides sound event detection capability, supporting the detection of various common human-computer interaction events such as background music (BGM), applause, laughter, crying, coughing, and sneezing.

• Efficient Inference: The SenseVoice-Small model adopts a non-autoregressive end-to-end framework, achieving extremely low inference latency. Processing 10 seconds of audio only takes 70ms, 15 times faster than Whisper-Large.

• Convenient Finetuning: Provides convenient finetuning scripts and strategies, allowing users to easily address long-tail sample issues based on their business scenarios.

• Service Deployment: Provides a service deployment pipeline that supports multiple concurrent requests, with client languages including Python, C++, HTML, Java, C#, and more.

Multilingual speech recognition performance and inference efficiency were tested on AISHELL-1, AISHELL-2, Wenetspeech, Librispeech, and Common Voice. For Chinese and Cantonese, the SenseVoice-Small model delivers excellent results.

github: https://github.com/FunAudioLLM/SenseVoice