OSUM: An Open-Source Speech Model from Northwestern Polytechnical University Supporting Multiple Speech Tasks 201

OSUM (Open Speech Understanding Model) is an open-source speech model proposed and released by the Audio, Speech and Language Processing Group (ASLP@NPU) of the School of Computer Science and Technology at Northwestern Polytechnical University, designed to support a variety of speech processing tasks.

OSUM Features:

Multi-task Support:

• Automatic Speech Recognition (ASR): Converts speech into text.

• Speech Recognition with Timestamps (SRWT): Recognizes speech content and annotates timing information.

• Voice Event Detection (VED): Identifies specific events or actions within speech.

• Speech Emotion Recognition (SER): Analyzes emotional states in speech.

• Speaking Style Recognition (SSR): Identifies the speaker's speaking style or tone.

• Speaker Gender Classification (SGC): Determines the speaker's gender based on voice.

• Speaker Age Prediction (SAP): Estimates the speaker's age.

• Speech-to-Text Chat (STTC): Converts spoken conversations into readable text.

Model Architecture: OSUM combines the Whisper encoder with the Qwen2 large language model (LLM), and uses LoRA technology to fine-tune the LLM, enabling efficient multi-task training.

Training Strategy: It adopts an ASR+X training strategy, simultaneously optimizing automatic speech recognition (ASR) and secondary tasks (X), achieving efficient and stable multi-task training while reducing resource consumption.

Deep Understanding Capability: The OSUM model can convert speech information into text and also supports emotion recognition in speech, providing users with a more intelligent interactive experience.

Timestamp Support: In speech recognition, OSUM provides timestamped output, facilitating subsequent processing and analysis.

Multi-platform Compatibility: It supports running on different computing platforms, including Huawei Ascend 910B and NVIDIA platforms, increasing the flexibility of its applications.

OSUM Applications:

• Intelligent Assistants: Used in smart home devices and virtual assistants to understand and respond to user voice commands.

• Multilingual Services: Similar to the YouTube Dubbing plugin, it can be used for real-time translation and dubbing, breaking down language barriers.

• Emotion Analysis: Analyzes user emotions in customer service and mental health applications to provide personalized services.

• Education Sector: Automatically evaluates students' speaking assignments and provides feedback.

• Accessibility Technology: Helps hearing-impaired individuals understand speech information.

• Market Research: Analyzes emotions and opinions in consumer feedback.

• Speech Transcription: In scenarios such as meeting notes and dictation conversion, OSUM can be used for efficient speech transcription.
  

How to Use OSUM:

Installation and Import: Users can download the OSUM code and model weights from the Hugging Face page or the GitHub project page, and follow the provided installation guide to install it. Example code for importing the OSUM model into a Python environment is as follows:

from osum import OSUM
model = OSUM.from_pretrained("ASLP-lab/OSUM")

Executing Tasks: Users can use the OSUM model to perform various speech tasks, such as speech recognition. The following is example code for a speech recognition task:

result = model.transcribe("audio.wav")
print(f"识别结果：{result}")

In terms of usage, users can also fine-tune using pretrained model checkpoints to further improve the model's performance on specific tasks.

GitHub: https://github.com/ASLP-lab/OSUM

Project Page: https://aslp-lab.github.io/OSUM.github.io/