Dolphin: An Automatic Speech Recognition Model for Eastern Languages Co-Released by Datatang and Tsinghua University 139

What is Dolphin?

Dolphin is an automatic speech recognition model designed specifically for Eastern languages, jointly developed by DataoceanAI and the Speech and Audio Technology Laboratory of the Department of Electronic Engineering at Tsinghua University.Dolphin supports 40 Eastern languages and 22 Chinese dialects. It features high accuracy and strong robustness, capable of adapting to a variety of complex speech environments. Dolphin is widely used in scenarios such as speech-to-text, voice interaction, and voice content analysis, and supports one-click installation and multiple invocation methods.

Dolphin Model Features

• Multilingual Support: Dolphin supports 40 Eastern languages, including those from East Asia, South Asia, Southeast Asia, and the Middle East, as well as 22 Chinese dialects (including Mandarin).

• Language Adaptability: Automatically identifies the language of the input speech, requiring no manual specification.

• High Accuracy: Across multiple test sets, the Word Error Rate (WER) is significantly lower than that of comparable models.

• Strong Robustness: Adapts to complex speech environments involving different accents, speaking speeds, and background noise.
  

• High Performance: On the DataoceanAI, Fleurs, and CommonVoice test sets, Dolphin achieves a significantly lower WER compared to Whisper models of the same size. For example, the average WER is reduced by 63.1% for the base version and 68.2% for the small version.

• Open Source: The base and small versions of the Dolphin model, along with the inference code, are fully open-sourced.

Dolphin Technical Architecture

• Network Structure: Based on the CTC-Attention architecture, it employs an E-Branchformer encoder and a Transformer decoder, with the introduction of a 4x downsampling layer. This structure combines the sequence modeling capabilities of CTC with the context-capturing ability of the attention mechanism, effectively improving the model's recognition accuracy and efficiency.

• Multi-task Format: Drawing on the design approaches of Whisper and OWSM, Dolphin focuses on ASR and makes several key modifications, such as removing the use of "previous text" and its associated tags, simplifying the input format. In addition, Dolphin introduces a two-level language tag system to better handle linguistic and regional diversity.

Data Foundation

Dolphin's training dataset integrates proprietary data from DataoceanAI with multiple open-source datasets, totaling more than 212,000 hours. Among these, the DataoceanAI dataset contains 137,712 hours of audio covering 38 Eastern languages.

Application Scenarios

• Speech-to-Text: Suitable for scenarios such as meeting transcription and voice input methods.

• Voice Interaction: Used in AI intelligent voice assistants, smart home applications, and more, enhancing the interactive experience.

• Voice Content Analysis: Quickly transcribe and analyze audio media content.

Dolphin Usage Tutorial

1. Installation: Users can perform a one-click installation using the command pip install -U dataoceanai-dolphin.

2. Command Line Invocation: For example, use dolphin audio.wav for speech recognition. You can also specify the model version, language, and region tags.

3. Python Invocation: By importing the dolphin module and loading the audio and model, you can perform speech recognition.

Related Links

Github Address: https://github.com/DataoceanAI/Dolphin

Project Homepage: https://huggingface.co/DataoceanAI