Spatial Speech Translation: A Cross-Space Speech Translation Project 60

What is Spatial Speech Translation?

Spatial Speech Translation is a practical cross-space speech translation project. Through binaural wearable devices, it is capable of performing speech translation under multi-speaker and noisy conditions, while preserving the original directional sense of the sound. The project supports real-time speech translation and can achieve fast inference on Apple M2 chips. In addition, during the translation process, it preserves the speaker's tone and emotion, making the translated speech more natural and smooth.

Main Features of Spatial Speech Translation

• Multi-speaker speech separation and translation: Capable of accurately separating target speech and translating it under multi-speaker and noisy conditions.

• Real-time speech translation: Supports real-time operation on Apple chips, providing a low-latency translation experience.

• Spatial cue preservation: Through binaural rendering technology, the spatial cues of the input speech (such as direction and distance) are preserved in the translated output speech.

• Multi-language support: Provides translation models from French to English, with plans to expand to other languages such as German and Spanish.

• Expressive speech translation: Supports translation that preserves the speaker's intonation and emotion, making the translated speech more expressive.

Technical Implementation of Spatial Speech Translation

• Joint separation and localization model: By training separation and localization models, it can identify the voices of different speakers and determine their spatial positions.

• Streaming speech processing: Based on StreamSpeech technology, supports real-time streaming speech input and translation.

• Binaural rendering: Leveraging the characteristics of binaural wearable devices to achieve spatialized output of translated speech.

• Multi-step training process: Includes pretraining, separation-aware fine-tuning, and expressive speech generation, improving model performance and adaptability.

How to Use Spatial Speech Translation

• Environment setup: Install necessary dependencies via Conda and pip, including tools required for speech separation, streaming speech processing, and binaural rendering.

• Model training: Provides detailed training steps, including training of the separation and localization model, pretraining and fine-tuning of the speech translation module.

• Inference and evaluation: Supports translation of separated speech and provides evaluation scripts to verify model performance.

Advantages and Innovations

• Multi-speaker support: First to achieve speech translation in multi-speaker environments.

• Spatial cue preservation: Provides an immersive speech translation experience through binaural rendering technology.

• Real-time capability: Supports real-time speech translation, suitable for dynamic communication scenarios.

• Expressive translation: Preserves the speaker's intonation and emotion, making translations more natural.

Applications of Spatial Speech Translation

• International conferences and business events: At international conferences, business negotiations, or seminars, attendees come from different countries and speak different languages. With binaural devices, everyone can hear the translated speech in real time and perceive the speaker's voice direction and distance.

• Tourism and cultural experiences: When tourists visit historical sites, museums, or attend cultural activities, they can use binaural devices to hear real-time translation of commentary while knowing the spatial position of the narrator's voice.

• Remote meetings and collaboration: In remote meetings, participants are in different locations and speak different languages. Through binaural speech translation technology, everyone can hear translated speech in real time and perceive the speaker's position in virtual space.

• Education and training: In multilingual educational environments, students can use binaural devices to hear real-time translation of the teacher's lectures and perceive the direction and distance of the teacher's voice, which is very helpful for language learners to understand and imitate speech.

• Customer service and reception: At airports, hotels, or customer service centers, staff can use binaural devices to communicate with customers of different languages in real time, providing more natural and efficient service.

Project Address

https://github.com/chentuochao/Spatial-Speech-Translation