OpenVoice: Versatile Instant Voice Cloning 161

OpenVoice, a versatile instant voice cloning method that requires only a short audio clip from the reference speaker to replicate their voice and generate speech in multiple languages. OpenVoice supports fine-grained control over voice style, including emotion, accent, rhythm, pauses, and intonation, as well as cloning the reference speaker's tone color. OpenVoice also achieves zero-shot cross-lingual voice cloning for languages not included in the large-scale speaker training set. OpenVoice is also highly computationally efficient, costing dozens of times less than commercial APIs that perform worse.

OpenVoice paper: https://arxiv.org/pdf/2312.01479.pdf

OpenVoice project: https://github.com/myshell-ai/OpenVoice

OpenVoice Features:

Accurate tone color cloning. OpenVoice can accurately clone the reference timbre and generate speech in multiple languages and accents.

Flexible voice style control. OpenVoice supports fine-grained control over voice styles (such as emotion and accent) as well as other style parameters (including rhythm, pauses, and intonation).

Zero-shot cross-lingual voice cloning. Neither the language of the generated speech nor the language of the reference speech needs to be present in the large-scale speaker multilingual training dataset.

OpenVoice Usage:

1. Flexible voice style control. See demo_part1.ipynb for example usage of how OpenVoice achieves flexible style control over cloned voices.

2. Cross-lingual voice cloning. See demo_part2.ipynb for examples of languages that are visible or unseen in the MSML training set.

3. Advanced usage. The base speaker model can be replaced with any model you prefer (any language and style). Please use the functionality demonstrated in the demo to extract the tone embedding of the new base speaker. se_extractor.get_se

4. Tips for generating natural speech. There are many single-speaker or multi-speaker TTS methods that can generate natural speech and are readily available. Simply replace the base speaker model with the one you like, and you can elevate voice naturalness to the level you desire.

Languages

We only support English and Chinese for this version of the model weights. OpenVoice can be adapted to any other language, provided a base speaker is supplied.

Since May 2023, OpenVoice has been powering the instant voice cloning feature on myshell.ai. As of November 2023, the voice cloning mode has been used tens of millions of times by users worldwide, witnessing explosive growth in the platform's user base.