Voicebox: Generative AI Speech Model for Voice Generation Tasks 151

Voicebox, a sophisticated generative AI speech model developed by Meta ai, can perform speech generation tasks such as editing, sampling, and stylization, but it was not specifically trained to complete these tasks through in-context learning.

Voicebox can generate high-quality audio clips and edit pre-recorded audio (such as removing car horns or dog barking) while preserving the content and style of the audio. The model is also multilingual and can produce speech in six languages.

Voicebox Features:

The versatility of Voicebox enables a variety of tasks, including:

• In-context text-to-speech synthesis: Using audio samples as short as two seconds, Voicebox can match the dubbed audio style and use it for text-to-speech generation.

• Speech editing and noise reduction: Voicebox can recreate portions of speech interrupted by noise or replace misspoken words without re-recording the entire speech. For example, you can identify a speech segment interrupted by a dog barking, trim it, and instruct Voicebox to regenerate that segment — just like an eraser for audio editing.

• Cross-lingual style transfer: When given a sample of someone's voice and a passage of English, French, German, Spanish, Polish, or Portuguese text, Voicebox can read the text in any of these languages, even when the sample voice and the text are in different languages. This feature could potentially be used in the future to help people communicate in a natural, authentic way, even when they speak different languages.

• Diverse speech sampling: After learning from diverse data, Voicebox can generate speech that more authentically represents how people speak in the real world and across the six languages mentioned above.

Voicebox Advantages:

• Advanced noise reduction and style conversion tools improve audio quality and versatility.

• Extensive training on 50,000+ hours of speech data ensures high accuracy and natural-sounding output.

• The ability to work with six different languages makes it an ideal choice for global applications and cross-lingual tasks.

Voicebox Disadvantages:

• Limited availability due to potential misuse may restrict access for legitimate users and developers.

In the future, versatile generative AI models like Voicebox could provide natural-sounding voices for virtual assistants and non-player characters in virtual universes. They could allow visually impaired individuals to hear AI reading their friends' written messages aloud, offer creators new tools to easily produce and edit video soundtracks, and much more.

Voicebox represents an important step forward in our generative AI research, and we look forward to continuing to explore the audio field and seeing how other researchers build on our work.