TangoFlux : modèle de génération d'effets sonores à partir de texte (TTA) 110

TangoFlux is a high-performance text-to-audio (TTA) generation model jointly developed by the Singapore University of Technology and Design (SUTD) and NVIDIA. It has approximately 515 million parameters and can generate up to 30 seconds of 44.1 kHz stereo audio in just 3.7 seconds on a single A40 GPU. TangoFlux can not only generate sound effects such as bird calls, whistles, and explosions, but also produce music.

TangoFlux Technical Principles

• Variational Autoencoder: Uses a VAE to encode audio waveforms into a latent representation, then reconstructs the original audio from this latent representation.

• Text and Duration Embeddings: Controls the content and duration of the generated audio based on text and duration embeddings.

• FluxTransformer Architecture: Combines the Diffusion Transformer (DiT) and Multimodal Diffusion Transformer (MMDiT) to process text prompts and generate audio.

• Flow Matching: Learns a mapping from a simple prior distribution to a complex target distribution to generate samples.

• CLAP-Ranked Preference Optimization: Optimizes audio alignment based on iteratively generated preference data pairs.

TangoFlux Main Features

• Rapid Audio Generation: TangoFlux can generate high-quality audio content in a very short time, making it suitable for application scenarios that require fast response.

• High-Quality Audio Output: The generated audio is of high quality and can clearly reproduce various sound events, making it suitable for music, sound effects, and many other types of audio generation.

• Support for Long Audio Generation: The model is capable of handling audio generation tasks up to 30 seconds long, making it suitable for producing longer audio content.

• Diverse Sound Effect Generation: TangoFlux can not only generate music but also a wide variety of sound effects such as bird calls, whistles, and explosions, suitable for a range of creative and entertainment projects.

TangoFlux Application Scenarios

• Film and Game Audio Production: TangoFlux can quickly generate realistic sound effects, scores, and voiceovers based on scripts or scene descriptions, shortening production cycles, reducing costs, improving the audio quality of the work, and enhancing the interactivity and immersion of games.

• Source of Inspiration for Music Creation: It can generate diverse musical snippets according to the creator's settings, helping to overcome creative bottlenecks, promoting musical innovation, and bringing more novel works.

• Enhanced VR and AR Experiences: Generates realistic real-time audio based on the virtual environment and user interactions, enhancing the immersion of VR games and strengthening the perception of virtual-real fusion in AR applications.

• Audio Innovation in Advertising and Marketing: Generates unique audio based on advertising ideas and brand image to attract consumers, evoke resonance, and enhance brand awareness and reputation.

TangoFlux Project Resources

• Official website: tangoflux.github.io

• arXiv technical paper: https://export.arxiv.org/pdf/2412.21037

TangoFlux also employs the CLAP-Ranked Preference Optimization (CRPO) framework, which improves the model's audio alignment capabilities through iteratively generating and optimizing preference data. This approach ensures not only the quality of the generated audio but also better alignment with user preferences and the intent of the input text. In addition, TangoFlux's training is based on non-proprietary datasets, making the model more open and accessible, and facilitating further research and applications.

TangoFlux is mainly used for multimedia content creation, such as generating background music, sound effects, and voiceovers in film, gaming, advertising, and video production, thereby improving production efficiency and reducing costs. Its technical principles include the Variational Autoencoder (VAE), text and duration embeddings, and the FluxTransformer architecture, which together support its highly capable audio generation.

Related Information:

Text-to-Speech Software