M2UGen: A Multimodal Music Understanding and Generation Framework 124

Tencent has released the multimodal music generation model M2UGen, which supports generating music from images and videos. M2UGen is a trend-setting framework that integrates music understanding and multimodal music generation tasks, aiming to assist users in music art creation. With its powerful capabilities, M2UGen provides a comprehensive music generation and editing experience.

M2UGen demo address:https://huggingface.co/M2UGen

M2UGen project address:https://github.com/shansongliu/M2UGen

M2UGen paper address:https://arxiv.org/abs/2311.11255

The M2UGen model is a music understanding and generation model capable of music Q&A and music generation from text, images, video, and audio, as well as music editing. The model leverages encoders such as MERT for music understanding, ViT for image understanding, and ViViT for video understanding, along with the MusicGen/AudioLDM2 model as the music generation model (music decoder), as well as adapters and the LLaMA 2 model.

Music Generation and Understanding

We introduce the M2UGen framework, which can simultaneously include music understanding and multimodal music generation tasks, aiming to assist users in music-related art creation.

Music-Oriented Dataset

We propose a systematic method for generating a large music-oriented multimodal instruction dataset, using the MU-LLaMA model and MosaicML's MPT-7B-Chat model to generate music-oriented data.

We also utilize the BLIP image captioning model and the VideoMAE captioning model to add captions for their respective modalities.

In addition to generating music from text, it also supports generating music from images, video, and audio, and can also edit existing music. The project utilizes encoders like MERT for music understanding, ViT for image understanding, and ViViT for video understanding, and uses the MusicGen/AudioLDM2 model as the music generation model (music decoder). Users can easily remove or replace specific instruments, and adjust the tempo and speed of the music. This enables users to create music that matches their unique creative vision.

In addition, M2UGen incorporates adapters and the LLaMA2 model, giving the model a variety of capabilities.