MusicInfuser: A Model for Generating Dance Videos from Music and Text 134

小编 · 2025-03-22

MusicInfuser is a project that aligns text-to-video diffusion models with music, enabling the generation of dance videos from both music and text. By introducing a music-video cross-attention mechanism, it synchronizes generated videos with musical rhythm.

MusicInfuser: A Model for Generating Dance Videos from Music and Text

What is MusicInfuser?

MusicInfuser is a project that aligns a text-to-video diffusion model with music, capable of generating dance videos based on music and text. By introducing a music-video cross-attention mechanism, the generated videos can be synchronized with the music rhythm. MusicInfuser: A model that can generate dance videos based on music and text.jpg

MusicInfuser Features

Music-driven dance video generation: Generates high-quality dance videos based on input music and text prompts.
Flexible text prompts: Users can specify dance style, scene, dancer characteristics, etc. through text prompts.
Long sequence generation support: Although the model is initially trained on 73-frame sequences, it can be extrapolated to generate longer video sequences.
Random seed control: By setting a random seed, users can generate different dance results.

MusicInfuser Applications

Creative video production: Provides video creators with a music-driven dance video generation tool, improving content creation efficiency.
Entertainment and performance: Used to generate personalized dance performance videos, applicable to virtual reality, augmented reality, and other scenarios.
Education and training: Helps dance learners better understand the coordination of dance movements with music rhythm through generated videos.

How to Use MusicInfuser

1. Installation

1. Clone the repository:

git clone https://github.com/SusungHong/MusicInfuser
cd MusicInfuser

2. Create and activate a Conda environment:

conda create -n musicinfuser python=3.10
conda activate musicinfuser

3. Install dependencies:

pip install -r requirements.txt
pip install -e ./mochi --no-build-isolation

4. Download model weights:

python ./music_infuser/download_weights.py weights/

2. Inference (Generate Video)

Run the following command to generate a video:

python inference.py --input-file {path to MP3 or MP4 file} \
                    --prompt {text prompt} \
                    --num-frames {number of frames to generate}

input-file: The input file (MP3 or MP4) from which to extract audio.
prompt: The text prompt for generating the dance. The more specific the prompt, the better the generated results usually are, but being overly specific may reduce the impact of the audio. The default value is "a professional female dancer dancing K-pop in an advanced dance setting in a studio with a white background, captured from a front view".
num-frames: The number of frames to generate. The default value is 145.

Other optional parameters:

seed: Random seed used to control the randomness of generation results. The default value is None.
cfg-scale: Classifier-free guidance (CFG) scale for the text prompt. The default value is 6.0.

3. Training

1. Data preprocessing:

bash music_infuser/preprocess.bash -v {dataset path} -o {output directory for processed videos} -w {path to pretrained Mochi model} --num_frames {number of frames}

2. Run training:

bash music_infuser/run.bash -c music_infuser/configs/music_infuser.yaml -n 1

Note: The current implementation only supports single-GPU training, and approximately 80GB of video memory is required to train a 73-frame sequence.