ACE-Step: A New Open-Source Foundation Model for Music Generation 61

What is ACE-Step?

ACE-Step is a new open-source music generation foundation model jointly developed by ACE Studio and StepFun. It overcomes the limitations of existing music generation methods in terms of generation speed, musical coherence, and controllability by integrating diffusion-based generation techniques, Sana's deep compression autoencoder, and lightweight linear transformers. The model can synthesize up to 4 minutes of music within 20 seconds on an A100 GPU, making it 15 times faster than LLM-based baselines. It also achieves better musical coherence and lyric alignment across melody, harmony, and rhythm metrics, and supports advanced control mechanisms such as voice cloning and lyric editing. At the same time, the paper lists limitations such as output inconsistency and poor performance in certain styles.

ACE-Step Core Architecture

ACE-Step achieves fast convergence by integrating diffusion models, deep compression autoencoders, and lightweight linear transformers, while leveraging MERT and m-hubert to align semantic representations (REPA) during training. This architectural design strikes a balance between generation speed, musical coherence, and controllability.

ACE-Step Advantages

• Generation Speed: ACE-Step can generate up to 4 minutes of music in just 20 seconds on an NVIDIA A100 GPU, making it 15 times faster than large language model (LLM)-based baseline methods.

• Musical Coherence: The model demonstrates superior musical coherence and lyric alignment across melody, harmony, and rhythm metrics.

• Detail Preservation: ACE-Step preserves fine-grained acoustic details and supports advanced control mechanisms such as voice cloning, lyric editing, remixing, and track generation.

ACE-Step Features

• Diverse Styles: Supports mainstream music styles and handles multiple description formats, including short tags, descriptive text, or application scenario descriptions. It also performs well on experimental input tests and can generate instrumental tracks with multiple instruments.

• Multilingual Support: Supports 19 languages, though some languages perform poorly due to data imbalance. The top 10 best-performing languages include English, Chinese, Russian, and others.

• Strong Controllability:

• Variations Generation: Uses training-free inference-time optimization techniques to generate variations with different similarity to the original track by adjusting the mixing ratio of initial noise and added Gaussian noise.

• Repaint: By adding noise to the target audio input and applying masking constraints during the ODE process, specific parts can be modified. It can also be combined with variations generation techniques to achieve local style, lyric, or vocal changes.

• Edit: Uses flow-edit technology to modify lyrics without changing the melody, vocal timbre, or background music, though only small sections of lyrics can be modified at a time.

ACE-Step Application Scenarios

• Lyric2Vocal (LoRA): LoRA fine-tuned on pure vocal data, capable of generating vocal samples directly from lyrics, used for vocal demonstrations, songwriting assistance, and more.

• Text2Samples (LoRA): LoRA fine-tuned on pure instrument and sample data, capable of generating conceptual music production samples from text descriptions, used for creating instrument loops, sound effects, and more.

ACE-Step Limitations

• Inconsistent Output: Sensitive to random seeds and input duration, resulting in unstable outputs.

• Uneven Style Performance: Performs poorly in certain specific styles (such as Chinese rap).

• Continuity Issues: Transitions during repainting or extension operations are unnatural.

• Vocal Quality Needs Improvement: Synthesized vocals are relatively rough and lack detail.

• Insufficient Control Granularity: Requires more fine-grained control over music parameters.

• Multilingual Lyric Optimization: Needs to improve support for multilingual lyrics, enhancing accuracy and naturalness.

ACE-Step Frequently Asked Questions

What is ACE-Step's biggest advantage compared to other music generation models?

Answer: ACE-Step's biggest advantage is that it resolves the trade-off between generation speed, musical coherence, and controllability that exists in current models. It can synthesize up to 4 minutes of music within 20 seconds on an A100 GPU, making it 15 times faster than LLM-based baseline models, while achieving better musical coherence and lyric alignment across melody, harmony, and rhythm metrics. It also features powerful controllability functions such as voice cloning and lyric editing—capabilities that are difficult for other models to achieve simultaneously.

What issues does ACE-Step have with multilingual support?

Answer: Although ACE-Step supports 19 languages, less common languages perform poorly due to data imbalance. When handling multilingual lyrics, accuracy and naturalness need improvement, and further optimization of multilingual lyric support is required to enhance the model's overall performance across different languages.

In ACE-Step's application scenarios, what creative stages are Lyric2Vocal and Text2Samples suitable for respectively?

Answer: Lyric2Vocal is LoRA fine-tuned on pure vocal data, suitable for vocal demonstrations in songwriting, helping creators quickly test how lyrics sound when sung. It can also be used to create guide tracks, assist songwriting, and conduct vocal arrangement experiments. Text2Samples is LoRA fine-tuned on pure instrument and sample data, suitable for rapidly creating instrument loops, sound effects, and various music elements in the early stages of music production, providing conceptual samples for music production and improving creative efficiency.

Related Links

https://ace-step.github.io/