NotaGen: AI Music Model Specialized in High-Quality Classical Sheet Music Generation 122

What is NotaGen?

NotaGen is an AI music generation model jointly developed by the Central Conservatory of Music, Beihang University, and Tsinghua University, focused on generating high-quality classical sheet music. By mimicking the training paradigm of large language models (LLMs), it is capable of generating high-quality classical scores.

Core Features of NotaGen

High-Quality Classical Score Generation

• NotaGen can generate scores in specific styles based on user-specified periods (e.g., Baroque, Classical, Romantic), composers (e.g., Bach, Mozart, Chopin), and instruments (e.g., keyboard, string quartet, orchestra).

Advanced Training Paradigm

• Pretraining: Pretrained on over 1.6 million musical pieces to learn the basic structures and patterns of music.

• Fine-tuning: Fine-tuned on approximately 9,000 high-quality classical works to enhance the understanding of specific styles and composers.

• Reinforcement Learning: Optimizes the quality and controllability of generated music through the CLaMP-DPO method, without requiring manual annotation or predefined rewards.

Multimodal Data Representation

• NotaGen uses Interleaved ABC Notation, which rearranges notes from different parts onto a single line and distinguishes them using the voice indicator [V:]. This representation can flexibly handle multi-track music and complex scores.

Model Architecture

• NotaGen adopts the Tunesformer architecture, which contains two layers of GPT-2 decoders: a patch-level decoder and a character-level decoder. The patch-level decoder captures temporal relationships between patches, and then the hidden states are passed to the character-level decoder to autoregressively predict the characters of the next patch.

Generated Samples

• NotaGen provides generated samples in a variety of styles, including keyboard, chamber music, orchestral, art songs, choral, and vocal-orchestral works. In addition, it has also attempted to generate music in popular styles.

NotaGen Technology

• Data Representation: Adopts interleaved ABC notation, supporting multi-track music and complex scores.

• Model Architecture: Based on Tunesformer, containing patch-level and character-level decoders.

• Training Paradigm:

• Pretraining: Pretrained on 1.6 million musical pieces.

• Fine-tuning: Fine-tuned on 9,000 high-quality classical scores, covering 152 composers.

• Reinforcement Learning: Optimizes the quality and controllability of generated music through the CLaMP-DPO method.

NotaGen Experiments and Evaluation

• Subjective A/B Testing: NotaGen outperforms baseline models in the quality of generated music, approaching the level of human composition.

• CLaMP-DPO Method: Optimizes the quality of generated music through feedback provided by the contrastive learning model CLaMP², without requiring manual annotation or predefined rewards.

NotaGen Use Cases

• Professional Composers: Serves as an inspiration tool to quickly generate high-quality scores.

• Music Enthusiasts: Lowers the barrier to music creation and provides personalized creative support.

• Music Education: Serves as a teaching tool to help students understand the characteristics of music from different styles and periods.

Project Page: https://electricalexis.github.io/notagen-demo

GitHub Repository: https://github.com/ElectricAlexis/NotaGen

Paper: https://arxiv.org/pdf/2502.18008