MuScriptor: An Open Model for Multi-Instrument Music Transcription
Simon Rouard, Michael Krause, Axel Roebel, Carl-Johann Simon-Gabriel, Alexandre Défossez
MuScriptor is an open-weight transformer for multi-instrument music transcription trained on large-scale real-world audio.
How can we build a robust, multi-instrument music transcription model that generalizes from synthetic data to complex, real-world audio?
Automatic Music Transcription (AMT) systems struggle to generalize because they rely on synthetic data that fails to capture the acoustic complexity of real-world, multi-instrument music productions. The authors train a decoder-only transformer on a massive dataset of 170,000 real-world recordings, using synthetic data only for pre-training and applying reinforcement learning to align the model with high-quality human transcriptions. This approach significantly outperforms existing baselines, achieving substantial gains in frame and multi-instrument F1 scores across diverse musical genres.
Paper Primer
The core mechanism hinges on a three-stage training pipeline: large-scale synthetic pre-training, supervised fine-tuning on real-world audio, and post-training via Group Relative Policy Optimization (GRPO). The model treats transcription as a language-modeling task, predicting a stream of MIDI-like tokens conditioned on both the audio spectrogram and a provided list of active instruments.
Real-world data fine-tuning is the primary driver of transcription quality.
Comparing models trained on synthetic data versus those fine-tuned on the 170k-recording real-world dataset. Fine-tuning on real data improves all metrics by approximately 20 percentage points.
MuScriptor significantly improves cross-domain generalization compared to existing state-of-the-art models.
Evaluation on the Dagstuhl ChoirSet benchmark. Frame F1 score increased from 51.0 (YourMT3+) to 80.7.
Why is instrument conditioning included as an input?
It allows users to customize transcription output and stabilizes predictions across segment boundaries, preventing the model from fluctuating in its instrument assignments.
How does this model differ from previous sequence-to-sequence AMT approaches like MT3?
While previous models relied heavily on synthetic data mixtures, MuScriptor prioritizes scale and diversity in real-world training data and introduces reinforcement learning to align outputs with high-quality human annotations.
For researchers and musicians, MuScriptor provides a robust, open-weight tool that bridges the gap between synthetic-trained models and the acoustic realities of professional multi-instrument music.
Motivation and Problem Framing
We expose the generalization gap between synthetic pre‑training and real‑world multi‑instrument transcription and outline our approach to bridge it.
Automatic Music Transcription (AMT) aims to turn audio recordings into symbolic scores, but current systems falter when faced with the rich timbral variety and overlapping notes of real multi‑instrument music.
The core problem is a generalization gap: models trained on large synthetic MIDI corpora do not transfer well to authentic recordings that contain diverse instruments and production effects.
**Figure 2.** Instrument frequency across $\mathcal{D}_{\text{Real}}$ (grouped according to `MT3_FULL_PLUS` schema [13]).
**Figure 3.** Distribution of active instruments per track across datasets. Boxes span the interquartile range (p25–p75) with median line; whiskers extend to the 5th and 95th percentiles. Diamond markers indicate the mean.
The generalization gap between synthetic and real‑world music data is the primary obstacle to reliable multi‑instrument transcription.
Evolution of Transcription Models
Survey of Automatic Music Transcription approaches and baselines.
Automatic Music Transcription (AMT) has progressed from instrument‑specific pipelines to unified sequence models, exposing a trade‑off between architectural sophistication and data diversity.
Probabilistic state‑space models that treat note onsets as hidden states and audio frames as observations, enabling Viterbi decoding of pitch sequences.
Decomposes a magnitude spectrogram into additive basis spectra and activation patterns, interpreting bases as instrument timbres and activations as note events.
Frames pitch detection as a binary classification problem per time‑frequency bin, using kernel‑based SVMs to separate note versus non‑note regions.
Uses RNNs (often LSTMs) to map sequences of audio frames to pitch probabilities, capturing temporal context through hidden states.
Applies 2‑D convolutions over mel‑spectrograms to classify pitch presence per frame, exploiting local time‑frequency patterns.
Jointly predicts note onset probabilities and framewise sustain activations, treating onset detection as a separate but related task to improve temporal precision.
Recasts transcription as a language‑modeling problem: the model consumes a mel‑spectrogram and autoregressively emits a stream of discrete MIDI‑like tokens (pitch, onset, duration, instrument).
MT3 is an encoder‑decoder Transformer that ingests a mel‑spectrogram and autoregressively outputs a flat sequence of MIDI‑like tokens, each encoding pitch, onset time, duration, and instrument class.
Enhances the MT3 encoder with a hierarchical attention transformer operating in the time‑frequency domain, adds a Mixture of Experts (MoE) layer, and applies cross‑dataset stem augmentation.
Replaces the spectrogram encoder in YourMT3+ with MusicFM, a self‑supervised audio foundation model, aiming to leverage large‑scale unsupervised pre‑training.
Attempts a universal AMT system trained on a modest dataset, but releases no pretrained weights and operates at a smaller scale than MT3.
Investigates the effect of large‑scale synthetic piano data for pre‑training, showing improvements on piano transcription but not extending to multi‑instrument scenarios.
Applies a reinforcement‑learning objective that optimizes groups of transcription policies relative to each other, aligning model outputs with high‑quality transcriptions beyond teacher‑forcing.
The MuScriptor Architecture
Method details the multi‑stage training pipeline, instrument conditioning, and RL fine‑tuning.
The main obstacle for multi‑instrument Automatic Music Transcription is the generalization gap between synthetic pre‑training and real‑world performance. To bridge this gap we adopt a three‑stage pipeline: large‑scale synthetic pre‑training, targeted real‑music fine‑tuning, and reinforcement‑learning post‑training, all while optionally conditioning on the instruments present.
The model generates a sequence of MIDI‑like tokens one after another, each token conditioned on all previously generated tokens and the audio context.
How does autoregressive generation differ from frame‑wise classification?
Frame‑wise models predict a pitch label for each time step independently, which can break musical continuity. Autoregressive transcription conditions each prediction on the entire history of previous tokens, preserving note‑level structure and allowing the model to enforce musical constraints such as note duration and instrument consistency.
We first expose the model to an effectively infinite synthetic audio distribution, then specialize it on a smaller real‑music corpus, and finally refine it with reinforcement learning that directly optimizes transcription metrics.
Step 1: Randomly augment and synthesize each of the 1 000 synthetic excerpts, producing 1 000 distinct audio–MIDI pairs.
Step 2: Train the decoder‑only Transformer on these pairs for 2 000 gradient updates, learning a generic mapping from mel‑spectrograms to token sequences.
Step 3: Switch to the real set of 200 excerpts, continue training for 500 updates to adapt to authentic timbres.
Step 4: Sample 8 candidate transcriptions per audio segment from the fine‑tuned model (temperature = 0.75) on the 50 RL excerpts.
Step 5: Compute the reward as the sum of Onset F1, Offset F1, and Frame F1 for each candidate, then calculate within‑group advantages.
Step 6: Update the model parameters with the REINFORCE objective weighted by the advantages, completing 100 RL updates.
This toy schedule shows how each stage contributes a distinct adaptation: synthetic pre‑training builds a broad foundation, real fine‑tuning narrows the domain, and RL aligns the model with the final evaluation criteria.
Why not train directly on the real dataset without the synthetic pre‑training stage?
Training only on the 200 real excerpts would severely limit exposure to diverse musical contexts, leading to overfitting and poor generalization. The synthetic stage supplies a vast variety of pitch, tempo, and instrument combinations, allowing the model to learn robust acoustic‑symbolic mappings before being specialized on the smaller real set.
Before the Transformer processes the mel‑spectrogram, we prepend embeddings that encode the set of instruments present in the full track, giving the model a global awareness of which timbres it may need to predict.
How does instrument conditioning differ from simply feeding a one‑hot instrument vector at the output layer?
Appending the instrument embeddings as a prefix lets the self‑attention layers integrate instrument context throughout the entire decoding process, influencing token generation at every step. A one‑hot vector applied only at the final linear projection would affect the output distribution but could not guide the intermediate representations that shape note timing and articulation.
Set the model to evaluation mode and sample G = 8 candidate transcriptions per audio segment using temperature $\tau$ = 0.75.
Compute the reward for each candidate as the sum of Onset F1, Offset F1, and Frame F1 against the ground‑truth MIDI.
Standardise the rewards within each segment to obtain advantages ĤA.
Switch the model back to training mode and update parameters with the REINFORCE loss weighted by the advantages.
Repeat for all batches of the DRL dataset.
RL fine‑tuning on the DRL dataset.
Performance and Ablation Studies
MuScriptor narrows the AMT generalization gap by staged training and instrument‑aware conditioning.
Fine‑tuning on real data and a reinforcement‑learning post‑training boost AMT performance by roughly +20 pp across all metrics.
Table 1 shows each training stage (synthetic‑only → +real → +RL) improves over the previous one, with the best model attaining the highest scores.
**Figure 1.** Piano roll for the guitar notes in a song from our test set $D_{Test}$. Blue indicates true positives (a note was correctly detected by the model), green indicates false negatives (the model has missed a note), and red indicates false positives (the model predicted a wrong note). Vertical lines indicate the segments on which the model is evaluated.
**Figure 4.** Impact of pre-training on $\mathcal{D}_{Synth}$ across different sizes of the fine-tuning set $\mathcal{D}_{Real}$ ($\alpha_{CFG} = 2$).
Questions & answers
What is MuScriptor and what is its main contribution?
MuScriptor is an open-weight, decoder-only transformer model for multi-instrument Automatic Music Transcription (AMT) that bridges the generalization gap between synthetic-trained models and real-world music by training on 170,000 real-world recordings and applying reinforcement learning to align outputs with high-quality human transcriptions.
What problem does MuScriptor address?
MuScriptor addresses the generalization gap in AMT systems, which struggle because they rely on synthetic data that fails to capture the acoustic complexity—rich timbral variety and overlapping notes—of real-world, multi-instrument music productions.
Why does the generalization gap matter for music transcription?
Current AMT systems trained primarily on synthetic data falter when faced with the timbral variety and overlapping notes of real multi-instrument music, making reliable transcription of professional recordings an unsolved problem.
What is the three-stage training pipeline used by MuScriptor?
MuScriptor uses large-scale synthetic pre-training to learn diverse acoustic-symbolic mappings, followed by supervised fine-tuning on real-world audio, and finally post-training via Group Relative Policy Optimization (GRPO) to align the model with high-quality human transcriptions.
How does MuScriptor treat the transcription task technically?
MuScriptor treats transcription as a language-modeling task, predicting a stream of MIDI-like tokens conditioned on both the audio spectrogram and a provided list of active instruments using a decoder-only transformer architecture.
What role does reinforcement learning play in MuScriptor's training?
Reinforcement learning via Group Relative Policy Optimization (GRPO) is applied as a post-training stage to align the model's outputs with high-quality human annotations, going beyond what supervised fine-tuning alone achieves.
Why is instrument conditioning included as an input to MuScriptor?
Instrument conditioning allows users to customize transcription output and stabilizes predictions across segment boundaries, preventing the model from fluctuating in its instrument assignments during decoding.
How does instrument conditioning work architecturally in MuScriptor?
Instrument embeddings are appended as a prefix so that the self-attention layers integrate instrument context throughout the entire decoding process, influencing token generation at every step rather than only at the final output projection.
How does autoregressive generation in MuScriptor differ from frame-wise classification?
Frame-wise models predict a pitch label for each time step independently, which can break musical continuity, whereas MuScriptor's autoregressive approach conditions each prediction on the entire history of previous tokens, preserving note-level structure and enforcing musical constraints such as note duration and instrument consistency.
Why does MuScriptor use synthetic pre-training if real-world data is the priority?
Training only on the approximately 200 real excerpts used for fine-tuning would lead to overfitting and poor generalization; the synthetic stage supplies a vast variety of pitch, tempo, and instrument combinations so the model learns robust acoustic-symbolic mappings before specialization on real data.
What datasets or data sources does MuScriptor use?
MuScriptor is trained on 170,000 real-world recordings for fine-tuning, with synthetic data used only for pre-training; the fine-tuning set includes approximately 200 real excerpts with high-quality human transcriptions, though the paper does not specify the names of all individual datasets.
What are MuScriptor's key results?
MuScriptor significantly outperforms existing baselines, achieving substantial gains in frame F1 and multi-instrument F1 scores across diverse musical genres; the paper does not provide specific numeric values in the sections available.
How does MuScriptor differ from prior sequence-to-sequence AMT models like MT3?
Unlike MT3 and similar models that relied heavily on synthetic data mixtures, MuScriptor prioritizes scale and diversity in real-world training data and introduces reinforcement learning (GRPO) to align outputs with high-quality human annotations.
What evaluation metrics are used to assess MuScriptor?
MuScriptor is evaluated using frame F1 score and multi-instrument F1 score across diverse musical genres, though the paper does not detail the specific benchmark datasets used for evaluation in the available sections.
What are the limitations of MuScriptor as described in the paper?
The paper does not explicitly enumerate limitations, but it acknowledges that training only on the small set of real excerpts without synthetic pre-training would cause overfitting, implying the model's performance is partly dependent on the quality and coverage of its large-scale training data.
Is MuScriptor open-weight and reproducible?
Yes, MuScriptor is described as an open-weight model, making it accessible to researchers and musicians; however, the paper does not specify a repository URL or detailed reproduction instructions in the available text.
Who are the intended users of MuScriptor?
MuScriptor is intended for researchers and musicians who need a robust tool for multi-instrument music transcription that generalizes to the acoustic realities of professional recordings.
Where and when was MuScriptor published?
MuScriptor is available as an arXiv preprint at arxiv.org/abs/2607.08168; the paper does not specify a conference or journal venue in the available text.
Key terms
- Automatic Music Transcription (AMT)
- The computational task of converting an audio recording into a symbolic musical representation such as a MIDI file or sheet music score.
- decoder-only transformer
- A neural network architecture that generates output tokens one at a time, each conditioned on all previously generated tokens, without a separate encoder module.
- GRPO (Group Relative Policy Optimization)
- A reinforcement learning algorithm used to fine-tune a model by rewarding outputs that better match a reference quality standard, here applied to align transcriptions with human annotations.
- MIDI-like tokens
- A discrete symbolic representation of music events (such as note-on, note-off, pitch, and instrument) encoded as a sequence of tokens analogous to words in a language model.
- frame F1 score
- An evaluation metric that measures the harmonic mean of precision and recall for correctly predicting which pitches are active at each individual time frame of audio.
- multi-instrument F1 score
- An evaluation metric that assesses transcription accuracy across multiple instruments simultaneously, penalizing both missed notes and incorrectly assigned instrument labels.
- instrument conditioning
- Providing the model with information about which instruments are present in a recording as an additional input, so it can tailor its transcription output accordingly.
- autoregressive transcription
- A transcription approach where each predicted musical token depends on all previously predicted tokens, allowing the model to maintain consistency in note structure and instrument assignments over time.
- frame-wise classification
- A transcription approach that independently predicts which pitches are active at each time step without considering the sequence of prior predictions.
- synthetic pre-training
- An initial training phase using artificially generated audio-score pairs to expose the model to a wide variety of musical conditions before fine-tuning on real recordings.
- supervised fine-tuning (SFT)
- A training stage in which a pre-trained model is further trained on a smaller, labeled dataset—here real-world recordings with human transcriptions—to specialize its capabilities.
- audio spectrogram
- A visual or numerical representation of how the frequency content of an audio signal changes over time, used here as the input feature for the transcription model.
- generalization gap
- The performance difference between a model trained on one type of data (e.g., synthetic) and its ability to perform well on a different, more complex type of data (e.g., real-world recordings).
- MT3
- A prior sequence-to-sequence AMT model that the paper references as a baseline, notable for relying heavily on synthetic data mixtures for training.
- open-weight model
- A machine learning model whose trained parameters are publicly released, allowing others to download, use, and build upon it without restriction.