**MuScriptor: Open-Source Multi-Instrument Music Transcription**
Automatic Music Transcription (AMT) converts audio recordings into symbolic notation, typically MIDI. While single‑instrument transcription has become reliable, separating and notating full multi‑instrument mixes remains challenging. To address this, Kyutai and Mirelo have released **MuScriptor**, an open‑weight model trained on real multi‑instrument recordings across many genres.
This article explains how MuScriptor works, what benchmark results show, and how to run it.
—
### What is MuScriptor?
At its core, MuScriptor is a decoder‑only Transformer for music transcription. It takes a short‑segment mel‑spectrogram as input and autoregressively predicts MIDI‑like tokens for pitch, timing, and instrument, following the MT3 tokenization scheme.
The release provides three model sizes on Hugging Face:
– **small** (103M parameters)
– **medium** (307M parameters, default)
– **large** (1.4B parameters)
The inference code is MIT‑licensed, while the weights are under CC BY‑NC 4.0, which restricts commercial use.
—
### How the Three‑Stage Training Pipeline Works
MuScriptor’s main innovation is data‑centric rather than architectural. Training proceeds in three stages:
1. **Pre‑training on synthetic data**
Roughly 1.45 million MIDI files are synthesized on the fly with extensive augmentations—pitch shifting, tempo changes, velocity adjustment, and instrument randomization. Over 250 soundfonts and random detuning create near‑infinite audio realizations.
2. **Fine‑tuning on real recordings**
An internal dataset of 170,000 recordings, totaling over 11,000 hours, provides aligned note annotations. Alignment uses audio‑symbolic synchronization with interpolation and dynamic time warping, followed by filtering based on warping distance.
3. **Reinforcement learning (RL) post‑training**
300 manually verified tracks are used for GRPO‑style RL, combining REINFORCE with group‑relative advantage normalization. The reward function sums onset, frame, and offset F‑scores to encourage cleaner transcriptions.
—
### Performance Results
Evaluation uses the **DTest** set, consisting of 372 held‑out tracks with precise annotations, reporting instrument‑agnostic metrics from `mir_eval`. The Multi F1 metric is the strictest, as it requires both correct timing and instrument labeling.
| Model (DTest) | Onset F1 | Frame F1 | Offset F1 | Drums F1 | Multi F1 |
|—————|———-|———-|———–|———-|———-|
| YourMT3+ (baseline) | 32.5 | 45.5 | 17.8 | 41.4 | 21.9 |
| MuScriptor · DSynth | 34.5 | 48.9 | 16.1 | 21.0 | 16.2 |
| MuScriptor · DSynth + DReal | 54.4 | 69.3 | 42.3 | 43.3 | 41.6 |
| MuScriptor · DSynth + DReal + DRL | **60.4** | **73.3** | **49.0** | **50.2** | **48.2** |
Each training stage improves results. Real‑data fine‑tuning alone boosts all metrics by roughly 20 points over synthetic‑only training, while RL post‑training further reduces false negatives and sharpens onset timing. Cross‑dataset tests, such as on Dagstuhl ChoirSet, show similar gains, though onset and offset remain challenging for chorals and similar styles.
—
### Getting Started
Installation and inference are straightforward:
“`bash
# pip install muscriptor (or: uv add muscriptor)
from pathlib import Path
from muscriptor import TranscriptionModel
# Downloads the default “medium” variant (also accepts “small” / “large”)
model = TranscriptionModel.load_model()
# Stream note events; optionally condition on known instruments
for event in model.transcribe(“audio.wav”, instruments=[“acoustic_piano”, “drums”]):
print(event) # NoteStartEvent / NoteEndEvent / ProgressEvent
# Or write a MIDI file directly
Path(“out.mid”).write_bytes(model.transcribe_to_midi(“audio.wav”))
“`
The default model is already RL post‑tuned, so `cfg_coef` should remain at 1. Running `uv muscriptor serve` launches a browser‑based web UI with a live piano roll.
—
### Use Cases
Because MuScriptor outputs standard MIDI, it supports a wide range of workflows:
– Producers can extract and rewrite MIDI basslines from mixes.
– Musicologists can convert historical recordings into editable scores.
– MIR researchers can feed transcriptions into chord or key recognition systems.
– Educators can display live piano rolls during playback for teaching.
– Developers can condition transcription on specific instruments, such as drums only.
—
### Strengths and Weaknesses
**Strengths**
– Trained on 170k real recordings spanning classical to heavy metal.
– Open weights plus MIT‑licensed inference code in three size variants.
– Multi F1 of 48.2 versus 21.9 for the YourMT3+ baseline on DTest.
– Instrument conditioning customizes output and stabilizes cross‑segment predictions.
– Streaming API and web UI for live transcription and visualization.
**Weaknesses**
– Weights use CC BY‑NC 4.0, limiting commercial deployment.
– Tokenizer does not capture velocity and cannot represent overlapping notes at the same pitch and instrument.
– Onset and offset accuracy remain lower for chorals and similar styles.
– The large model requires a GPU for practical use.
– The 5‑second segment size limits long‑range context and inference speed.
—
For more details, check out the paper, GitHub repo, and model weights. You can also follow the authors on Twitter and join their Subreddit and Newsletter for updates.
