YUE2: Unifying Symbolic and Audio Music Generation at Frontier Quality

Ruibin Yuan, Jiahao Pan, Junyan Jiang, Zhiyue Wu, Ziya Zhou, Jiankai Sun, Yizhi Li, Ge Zhang, Yicheng Gu, Zeyue Tian, Junyu Dai, Hanfeng Lin, Kai Li, Shangda Wu, Xuanjie Liu, Jiaming Wang, Zihan Liu, Yue Wang, Yinghao Ma, Hanzhi Yin, Kangrui Chen, Xinyue Zhang, Ziyang Ma, Mengqi Liao, Hejia Zhao, Guowei Huang, Chao Yan, Lei Ke, Jianwei Yu, Bei Liu, Joe Guo, Liumeng Xue, Gus Xia, Wei Xue, Yike Guo

YuE2 unifies symbolic and audio music generation by using a generated score to plan melody and harmony before rendering audio.

How can we unify symbolic music composition and high-fidelity audio generation to enable controllable, frontier-quality song creation?

Audio models generate complete songs but leave composition implicit, making it difficult for users to edit specific musical elements like melody or harmony. YuE2 introduces symbolic planning: a single Mixture-of-Transformers model first writes a readable ABC-notation score, then expands it into semantic tokens and acoustic latents for full-song audio. On WildSongBench, this approach achieves frontier quality, with experts preferring planned generation over unplanned alternatives for overall musicality and quality.

Paper Primer

YuE2 implements "progressive musical commitment," where the model settles high-level compositional decisions before resolving acoustic details. The core mechanism is a 28-layer AR–NAR Mixture-of-Transformers (MoT) that acts as a unified engine: it autoregressively predicts a symbolic score and semantic tokens, then uses bidirectional flow matching to realize the acoustic latents.

Symbolic planning significantly improves perceived song quality and musicality.

Expert preference in A/B testing for planned versus unplanned generation. 49.3% preference for planning versus 34.6% for unplanned generation (overall quality).

To enable this, the authors developed MERT2 and SheetSage2 to extract symbolic and semantic supervision from raw recordings. This allows the model to learn the hierarchy of music—from lead sheets to acoustic signals—without requiring manually aligned training data.

Why use a unified Mixture-of-Transformers instead of the common LM+DiT architecture?

The unified MoT allows the model to learn semantic-token prediction and acoustic generation in one backbone, which experts prefer over separate language model and diffusion Transformer architectures for overall quality, vocals, and accompaniment.

Does the generated score actually influence the final audio, or is it just a label?

The score is functional; experiments show that corresponding recordings achieve high melody and chord sequence similarity (0.9464 and 0.9246) to the generated plan, and local edits to the score successfully change the targeted musical content while preserving unedited sections.

YuE2 demonstrates that explicit symbolic planning is a viable path to high-fidelity, editable music generation, effectively bridging the gap between readable composition and finished audio.

System Overview and Contributions

YuE2 unifies symbolic planning and audio generation, enabling editable scores and zero‑shot cover creation.

We present three core contributions. First, YuE2 unifies symbolic and audio music generation through a symbolic planning layer, reaching frontier full‑song quality while supporting controlled editing and zero‑shot cover generation. Second, we demonstrate that symbolic planning improves perceived quality and that experts prefer a unified AR‑NAR mixture‑of‑Transformers over separate language‑model and diffusion pipelines.

Symbolic planning is like a composer drafting a sheet‑music score before the orchestra performs it — the model first writes a readable, editable score, then generates high‑fidelity audio conditioned on that plan.

The AR stream emits a wrapped symbolic score, then the NAR stream emits dense semantic tokens conditioned on that completed score.

YuE2 enables controlled editing and zero‑shot cover generation.

Introduction

Introducing YuE2, which bridges symbolic planning and high-fidelity audio generation.

Most existing music generators sit at one end of the representation hierarchy: symbolic models expose melody and harmony but stop before producing a finished recording, while audio models output complete songs but hide the compositional structure. This asymmetry leaves a gap—there is no single system that can both make composition explicit and deliver frontier‑quality audio. Bridging that gap requires a model that can progressively commit decisions, from high‑level score to low‑level acoustic detail.

YuE2 is a single model that first writes a readable musical score (melody, chords, tempo, form) and then expands that score into semantic tokens and acoustic latents, producing a full‑song audio waveform.

**Figure 1.** **YuE2 achieves frontier song quality, competitive with the evaluated proprietary systems.** Automatic evaluation on WSB (192 prompts), including Suno v6 and v6 Wild [66]. The quality index combines standardized SongBench [77] and SongEval [86] Global Avg; alignment combines standardized MuQ-MuLan [91], AllMusicCaps [3], and our Qwen3-Omni [80] score. Bubble area: AudioBox production quality (PQ) [70]; black outlines: the observed Pareto front on these two indices. Both YuE2 settings use symbolic planning; Bo8 means best-of-8. Appendix A.4 gives index construction and weight sensitivity; Figure 7 reports expert preferences.

The core disconnect is that symbolic control has been decoupled from audio fidelity; YuE2 reunites them.

The AR-NAR Mixture-of-Transformers

Describes the AR‑NAR hybrid transformer that mixes causal and bidirectional streams in a single backbone.

Generating music needs both precise token ordering and global acoustic context. A single autoregressive model can only see past tokens, while a pure non‑autoregressive model lacks the causal structure needed for symbolic scores. The AR‑NAR Mixture‑of‑Transformers solves this tension by running causal and bidirectional experts side‑by‑side in each layer.

The model hosts two experts in every layer—one that predicts tokens step‑by‑step (AR) and one that predicts acoustic latents all at once (NAR)—while they both read from the same attention matrix.

AR‑causal entries: $M_{A_0,A_0}=1$, $M_{A_0,A_1}=0$ (future AR not visible); $M_{A_1,A_0}=1$, $M_{A_1,A_1}=1$ (self‑and‑past).

NAR‑bidirectional entries: all $M_{N_i,*}=1$ for $i\in\{0,1\}$, allowing each NAR position to attend to every other position (AR and NAR).

Resulting mask: $$ M=\begin{bmatrix} 1 & 0 & 1 & 1\\ 1 & 1 & 1 & 1\\ 1 & 1 & 1 & 1\\ 1 & 1 & 1 & 1 \end{bmatrix} $$ where rows correspond to [$A_0$, $A_1$, $N_0$, $N_1$].

This toy mask shows that AR tokens cannot see future AR tokens but can see all NAR latents, while NAR latents see the full sequence, enabling parallel acoustic updates after the symbolic plan is fixed.

How does this differ from training two separate Transformers—one AR and one NAR?

Separate models would duplicate the self‑attention computation and could not share contextual information. By keeping a single attention matrix, the AR and NAR experts reuse the same pairwise similarities, reducing compute and allowing the NAR stream to attend to the already‑generated symbolic tokens.

Per‑layer processing for the AR‑NAR Mixture‑of‑Transformers.

Training Procedure

We train YuE2 by jointly optimizing AR cross‑entropy and flow‑matching losses across a balanced mix of four tasks.

The model must learn to predict symbolic scores, semantic tokens, and high‑fidelity audio simultaneously, which creates a tension: a single loss cannot satisfy all modalities without careful balancing.

Think of a musician practicing piano and drums at the same time – the practice routine interleaves piano exercises with drum beats so the performer becomes fluent in both, rather than mastering each in isolation.

Step 1: Sample T2 → loss consists of $L_{\text{FM}}$ only (AR loss omitted).

Step 2: Sample T4 → both $L_{\text{AR}}$ and $L_{\text{FM}}$ are computed, but $L_{\text{AR}}$ receives the 0.25 scaling.

Step 3: Sample T1 → all three components are present, yielding the full joint loss $L$.

After three steps the model has seen each task once, guaranteeing balanced gradient contributions.

Uniform task sampling prevents any single modality from dominating training, so the same checkpoint can later generate with any combination of inputs.

How does this task‑mixing differ from ordinary multi‑task learning?

Standard multi‑task setups often weight each task’s loss separately and may train on each task for many consecutive steps, which can cause the model to over‑fit to the dominant task. Here the tasks are sampled *per‑step* with a fixed 1 : 1 : 1 : 1 ratio, and the AR loss is deliberately down‑scaled, ensuring that every modality influences every update equally.

The training corpus comprises roughly 346 000 hours of music, packed into 24 576‑position contexts so whole songs are presented without fragmentation. The final model contains about 3.58 B parameters, and conditioning signals (text, lyrics) can be omitted during training to enable classifier‑free guidance at inference.

Target Synthesis

MERT2 learns unified targets by fusing dual encoder views into a shared quantized code.

Relying on a single encoder view leaves important musical relations unmodeled, causing the learner to miss complementary cues such as timbre versus harmony. The authors therefore introduce a dual‑view target that forces the model to satisfy two distinct, high‑level representations of the same audio.

Think of two translators independently rendering the same song into two languages, then forcing a single dictionary to capture both translations simultaneously — the dictionary entries become the shared quantized codes.

How does this differ from simply concatenating the two encoder feature vectors?

Concatenation would keep the two modalities separate, allowing the model to ignore the constraints imposed by a shared discrete code. By quantizing a fused bottleneck, the model must produce a single representation that can be decoded back into both original feature spaces, enforcing richer, cross‑modal consistency.

Fusion network adds the vectors element‑wise, yielding $[3,\,2,\,1,\,5]$.

First quantizer level maps $[3,\,2,\,1,\,5]$ to code $c_1=7$ (nearest centroid).

Residual $[3,\,2,\,1,\,5] - \text{centroid}_7 = [0.2,\,-0.1,\,0.3,\,0.4]$ passes to level 2, producing code $c_2=2$.

Levels 3 and 4 repeat the residual quantization, yielding codes $c_3=5$ and $c_4=1$.

The final four‑code stream for this slice is $(7,\,2,\,5,\,1)$, which both decoders can reconstruct back to their original feature vectors.

The shared code forces the two encoders to agree on a compact representation, so any downstream model trained on these codes automatically learns to respect both timbral and harmonic information.

With the four code streams cached, Stage 1 (Foundation Pretraining) trains a new audio encoder to predict them from masked audio frames. The encoder ingests 24 kHz mono audio, converts it to 128‑bin log‑mel features, downsamples to 25 Hz via a ConvNeXt frontend, and processes the sequence with a 24‑layer, 1,024‑dimensional Conformer.

After the encoder learns to reconstruct the multi‑view codes, two branches diverge. Stage 2‑FS keeps bidirectional attention to handle full‑song recordings (30–360 s) and supplies the SheetSage2 transcriber. Stage 2 switches to causal self‑attention, training the semantic‑tokenizer branch on the same targets but with an autoregressive constraint.

**Figure 4.** MERT2 learns from discrete targets jointly derived from two encoder views of the same audio. (A) Multi-View Target Synthesis. Frozen MuQ [91] and Qwen2-Audio-Instruct [9] encoders map the same recording to time-aligned features. A shared four-level residual vector quantizer produces four cached 25-Hz code streams. Separate decoders reconstruct both feature spaces from the same quantized representation. (B) Stage 1 — Foundation Pretraining. A bidirectional ConvNeXt-Conformer encoder [48, 27] is trained to predict these codes from masked audio. (C) Adaptation. Stage 2-FS — Full-Song Adaptation continues pretraining on complete recordings lasting 30–360 seconds while retaining bidirectional context, producing MERT2-FS for SheetSage2. Stage 2 — Causal Adaptation switches self-attention to a causal mask and continues along the semantic-tokenizer branch.

The MERT2 Tokenizer and SheetSage2

MERT2 Tokenizer and SheetSage2 enable compact symbolic encoding and full‑song transcription.

The core obstacle is that a music generation system needs a tiny, temporally precise symbolic stream that still captures all musical attributes of a full recording.

SheetSage2 turns a complete audio track into an editable lead‑sheet by emitting a single chronological event stream, each event anchored to a $10\,\text{ms}$ beat grid.

Encoder produces a 256‑dimensional vector for each $10\,\text{ms}$ frame (40 frames total).

Decoder emits event 1 at $t=0$: beat = 0, key = C, chord = Cmaj.

Event 2 at $t=100\,\text{ms}$: beat = 1, chord = Fmaj, melody note = E4.

Event 3 at $t=200\,\text{ms}$: beat = 2, chord = G7, melody note = G4.

Event 4 at $t=300\,\text{ms}$: beat = 3, chord = Cmaj, melody note = C5.

The event stream preserves exact rhythmic alignment while allowing the decoder to adjust harmonic and melodic content based on the full‑song context.

How does SheetSage2 differ from typical non‑autoregressive music transcribers?

Typical non‑autoregressive models predict all attributes in parallel, which forces them to ignore long‑range musical dependencies. SheetSage2 keeps the autoregressive decoder, so each event can attend to every previously generated event, capturing phrasing and harmonic progression that parallel models miss.

**Figure 5.** **SheetSage2-AR architecture and output representation.** A full-context MERT2-FS encoder with a frozen backbone and trainable low-rank adapters supplies a learned layer mixture to a six-layer autoregressive decoder. Task prompts select the target attributes, and grammar-constrained decoding produces a chronological sequence of timing, metrical, structural, harmonic, and melodic events. A deterministic builder converts the event sequence to ABC notation, which is rendered as a lead sheet. The right panel uses a reference bar to illustrate this representation. The excerpt is serialized independently, with the active key emitted at its first beat and omitted thereafter unless it changes.

The tokenizer compresses the encoder’s continuous states into a 25 Hz stream of discrete semantic IDs, each carrying 15 bits of musical information.

Compute cosine similarity between $u$ and each of the 32,768 codebook entries $e_i$.

Select the entry with highest similarity; suppose it is entry $c=7{,}842$.

Emit token ID $c$ (15 bits) for the current 40 ms frame.

During training the VQ loss pushes $u$ toward $e_c$, while the usage‑dependent update moves $e_c$ slightly toward $u$.

This quantization preserves the semantic content needed for downstream generation while drastically reducing bandwidth.

Why is causal self‑attention introduced before quantization?

Without causality the encoder could peek at future frames, producing token IDs that depend on information unavailable to the left‑to‑right decoder. The causal mask forces the encoder to generate each token using only past context, making the discrete stream compatible with the autoregressive generator.

**Figure 6.** MERT2 tokenizer training and deployment. (A) Causal Adaptation initializes from Stage 1, masks temporal spans of the input audio, switches the 24-layer Conformer [27] to causal attention, and continues the Stage 1 objective: predicting all four cached target streams at masked positions under $\mathcal{L}_{MLM}$. (B) Supervised Fine-Tuning continues the adapted encoder on full songs and adds lyric connectionist temporal classification (CTC) [25] plus mel and chroma reconstruction. (C) Semantic Quantization inserts a learned 32-dimensional, 32,768-entry clustered vector quantizer (CVQ) between layers 13 and 14. Adapted from CVQ-VAE [90], its online feature-anchor update returns rarely selected entries to the assignment distribution. During Stage 4, layers 14–23 and the CTC, mel, and chroma heads remain above the quantizer, allowing their losses to supervise the quantized states. (D) Deployment retains the log-mel/ConvNeXt frontend [48], Conformer layers 0–13 with causal self-attention, and the quantizer, producing one semantic ID $c$ every 40 ms (25 Hz; 375 bit/s) for autoregressive modeling and acoustic conditioning in Yue2.

Together, SheetSage2 and the MERT2 Tokenizer provide a compact, temporally aligned symbolic backbone that powers YuE2’s high‑fidelity audio generation.

Frontier Song Quality

YuE2 attains frontier full-song quality, surpassing all baselines on the WildSongBench evaluation.

YuE2 (best‑of‑8) achieves the highest SongBench Musicality and Global Avg among all evaluated systems.

SongBench Musicality = 6.2666, Global Avg = 6.9632

WildSongBench is a 192‑prompt benchmark that measures full‑song generation quality across diverse user requests, covering both Chinese and English prompts and code‑switched inputs.

We also probe two design choices: whether generating an explicit symbolic score before audio synthesis improves the final song, and whether jointly learning semantic‑token prediction and acoustic generation yields higher quality than training them separately.

Impact of Symbolic Planning

Removing Symbolic Planning hurts expert‑rated quality across all dimensions.

We ablate the Symbolic Planning layer from the full YuE2 pipeline and measure the resulting change in expert‑rated quality.

Overall expert preference falls from 49 % to 39 % when Symbolic Planning is removed.

Figure 8 shows a 10 % absolute drop in the “overall” bar for the no‑planning condition.

Melody preference declines by 12 % without Symbolic Planning.

In Figure 8 the “melody” segment drops from a planning‑favored majority to a lower majority for the no‑planning side.

Chord‑progression preference shows the smallest effect, with planning preferred by 39 % versus 31 % without.

Figure 8 reports the “chord progression” bar: planning 39 % vs. no‑planning 31 %.

Suno v5 MOS drops from 5.99 to 5.81 when Symbolic Planning is omitted.

Table (derived from raw scores) lists 5.9918 → 5.8317.

Suno v6 MOS falls from 5.66 to 5.56 without Symbolic Planning.

Raw scores show 5.6558 → 5.5644.

**Figure 8.** Symbolic planning improves perceived song quality, melody, and chord progression. Expert preferences for generation with versus without symbolic planning. Dark blue favors planning, medium blue denotes ties, and light blue favors generation without planning. Appendix D.4 details the listening protocol.

Unified MoT Performance

Unified MoT yields top scores across multiple music generation benchmarks.

Unified MoT delivers the highest scores on SongBench, AudioBox, and Q3O metrics.

Yue2 (Bo8) achieves the best values in these categories, surpassing the next‑best model by measurable margins.

Plan Realization

YuE2 (Bo8) sets new highs across all evaluation metrics.

We evaluate the unified system on five established music‑generation metrics and compare it against eight prior baselines.

YuE2 (Bo8) achieves the highest MuLan score among all systems.

MuLan = 6.96, beating the next best model (6.26) by 0.70 points.

On the remaining metrics, YuE2 (Bo8) matches or exceeds all baselines: Q3O scores are comparable to the best prior systems, and PER (lower is better) remains competitive.

Zero-Shot Cover Generation

Zero-shot cover generation shows YuE2 beating public baselines on text alignment and lyric accuracy.

YuE2 achieves higher text‑alignment scores than any public baseline on both MuLan and AllMusicCaps.

YuE2 scores 0.5068 on MuLan and 0.4054 on AllMusicCaps, while the best public baseline (Suno v5) scores lower on both metrics.

YuE2 attains competitive lyric accuracy with a phoneme error rate (PER) of 0.0844.

The lowest public‑system PER is 0.0627 (MiniMax Music 3), but YuE2 remains close while offering full symbolic planning.

The listening evaluation recorded 351 participations, yielding 4 439 retained pairwise judgments across three experiments on 192 prompts, covering overall quality, musicality, text alignment, audio quality, vocals, and accompaniment.

**Figure 7.** Expert preferences against proprietary song generators. Columns show YuE2 and YuE2 (best-of-8), both with symbolic planning. Dark blue favors the YuE2 setting named above the column; light blue favors the baseline named in the row. Baselines are Suno v4.5 [68], v5 [65], v5.5 [62], v6 and v6 Wild [66], and Mureka 9 [51].

Expert Listening Evaluation

YuE2 outperforms six proprietary baselines in expert listening and benefits from symbolic planning.

YuE2 unifies symbolic planning with high‑fidelity audio generation, letting a score steer composition before rendering the final waveform.

YuE2 is preferred over all six proprietary baselines for overall quality in expert listening.

57.3% of judged responses favor YuE2, while the strongest baseline receives at most 30.5%.

Against Suno v4.5 the baseline receives 30.5%, Suno v5 is nearly balanced at 40.4% vs 39.9%, and Suno v6 actually leads with 59.3% versus YuE2’s 31.6% on that baseline. Audio‑quality preference is a particular strength: YuE2 attains a tie‑adjusted 58.9% share (95 % CI 55.6–62.2%, p = 1.70 × 10⁻⁶) across the six proprietary systems.

Symbolic planning improves overall perceived quality compared with generation without planning.

49.3% of expert judgments favor the planning condition versus 34.6% for the no‑planning condition.

Musicality also benefits from planning (45.0% vs 29.4%). For melody the planning condition wins 44.0% to 29.5%, and for chord progression it wins 39.0% to 21.0%.

**Figure 12.** Expert preferences for YuE2 versus six proprietary song generators across all six criteria. Audio-quality preferences favor YuE2 against all six baselines; overall preferences favor it in two comparisons. YuE2 uses melody-and-chord planning and selects the candidate with fewer lyric errors from two generated songs. Baselines are Suno v4.5 [68], v5 [65], v5.5 [62], v6 and v6 Wild [66], and Mureka 9 [51]. Dark blue favors YuE2, light blue favors the named baseline, and medium blue denotes ties. Baseline order is shared by the left and right panels. Each bar sums to 100% of judged responses, excluding unable-to-judge answers. Counts and CR1 intervals are in Appendix D.4.4.

Representation Retention

Unified planning boosts expert preference and melodic coherence, as shown by quantitative and score analyses.

YuE2’s unified model is preferred overall by experts, achieving a 53% share of judgments.

Figure 10 displays expert preference percentages across six criteria, with YuE2 leading in five.

Figure 9 illustrates how symbolic planning yields a varied melodic motif that recurs an octave higher in the chorus, strengthening thematic continuity. In contrast, the unplanned version repeats the same note pattern, reducing melodic recall. Together with the expert preference data, these results confirm that both planning and unified generation enhance perceived song quality.

**Figure 9.** Planning develops a recurring melodic idea in this paired example. Both songs use the same prompt and lyrics. Read each staff from left to right; higher notes indicate higher pitch, and letters above the staff name the accompanying chords. Purple marks phrases with repeated lyric openings. With planning (left), the verse varies a low–low–high–high note pattern; the chorus recalls it an octave higher with longer phrase endings, linking the sections through recognizable variation. Without planning (right), the verse repeats the same B–D–E–E pattern, with less variation and a less direct melodic connection to the chorus. Gray marks a chord transition where experts hear the bass descent break, weakening harmonic continuity. The left score starts from the generated plan and the right from a SheetSage2 transcription; experts corrected both against the completed recordings for display, without regenerating audio. Appendix E gives the full scores and further analysis.

**Figure 10.** **Experts prefer unified generation on five of six criteria; text alignment is nearly balanced.** YuE2’s Mixture-of-Transformers (MoT) [14] learns semantic-token prediction and acoustic generation in one backbone; the baseline uses a separate language model and diffusion Transformer (LM+DiT) [81]. Both use melody-and-chord planning, the same training-data volume and representations, and the same prompts and two-candidate selection protocol. Dark blue favors YuE2, medium blue denotes ties, and light blue favors LM+DiT. Bars show percentages of judged expert responses (206–208 per criterion), excluding unable-to-judge answers.

SheetSage2 Evaluation

Expert listeners prefer YuE2’s unified MoT over the separate LM+DiT baseline.

YuE2’s unified MoT is preferred over the separate LM+DiT baseline for overall quality.

209 retained expert evaluations yielded 206–208 judged responses per criterion; 53.4% favored YuE2, 35.6% favored LM+DiT, the remainder were ties.

The expert panel also rated musicality, vocals, and accompaniment, consistently favoring YuE2, while text‑alignment scores were nearly balanced (34.0% vs 33.5%).

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers