Qwen-Music Technical Report

Jin Xu, Kangdi Wang, Ruibin Yuan, Shun Lei, Xiong Wang, Xize Cheng, Xueyao Zhang, Yang Zhang, Yiheng Chen, Yongqi Wang, Yue Wang, Zhifang Guo, Zihan Liu, Zijian Lin, Dake Guo, Hangrui Hu, Lei Xie, Linhan Ma, Wei Xue, Wenxiang Guo, Xinfa Zhu, Xipin Wei, Yangze Li, Yuanjun Lv, Yuxuan Wang, Yunfei Chu, Zhiyong Wu

Qwen-Music uses explicit melody planning and a three-stage neural renderer to generate high-fidelity, controllable songs.

How does Qwen-Music achieve high-fidelity, controllable music generation with vocals by decoupling semantic tokenization from neural rendering?

Music generation models often struggle to maintain structural coherence and melodic stability over long durations because they conflate high-level semantic planning with low-level acoustic rendering. Qwen-Music separates these tasks: a language model plans the song structure and vocal melody using an intermediate "Melody-CoT" representation, while a separate diffusion-based renderer synthesizes the final 48kHz stereo waveform. This architecture achieves state-of-the-art performance on 13 of 16 objective musicality metrics and is preferred by professional human raters over leading proprietary systems like Suno V5 and MiniMax Music 2.6.

Paper Primer

The system hinges on "Melody-CoT," a chain-of-thought mechanism that forces the language model to generate a coarse vocal melody contour before predicting the full sequence of semantic tokens. This acts like a musical blueprint: the model sketches the melody first to ensure structural alignment, then fills in the acoustic details, preventing the "drift" common in models that attempt to predict raw audio or dense tokens directly.

Qwen-Music achieves superior subjective preference against major commercial baselines.

Blind A/B testing by 50 professional music producers. 59.1% win rate vs. MiniMax Music 2.5+ and 55.4% vs. Suno V5.

The system demonstrates robust reference-melody preservation for cover song generation.

Mean Absolute Error (MAE) in semitones compared to reference audio. Outperforms MiniMax Cover on most metrics in real-world popular song sets.

Why use an explicit melody planning step instead of letting the model learn melody implicitly?

Implicit modeling often fails to resolve compositional structure and arrangement simultaneously. By planning the melody first, the model gains a stable compositional guide that improves structural coherence and allows for better control during cover song generation.

What is the role of the "Band-Mode Refiner" in the rendering stage?

It corrects frequency-dependent artifacts by applying band-specific corrections—phase-only for low frequencies, joint magnitude-phase for mid-range, and magnitude-only for high frequencies—to improve spectral fidelity after the initial spectrogram reconstruction.

Introduction to Qwen-Music

Qwen-Music splits music generation into semantic tokenization and neural rendering for controllable, high‑fidelity songs.

Existing music generators try to model waveforms end‑to‑end, which forces a single model to learn both long‑range musical planning and fine‑grained acoustic detail. This coupling creates brittle, low‑fidelity results, especially when users demand precise control over melody, style, or vocal timbre.

Music Semantic Tokens are a compact, 25 Hz stream of discrete codes that capture the high‑level musical meaning—melody, harmony, rhythm—while discarding raw waveform detail.

By decoupling semantic composition from acoustic rendering, Qwen‑Music can plan long‑range structure with the LLM and then render high‑resolution waveforms with a dedicated neural renderer, solving the core fidelity‑control trade‑off.

**Figure 1.** Qwen-Music is a powerful and controllable music generation model capable of producing songs with complete vocal singing. It generates complete songs from text descriptions, lyrics, and musical attributes, and uses reference audio for cover song generation with different styles and vocal characteristics.

**Figure 4.** Overview of the Qwen-Music inference pipeline. Given a user request, Qwen-Music first rewrites the natural-language description into a structured textual condition, including musical tags (such as genre, singer characteristics, instrumental arrangement, etc.) and generated lyrics. Qwen-Music-LLM then generates Music Semantic Tokens, optionally using melody tokens extracted from a reference song for melody cloning. Finally, Qwen-Music-Render takes both the rewritten textual condition and the generated Music Semantic Tokens as input and performs generative rendering to produce high-fidelity 48 kHz stereo waveforms.

The key shift is moving from monolithic end‑to‑end generation to a decoupled semantic‑acoustic modeling pipeline.

Semantic Tokenization

The tokenizer compresses raw audio into a 25 Hz token stream via staged training and a VQ bottleneck.

The tokenizer must compress a raw waveform into a low‑bitrate discrete stream while retaining the musical structure needed for downstream generation.

First a continuous music representation is learned, then causality and discretization are introduced step‑by‑step, ending with a vector‑quantized bottleneck that yields a $25\,\text{Hz}$ token stream.

How does this staged VQ tokenizer differ from a naïve end‑to‑end quantization of audio?

In a naïve pipeline the encoder would be forced to produce discrete codes from the start, which destabilizes training and discards useful continuous features. By first learning a high‑capacity continuous representation and only later inserting the VQ bottleneck, the model preserves lyrical and spectral information and achieves near‑full codebook utilization.

Stage 1: Bidirectional BestRQ pretraining on $30\,\text{s}$ crops with masked span prediction against a frozen random‑projection target.

Stage 2: Causal adaptation – resume from the Stage 1 checkpoint, switch self‑attention to left‑only masking, keep the same objective.

Stage 3: Multi‑task SFT – add CTC, Mel, and chroma heads; train on full songs (≤ $300\,\text{s}$) with a combined loss $L_{\text{SFT}}$.

Stage 4: Vector‑Quantized tokenizer – insert a single VQ codebook at an intermediate Conformer layer, blend quantized and continuous paths with a linear gate $\alpha$, and fine‑tune the whole model.

Encoder produces 50 continuous latent vectors $h_1,\dots,h_{50}$.

Each $h_t$ is matched to the nearest codebook entry, yielding discrete indices $c_t\in\{0,\dots,7\}$.

The gating scalar $\alpha$ ramps from $0$ to $1$ over the first 10 frames; for $t\le10$, the output is $h_t+\alpha\,(e_{c_t}-h_t)$, gradually replacing the continuous vector with its quantized counterpart $e_{c_t}$.

After frame 10, $\alpha=1$, so the output stream consists entirely of the selected codebook entries.

The resulting token sequence is $[c_1,\dots,c_{50}]$, a $25\,\text{Hz}$ representation that can be fed to an autoregressive model.

The gradual gate prevents a sudden shock to the network, ensuring the encoder adapts to the discrete space without collapsing performance.

**Figure 5.** Overview of Qwen-Music-Tokenizer. The tokenizer maps music waveforms to 25 Hz Music Semantic Tokens through BestRQ pretraining, causal adaptation, multi-task SFT, and VQ tokenizer training.

LLM and Melody Planning

We introduce Melody‑CoT, a planning step that guides LLM music generation via intermediate melody tokens.

Text prompts specify style and lyrics but omit the melodic contour, forcing a model to learn composition and arrangement simultaneously. This gap makes direct prediction of full‑mixture Music Semantic Tokens brittle. To resolve it, we insert an explicit melody‑planning stage before token generation.

Consume global music tags and structured lyrics as textual conditioning.

Optionally generate a coarse melody plan (Melody‑CoT) as a sequence of melody tokens.

Condition on the melody plan (if present) and generate full‑mix Music Semantic Tokens.

Pass the generated Music Semantic Tokens to Qwen‑Music‑Render for high‑fidelity waveform synthesis.

An autoregressive language model that produces discrete music tokens, optionally guided by an intermediate melody outline.

How does Qwen‑Music‑LLM differ from a standard text‑only language model?

It predicts discrete music tokens instead of words, and it can ingest an explicit melody plan that shapes the musical contour before generating the full token sequence.

First sketch a coarse melody outline, then let the LLM fill in the full music token sequence—like drafting a melody before arranging the full orchestration.

Why not generate the full Music Semantic Tokens directly without an intermediate melody plan?

Without an explicit melody scaffold the model must infer both high‑level structure and fine‑grained details simultaneously, which makes learning unstable and often yields incoherent melodic lines.

Transforms a vocal pitch contour into a compact token sequence that encodes only the shape of the melody, discarding absolute pitch and timbre.

How does using relative MIDI offsets avoid leaking key or singer information?

Subtracting the median MIDI value removes the absolute pitch level, so only the contour (intervals) remains; this discards key, vocal range, and timbral cues while preserving the melodic shape needed for conditioning.

Convert to MIDI: $[69, -, 70, -]$ (Hz→MIDI rounding).

Median of voiced MIDI values is $\bar{m}=69$.

Compute relative offsets: $[0, -, 1, -]$.

Clip to $[-127,127]$ (no change) and shift by 127: tokens become $[127, 255, 128, 255]$.

This example shows how unvoiced frames become token 255 while the melodic contour is captured by small offsets around the median.

Neural Rendering Architecture

Qwen‑Music‑Render turns discrete music tokens into high‑fidelity audio via a three‑stage neural pipeline.

Turning the compact discrete tokens produced by the language model into a realistic waveform is non‑trivial: the tokens capture long‑range structure but lack the fine‑grained spectral detail needed for high‑fidelity audio.

The renderer is a three‑stage pipeline that first diffuses a latent representation, then decodes a coarse spectrogram, and finally refines band‑specific magnitude and phase residuals before inverse STFT.

Why does the renderer use three distinct stages instead of a single end‑to‑end decoder?

Stage 1 provides a smooth, high‑dimensional latent that is easy to model with diffusion; Stage 2 converts that latent into a spectrogram that respects the physics of audio; Stage 3 fixes systematic magnitude/phase errors that are band‑dependent, which would be hard to learn jointly in a monolithic network.

DiT treats the latent sequence like a time‑series and uses transformer blocks, cross‑attention, and timestep‑dependent AdaLN to denoise it step by step.

How does this DiT differ from a standard diffusion model that operates on image pixels?

Instead of 2‑D convolutions on pixel grids, DiT processes a 1‑D latent sequence with self‑ and cross‑attention, and its conditioning includes both Music Semantic Tokens and free‑text embeddings, which are absent in typical image diffusion pipelines.

**Figure 6.** Overview of Qwen-Music-Render. Music Semantic Tokens and text conditions guide a semantic-conditioned DiT that predicts acoustic latents. Spec-VAE decodes the latents into complex spectrograms, and the Band-Mode Refiner corrects band-dependent magnitude and phase details before inverse STFT synthesis to 48 kHz stereo waveforms.

Spec‑VAE compresses a complex STFT spectrogram into a low‑dimensional latent and then reconstructs a coarse spectrogram, providing the bridge between diffusion latents and waveform synthesis.

Why replace the vector‑quantized tokens with a continuous latent in Spec‑VAE?

Continuous latents avoid the hard discretisation bottleneck of VQ, making them compatible with the flow‑matching objective of the diffusion stage and allowing the model to learn smooth transformations.

Spec‑SnakeBeta endows each STFT frequency bin with its own periodic modulation, letting the activation adapt to the spectral characteristics of that band.

Compute the sinusoidal term: $\sin^{2}(0.8\cdot\exp(\alpha_f))$ yields approximately $[0.12, 0.34, 0.71, 0.95]$ for the four bins.

Since $\beta_f=0$, the denominator $1/\big(\exp(\beta_f)+\varepsilon\big)\approx1$, so the added term equals the sinusoid values.

Final activation $= x +$ sinusoid $= [0.92, 1.14, 1.51, 1.75]$.

Because $\alpha_f$ grows with frequency, higher‑frequency bins receive a larger sinusoidal boost, illustrating how Spec‑SnakeBeta tailors non‑linearity to spectral location.

In what way does Spec‑SnakeBeta differ from the original SnakeBeta used in BigVGAN?

The original SnakeBeta shares a single $\alpha$ across all channels, producing the same periodic modulation regardless of frequency. Spec‑SnakeBeta assigns a distinct $\alpha_f$ to each STFT frequency bin, allowing the activation shape to vary across the spectrum.

The refiner is a ConvNeXt‑1D module that applies separate correction strategies to low, mid, and high frequency bands, fixing residual magnitude and phase errors left by the decoder.

Why does the refiner treat low, mid, and high bands differently?

Empirical analysis shows that phase errors dominate in the low band, while magnitude errors are more pronounced at high frequencies; the mid band exhibits both. Tailoring the correction per band lets each sub‑module focus on the error type that matters most for that frequency range.

**Figure 7.** Spec-SnakeBeta analysis. (a) Activation curves: the standard SnakeBeta (black dashed) applies a single shared $\alpha$ to all frequency bins; Spec-SnakeBeta adapts the periodic modulation per frequency, producing distinct nonlinear shapes at 0.5, 3, 10, and 20 kHz. (b) Deviation of learned $\alpha_f$ from log-frequency initialization ($\Delta\alpha_f = \alpha_{learned} - \alpha_{init}$) across encoder layers—positive values indicate the network increases periodic modulation beyond the physical prior, with the strongest deviations concentrated in 0–5 kHz.

Stage 1: pre‑train Spec‑VAE alone with reconstruction loss only.

Stage 2: add waveform‑domain adversarial training (LSGAN with feature matching) while keeping the Spec‑VAE encoder‑decoder trainable.

Stage 3: freeze the Spec‑VAE encoder‑decoder and train the Band‑Mode Refiner using both waveform and spectral discriminators.

What would happen if we trained all three components jointly from scratch?

Joint training would force the diffusion transformer, Spec‑VAE, and refiner to compete for gradient signal, often leading to divergence or poor spectral fidelity because the refiner would receive noisy decoder outputs before the decoder itself had learned a reasonable reconstruction.

The acoustic data quality pipeline filters raw audio collections to retain only genuinely high‑fidelity recordings, using a noise‑floor‑aware cutoff detector and a suite of metric‑level checks.

Training Pipeline

Training the LLM with a quality‑graded curriculum and staged alignment for high‑fidelity music.

Training on raw music data is hampered by uneven quality, which can corrupt the LLM’s learned representations. To prevent low‑quality samples from dominating, the authors devise a curriculum that gradually raises the data quality.

The model first sees a broad, mixed‑quality corpus and then progressively focuses on higher‑quality buckets, letting it learn general patterns before refining on pristine examples.

Percentile thresholds assign $Q1$ to 9.2, $Q2$ to 8.5, $Q3$ to 7.8, $Q4$ to 6.4, $Q5$ to 5.9, $Q6$ to 4.3, and $Q7$ to 3.1 and 2.0.

Stage 1 uses $Q3$–$Q6$ samples (7.8, 6.4, 5.9, 4.3) with dynamic sampling that starts with the lower‑quality $Q6$ and gradually shifts to $Q3$.

Stage 2 switches to $Q2$ (8.5) and applies a decaying learning rate, consolidating the learned structure.

Stage 3 focuses on $Q1$ (9.2) and adds any extra high‑quality samples, yielding the final refined model.

This toy flow shows how the curriculum discards the worst samples and progressively concentrates training on higher‑quality data, preserving diversity early while sharpening quality later.

How does this curriculum differ from simply discarding low‑quality data at the start?

Instead of a hard cutoff, it uses a progressive schedule that keeps lower‑quality examples early to teach broad coverage, then gradually replaces them with higher‑quality data, preserving diversity while improving final quality.

After the curriculum, the model may still misalign with human musical preferences, so a staged post‑training aligns it further.

The model first fine‑tunes on clean, high‑quality examples, then iteratively learns from preference feedback, and finally refines via on‑policy optimization, each step building on the stability of the previous one.

Phase I selects the higher‑scoring candidates (7.0, 8.2, 5.8) for supervised fine‑tuning, establishing a clean generation prior.

Phase II runs DPO: rollouts are generated, scores are paired (e.g., 7.0 vs 6.5), and the policy is updated to prefer the higher‑scoring output.

Phase III invokes GSPO: the updated policy samples new outputs, receives sequence‑level rewards, and the gradients directly adjust the policy toward higher musicality.

This example shows how each phase adds a layer of alignment—static supervision, offline preference learning, then online reward‑driven refinement—culminating in a policy that consistently produces higher‑quality music.

Why not apply the online GSPO directly after pre‑training instead of using the earlier phases?

GSPO relies on a stable policy; without the supervised and offline alignment phases the model would still produce noisy outputs, making the on‑policy gradients noisy and potentially destabilizing training.

Generation Performance

Qwen-Music outperforms competitors in blind preference tests and objective metrics across text‑to‑music and cover‑song tasks.

Qwen-Music wins 66.7 % of blind A/B preference tests against MiniMax Music 2.6.

Figure 2 (blind A/B preference results)

**Figure 2.** Blind A/B preference results between Qwen-Music and leading proprietary systems, including MiniMax Music 2.5+, MiniMax Music 2.6, Mureka V8, Suno V5, and Suno V5.5, judged by professional human raters. Each pair is evaluated under anonymized and randomized presentation, and values indicate the percentage of expert votes preferring each system.

MiniMax Music is a proprietary music‑generation system that focuses on maximizing melodic diversity while keeping computational cost low.

Suno is a commercial music‑generation service that prioritizes stable, high‑quality vocal synthesis using a large‑scale diffusion backbone.

Qwen-Music attains the highest SongBench musicality score of 7.47.

Table 5 (objective text‑to‑music results)

Qwen-Music achieves the lowest Melody MAE of 1.48 semitones on AI‑generated reference melodies.

Table 6 (cover‑song generation results)

Rendering Fidelity Analysis

Assessing how each render component impacts high‑fidelity audio quality.

We isolate three design axes of the DiT rendering pipeline—latent backbone, rewritten textual conditioning, and classifier‑free guidance (CFG)—and then test whether the Band‑Mode Refiner adds measurable acoustic benefit.

Across all three evaluation axes—spectral fidelity, mel‑scale response, and stereo image preservation—the full decoder (Spec‑VAE + Refiner) consistently outperforms the baseline Spec‑VAE.

Band‑Mode Refiner improves acoustic reconstruction, lowering Mel Distance by 0.14 and achieving the best scores on STFT Distance, Mel Distance, and Spectral Pan Error.

Table 9 shows Spec‑VAE + Refiner reaches Mel Distance 0.461 versus 0.599 for Spec‑VAE, and the lowest STFT Distance (0.870) and Spectral Pan Error (0.264) among all models.

Authors and Contributions

Lists the core and additional contributors, explains ordering, and notes the corresponding author.

Core Contributors are Shun Lei, Xiong Wang, Xize Cheng, Xueyao Zhang, Yang Zhang, Yiheng Chen, Yongqi Wang, Yue Wang, Zhifang Guo, Zihan Liu, Zijian Lin, Jin Xu*, Kangdi Wang, and Ruibin Yuan.

Additional Contributors include Dake Guo, Hangrui Hu, Lei Xie, Linhan Ma, Wei Xue, Wenxiang Guo, Xinfa Zhu, Xipin Wei, Yangze Li, Yuanjun Lv, Yuxuan Wang, Yunfei Chu, and Zhiyong Wu; names are ordered alphabetically by given name, read top‑to‑bottom then left‑to‑right in the multi‑column layout, and the asterisk denotes the corresponding author, for a total of 23 authors.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers