Unified Audio Intelligence Without Regressing on Text Intelligence

Zhifeng Kong, Sang-gil Lee, Jaehyeon Kim, Boxin Wang, Zihan Liu, Sungwon Kim, Yang Chen, Arushi Goel, Rajarshi Roy, Wenliang Dai, Zhuolin Yang, Yangyi Chen, Dongfu Jiang, Sreyan Ghosh, Tuomas Rintamaki, Andrew Tao, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping

Audex is a unified audio-text LLM that achieves state-of-the-art audio generation and understanding without degrading text reasoning capabilities.

Multimodal models often suffer from "text regression," where adding audio or vision capabilities degrades the reasoning and agentic performance of the underlying text-only LLM. Audex solves this by using a single, unified Transformer decoder that treats audio and text tokens identically, trained via a multi-stage supervised fine-tuning recipe that preserves the backbone's long-context and reasoning abilities. The model achieves state-of-the-art results in audio understanding and generation while maintaining or exceeding the performance of its text-only backbone on core reasoning benchmarks.

Paper Primer

Audex is a single Mixture-of-Experts (MoE) Transformer decoder that ingests audio via a frozen encoder and MLP adapters, projecting features into the text embedding space. It treats text and quantized audio tokens uniformly during autoregressive generation, allowing the model to handle audio-to-text, text-to-audio, and speech-to-speech tasks within a single architecture.

The core mechanism is a multi-stage training recipe: after an initial audio-adapter warmup, the model undergoes a multi-stage supervised fine-tuning (SFT) process that gradually introduces audio tasks while keeping text-only data in the mix to prevent catastrophic forgetting. This is followed by text-only reinforcement learning (RL) to further align the model without regressing on its audio-processing capabilities.

Audex preserves and occasionally improves upon the reasoning capabilities of its text-only backbone.

On benchmarks like IMO AnswerBench and AIME 2026, Audex-30B-A3B achieves higher accuracy than its text-only backbone, Nemotron-Cascade-2-30B-A3B. Outperforms proprietary and open-source multimodal baselines on reasoning and alignment tasks while matching state-of-the-art audio understanding performance.

Why is a multi-stage training strategy preferred over a single-stage consolidated approach?

While single-stage training is more efficient, it causes the model to fail on long-context tasks (e.g., NIAH) because the attention mechanism undergoes drastic, unstable changes when forced to learn long text and long audio simultaneously.

How does the model handle the different structural requirements of speech versus non-speech audio?

Audex uses a separate codec dispatch design: speech is tokenized at 50 tokens/s using a single-layer codec, while non-speech audio uses a multi-layer residual vector quantization (RVQ) codec at 200 tokens/s to capture more complex auditory structures.

Audex demonstrates that high-performance multimodal intelligence can be achieved by treating audio as a first-class token type within a standard LLM architecture, provided the training pipeline is carefully staged to protect the backbone's core reasoning logic.

Abstract and Introduction

Audex unifies audio and text processing in a single decoder, tackling multimodal regression.

Current multimodal LLMs that add audio or vision outputs often lose performance on core text benchmarks, a regression that threatens the reasoning and knowledge capabilities essential for AGI. Audex addresses this gap by embedding audio inputs and quantized audio tokens into the same space as text and using a single Transformer decoder, preserving text intelligence while delivering state‑of‑the‑art audio understanding and generation. Adding audio output is like inserting a new instrument into an orchestra; without careful mixing the original melody can be drowned out.

Speech and Audio Generation

Existing speech and audio generation pipelines are fragmented, unstable, and lack unified control.

Current speech and audio generation systems treat text‑to‑speech (TTS) and text‑to‑audio (TTA) as separate tasks, often using diffusion or autoregressive models that rely on cross‑attention to semantic text embeddings. This coupling entangles language understanding with acoustic generation, making it hard to control generation, to share parameters across tasks, and to maintain stability when extending to variable‑length outputs.

TTS converts a written sentence into a spoken waveform, enabling voice‑driven interfaces and accessibility applications.

TTA expands generation beyond speech to arbitrary sounds—music, effects, or environmental audio—directly from a textual description.

Both TTS and TTA are evaluated with task‑specific metrics: prompting word error rate (WER) and speaker similarity (SIM) for TTS, and Fréchet Distance in the OpenL3 embedding space (FDopenl3) for TTA. Lower values indicate better fidelity.

Table 6 shows that Audex attains a very low prompting WER, yet its fixed‑voice speaker similarity lags behind other systems—an artifact of the chosen codec and training recipe. Table 7 reports competitive FDopenl3 scores on AudioCaps and SongDescriber, but the authors note a marginal drop after text‑RL fine‑tuning, underscoring the sensitivity of audio quality to training objectives.

CFG scales the model’s conditional logits by a factor $λ$, trading off fidelity against diversity; higher $λ$ pushes the output toward the conditioning signal.

Top‑k sampling truncates the probability distribution to the $k$ most likely tokens, while temperature rescales probabilities to control randomness.

Figure 4 (CFG study) reveals that TTA quality peaks when $λ\in[3,5]$, whereas TTS prefers $λ\approx1.5$. Large $λ$ values degrade quality for both tasks, but the degradation is steeper for TTA. The hyper‑parameter sweep also shows that TTA benefits from higher randomness (top‑k = 80, temperature = 1.0) while TTS works best with low randomness (temperature = 0.1).

The figure displays three line charts showing the impact of "Audio Gen. + Audio Und. SFT Progress (%)" on performance metrics across three benchmarks: MMLU-Pro, GPQA-Diamond, and AIME 2025. Each chart compares two conditions represented by different colors: olive green (0.69) and dark red (0.56).

Audio Understanding and ASR

Audex is evaluated on ASR, AST, and audio understanding, revealing strengths and gaps.

Current audio‑language models struggle to handle the full spectrum of speech‑to‑text, speech‑to‑text translation, and general audio reasoning in a unified way, especially under noise and across many languages.

We evaluate Audex using a standardized audio question‑answering template: fixed prompts and greedy sampling for ASR/AST, and top‑p = 0.9 with temperature = 0.7 for open‑ended audio understanding benchmarks.

The table lists various audio-based AI models and their corresponding accuracy scores. The models are ranked in descending order of accuracy, with "Fun-Realtime-Audiochat" and "Step-Audio R1.1 Realtime" tied for the highest score of 97.6, and "Moshi" having the lowest score of 4.4. Two additional models, "Audex 30B-A3B" and "Audex 2B," are highlighted in a separate green-shaded row at the bottom with scores of 90.0 and 64.3, respectively.

**Table 11.** Evaluation Results on Fleurs AST (xx→en) Using BLEU|COMET (↑).

**Table 12.** Audio Understanding Evaluation Results.

ASR/AST Details

We compile a comprehensive benchmark suite to evaluate unified audio‑text models across diverse tasks.

Prior work evaluates audio and text capabilities in isolation, using disparate datasets and inconsistent prompting, which makes it impossible to gauge whether a single model truly unifies both modalities without sacrificing performance on either side. This fragmented evaluation landscape leaves a blind spot: researchers cannot reliably compare unified audio‑text systems because there is no single, self‑contained benchmark suite that spans reasoning, knowledge, alignment, long‑context, agentic, TTS, ASR/AST, speech interaction, audio understanding, and text‑to‑audio tasks. The appendix therefore assembles a unified set of benchmarks and standardized evaluation protocols so that any model can be assessed holistically across the full spectrum of audio‑text intelligence.

Model Architecture Details

Audex integrates audio encoding, dual codecs, and a MoE LLM backbone to handle speech and general audio.

Audio modeling demands handling both speech, which is relatively low‑entropy, and general sounds, which are richer and require finer granularity. A naïve single‑codec design either wastes capacity on speech or under‑represents complex audio. Audex resolves this tension by pairing two specialized codecs with a shared LLM backbone.

**Figure 1:** Audex-30B-A3B architecture. The LLM backbone is Nemotron-Cascade-2-30B-A3B. Audio inputs are encoded into continuous embeddings using an audio encoder and MLP adapters. Audex uses an extended vocabulary and directly predicts discrete speech and audio tokens in addition to text tokens.

The backbone is a massive mixture‑of‑experts transformer that routes each token to a small subset of experts, keeping compute scalable while preserving a huge parameter count.

How does this MoE backbone differ from a standard dense transformer of similar size?

In a dense transformer every token passes through all feed‑forward layers, incurring $O(d_{model}^2)$ work per layer. The MoE backbone activates only 6 out of 128 experts, reducing the per‑token cost to roughly $6/128$ of a dense layer while still exposing the full parameter pool for specialization.

AF‑Whisper converts raw 16 kHz audio into a short sequence of high‑dimensional features, then a tiny MLP reshapes them to the LLM’s embedding size.

Why not feed the 1280‑dim encoder output directly into the LLM without an adapter?

Directly feeding mismatched dimensions would break the LLM’s learned embedding matrix and positional encodings. The adapter learns a compatible projection, ensuring that audio features occupy the same vector space as text tokens.

Audex treats speech and general audio with separate codecs, each tuned to the signal’s complexity, then merges their token streams into a single vocabulary.

Speech token count: $10$ tokens, each drawn from a 65 536‑size codebook.

Non‑speech token count: $40$ tokens, each drawn from a 4 096‑size sub‑vocab (first 4 RVQ layers).

Both token sets are concatenated to the shared embedding table, which now holds \$131\,072 + 65\,536 + 8\,192 + 5 = 204\,805$ entries.

The table is padded to $205\,312$ entries to satisfy the $512$‑multiple requirement of Megatron‑LM.

During generation, the model predicts a sequence of mixed tokens; a downstream decoder interprets each token according to its prefix (<speechcodec_…> vs <audiocodec_…>).

This example shows why speech can be modeled with far fewer tokens per second without sacrificing quality, while non‑speech benefits from a higher token rate to capture richer acoustic detail.

How does this dual‑codec design avoid the “one‑size‑fits‑all” token budget problem?

By assigning a low‑rate, large‑vocab codec to speech (where redundancy is high) and a higher‑rate, smaller‑vocab codec to non‑speech, Audex allocates bits where they matter most, preventing wasteful token inflation for speech and under‑representation for complex sounds.

Raw 16 kHz audio is chunked into 30‑second windows and padded.

AF‑Whisper encodes each window into 750 frames of 1280‑dim features (25 Hz).

Two‑layer MLP adapters project features to $d_{model}$ and prepend them to the token stream.

The LLM backbone (MoE transformer) consumes the combined text‑and‑audio token sequence.

During generation, the model emits tokens from the unified vocabulary, including <speechcodec_…> and <audiocodec_…> markers.

Speech tokens are decoded by a causal ConvNeXt streaming decoder; non‑speech tokens are passed through an enhancement VAE and upsampled to 48 kHz.

Streaming ConvNeXt decoder for speech tokens (causal, small look‑ahead).

Training Data and Templates

Unified chat templates let a single model ingest and emit text, speech, and audio.

Training a single model on heterogeneous modalities creates a coordination problem: each modality expects a different input‑output convention, and naïve concatenation leads to ambiguous token streams.

Think of the template as a mail envelope: a special token ( or <|text to speech|>) stamps the envelope with the destination modality, and the model reads the address before opening the content.

The model reads the <|text to speech|> token and switches its decoder to the speech‑codec vocabulary.

It then attends to the literal word “hello” and produces a sequence of codec tokens representing a 0.5‑second waveform.

The assistant turn ends with

The special token not only selects the output modality but also conditions the attention masks so that only the speech‑codec sub‑network is active, preventing accidental text token generation.

How does the audio‑QA template differ from a standard text chat turn?

In audio‑QA the user turn begins with the token, which tells the model to treat the following segment as raw audio input rather than text. The assistant then replies with text tokens only, because the task is question answering, not generation of new audio.

The training corpus combines three pillars: (1) large‑scale text corpora (mirroring LLM post‑training data), (2) speech corpora for ASR/AST/TTS, and (3) non‑speech audio corpora for audio understanding and TTA.

**Table 2.** Audex data statistics grouped by data types and tasks. The number of samples refer to the number of input–output pairs for text and audio tasks, where partial duplications are considered as different samples (e.g. different inputs for the same output, or different outputs for the same inputs). The total audio hours are measured under 16kHz. The total audio tokens are measured with X-Codec (RVQ4) for general audio and X-Codec2 for speech. The total text tokens are measured with the Audex tokenizer. The total tokens is a simple addition of audio and text tokens.

To make ASR robust, each clean utterance is duplicated with real‑world noise and room impulse responses from the 5th DNS Challenge, producing noisy and reverberant variants while keeping the original transcript as the target.

For TTS we adopt a synthetic voice from Tortoise (high‑fidelity reconstruction with X‑Codec 2) and convert roughly 8 % of the TTS corpus using Qwen3‑TTS‑12Hz‑1.7B‑Base, then enforce a length window of 0.1 – 120 seconds to keep training stable.

**Figure.** (a) $FD_{openl3}$ ($\downarrow$) vs CFG value $\lambda$ on TTA benchmakrs. (b) WER ($\downarrow$) vs CFG value $\lambda$ on the Seed-TTS-Eval benchmark.

Training Loss and CFG

Defines the loss formulation and classifier‑free guidance that drive Audex training.

When training on many concatenated samples, naïve averaging of cross‑entropy losses can bias the optimizer toward longer sequences, causing instability and wasted compute.

The model is trained to assign high probability to each token given its left context, which directly maximizes the likelihood of the observed text.

Why maximize the log‑likelihood instead of minimizing a raw error?

Maximizing log‑likelihood is equivalent to minimizing cross‑entropy, which yields a smooth, differentiable loss that directly measures how well the model’s predicted distribution matches the true data distribution.

When several samples share a batch, we first normalize each sample’s loss by the square root of its length, then average those normalized values—this prevents long samples from dominating the gradient.

Lengths: $|L_A|=4$, $|L_B|=9$.

Square‑rooted lengths: $\sqrt{4}=2$, $\sqrt{9}=3$.

Average of square‑rooted lengths: $(2+3)/2 = 2.5$.

Sum of raw losses: $12.0 + 27.0 = 39.0$.

Scaled‑Loss = $2.5 \times \frac{39.0}{2.5} = 39.0$ (the scaling factor cancels, showing that the loss is effectively the sum of raw losses but each sample’s contribution is already balanced by the square‑root normalization).

Without the square‑root normalization, Sample B (nine tokens) would dominate the gradient, skewing learning toward longer sequences.

Why use the square root of the sample length instead of the length itself?

The square root grows more slowly than the length, so it reduces the disproportionate influence of very long samples while still reflecting their size; this yields a smoother trade‑off between short and long examples.

MoE (Mixture‑of‑Experts) layers route tokens to a subset of experts; we enforce balanced expert utilization without adding extra loss terms, using a tiny update rate and a modest top‑K scaling.

How does removing the auxiliary loss affect expert utilization?

Without an extra auxiliary term, the routing updates (controlled by the $10^{-3}$ rate) and the small load‑balancing coefficient $10^{-6}$ together keep the expert selection distribution smooth, preventing any single expert from being over‑ or under‑used.

We train the model to predict both conditioned and unconditional token distributions; at inference we blend them, amplifying the conditioned signal by a factor $\lambda$ while preserving the model’s learned prior.

Why does CFG add the unconditional distribution instead of simply scaling the conditioned logits?

Subtracting the unconditional logits removes the model’s generic prior bias, so the added term isolates the pure conditioning signal; scaling this difference amplifies the effect of the condition without distorting the learned vocabulary probabilities.

Training Pipeline and Stages

Describes the padding trick for CFG and the staged fine‑tuning pipelines.

Classifier‑free guidance (CFG) requires occasional replacement of real tokens with a neutral placeholder. Performing this replacement on‑the‑fly breaks the parallel data pipeline, so we pre‑process the dataset instead.

Instead of swapping tokens during training, we turn a small slice of the audio corpus into pure padding sequences ahead of time, preserving batch parallelism.

Original batch token counts: [5, 5, 5].

Select caption 2 → replace its tokens with $\emptyset$ → new token counts: [5, 5, 5] (length unchanged).

During training the model receives: real tokens for captions 1 & 3, all‑padding for caption 2.

The loss on the padded example is zero because $\emptyset$ contributes no gradient, yet the batch remains fully parallel.

This shows how CFG can be enabled without any runtime token‑swap logic, preserving the efficiency of the existing data loader.

How does this differ from the on‑the‑fly token flipping used in other CFG implementations?

On‑the‑fly flipping requires a conditional branch inside the training loop, breaking the uniform batch shape and forcing extra synchronization. Pre‑processing creates a static dataset, so the same high‑throughput pipeline can be used unchanged.

With CFG in place we turn to supervised fine‑tuning (SFT). Two strategies are explored: a multi‑stage pipeline that adds tasks gradually, and a single‑stage pipeline that trains everything together after a brief audio warmup.

We train the model floor‑by‑floor, first solidifying the text foundation, then adding audio components stepwise, so each new floor rests on a stable base.

Stage 1 loss computed on 2‑token sequences; parameters updated only in the text sub‑network.

Stage 2 introduces 3 padding audio tokens; only the new audio embedding matrix and MLP adapters receive gradient updates.

Text token embeddings remain unchanged, preserving the language knowledge learned in Stage 1.

After Stage 2 the model can embed audio tokens but cannot yet generate them.

This illustrates why isolating parameter updates per stage protects previously acquired capabilities.

Why freeze the text token embeddings during the audio warmup stage?

Training on audio data alone drives the shared transformer layers toward audio‑specific patterns, which can overwrite the delicate language representations. By freezing the text embeddings we keep the language subspace intact while still allowing the new audio components to learn.

After a brief audio warmup we throw all modalities into one big training run, letting the model learn cross‑modal interactions from the start.

How does the single‑stage approach avoid the degradation risk that motivated the multi‑stage pipeline?

The initial audio warmup still gives the audio components a head start, but the subsequent joint training quickly aligns them with the already‑stable text representations, preventing the prolonged isolation that caused drift in the multi‑stage case.

**Table 3.** Data blending weights (upper) and key training hyper-parameters at different SFT stages.

Training Stage Results

We show how freezing text embeddings during audio warmup preserves language ability while boosting audio performance.

Fine‑tuning a multimodal LLM on audio data often erodes the strong text abilities it was pretrained with. The authors therefore introduce a lightweight “audio warmup” that isolates the text sub‑space during the first SFT stage. This resolves the regression without sacrificing audio learning.

During the initial audio‑only SFT stage we mask gradients so that only the audio token embeddings are updated, leaving the pretrained text embeddings untouched.

Apply the gradient‑mask: set $g_t \leftarrow (0,0,0,0)$ while keeping $g_a$ unchanged.

Update the audio embedding: $a \leftarrow a - \eta \, g_a$ (e.g., with learning rate $\eta=0.1$).

Leave the text embedding unchanged: $t \leftarrow t$.

After one step, the audio embedding moves toward better audio representations, while the text embedding retains its original values.

Masking the text gradients isolates the language knowledge, preventing any drift caused by the audio loss.

How does this differ from simply training all embeddings together?

When all embeddings are trained, the audio loss back‑propagates into the text embedding matrix, overwriting the pretrained lexical patterns. The freeze‑text mask blocks that back‑propagation, so text knowledge stays intact while audio embeddings still receive the full learning signal.

**Figure 5.** Audio Gen. SFT Results between Freezing and Unfreezing Text Embeddings in Audio Warmup.

Main Results Overview

Audex‑30B‑A3B matches text performance while excelling in audio tasks.

Table 1 summarizes Audex‑30B‑A3B’s performance across text‑only and audio benchmarks, highlighting where it improves upon its backbone and where it lags behind proprietary models.

Audex‑30B‑A3B reaches 93.3 % accuracy on IMO AnswerBench, surpassing its text‑only backbone by +0.5 %.

Table 1 reports 93.3 % for Audex‑30B‑A3B versus 92.8 % for Nemotron‑Cascade‑2‑30B‑A3B.

**Table 1. Main results.** We report the results of Nemotron-Lab-Audex-30B-A3B alongside those of its text-only backbone, Nemotron-Cascade-2-30B-A3B. For the baseline models and their text-only backbones, we use numbers from the official report when available; otherwise, we evaluate them, as indicated by $\ddagger$, using the recommended settings. We mark tool-integrated reasoning results with $\diamond$ for math and code reasoning.

Overall, Audex‑30B‑A3B delivers a balanced profile: it preserves text capabilities, excels in audio tasks, and remains competitive with proprietary systems.

SFT Comparison and Hyperparameters

We compare multi‑stage versus single‑stage SFT and detail the training hyper‑parameters.

This section evaluates the two SFT curricula—multi‑stage, which adds capabilities incrementally, and single‑stage, which consolidates all tasks at once—while also exposing the hyper‑parameters that govern each stage.

**Figure 3.** Audex training stages. In the SFT stage, we study two training curriculums: (1) the multi-stage curriculum – which adds one type of capabilities at a time, and (2) the consolidated single-stage curriculum – which combines all types of task together. The resulting model is called Audex SFT. We then apply cascaded RL in the text domain similar to Nemotron-Cascade 2, and obtain Audex.

Multi‑stage SFT retains long‑context performance while single‑stage SFT collapses on NIAH.

Table 4 reports NIAH scores of 99.3 for multi‑stage versus 86.8 for single‑stage.

Training hyper‑parameters are listed in Table 3: a global batch size of 64, cosine LR schedule from $5\times10^{-5}$ down to $5\times10^{-6}$, and AdamW ($\beta_1\!=\!0.9$, $\beta_2\!=\!0.98$, weight decay $0.1$). Module‑specific adapters use smaller batches (16) and lower peak LR ($2\times10^{-5}$).

Speech Interaction Evaluation

Audex models set new highs on speech benchmarks, reaching 97.6% accuracy and a 90 overall VoiceBench score.

We evaluate speech interaction (speech‑to‑text and speech‑to‑speech) with a fully unified cascade: the same Audex model handles automatic speech recognition, textual reasoning, and optional text‑to‑speech generation.

**Table.** Benchmarks Results at Different Stages

Audex 30B‑A3B scores 90 on VoiceBench, the highest overall among the evaluated models.

Table 14 shows Audex 30B‑A3B achieving a composite VoiceBench score of 90, surpassing the next best entry by 5 points.

**Table 14.** Evaluation Results on VoiceBench.

Even the 2B‑parameter Audex variant outperforms several strong 7B‑plus baselines, showing that the unified architecture scales gracefully from modest to large sizes.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers