AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
Ziyang Ma, Zhikang Niu, Wenming Tu, Tianrui Wang, Ruiqi Yan, Junxi Liu, Yanru Huo, Nickk Huang, Yang Liu, Qicong Xie, Zeyu Xie, Hui Wang, Haitao Li, Zixuan Jiang, Yalin Li, Jie Fang, Yifan Duan, Zeyue Tian, Guangzheng Li, Haina Zhu, Shuyi Wang, Jinwen Wang, Mingyu Cui, Tian Tan, Auden, Sen Liang, Steve Yves, Shan Yang, Liefeng Bo, Zilong Zheng, Kai Yu, Eng-Siong Chng, Xie Chen
AuK is a unified foundational model for speech generation and editing that uses a hybrid flow-transformer to handle diverse audio tasks via natural-language instructions.
How can a single foundational model unify diverse speech tasks—generation, editing, and enhancement—under a shared natural-language instruction interface?
Speech systems are currently fragmented into task-specific models, forcing users to switch between different tools for generation, editing, and enhancement. This lack of a unified interface makes it difficult to perform complex, multi-step audio workflows. AuK solves this by mapping natural-language instructions and optional audio context to a target waveform using a hybrid transformer backbone that combines dual-stream MMDiT blocks for semantic-acoustic interaction and single-stream DiT blocks for latent refinement. The resulting model achieves leading performance on zero-shot generation and instruction-guided editing, while the distilled AuK-Flash variant provides a 4.5× speedup over the teacher model with 4-step inference.
Paper Primer
AuK unifies five task families—generation, content editing, enhancement, paralinguistic editing, and acoustic editing—under a single interface. The core mechanism hinges on a hybrid transformer backbone that processes semantic and acoustic streams through dual-stream MMDiT blocks before merging them into single-stream DiT blocks to predict the rectified-flow velocity of the target latent.
AuK-Flash achieves a 4.5× wall-clock speedup over the full 32-NFE teacher model.
The model uses consistency initialization and task-routed Decoupled DMD to distill the teacher into a 4-step, CFG-free student.
To handle the distinct requirements of generation and editing, the authors employ a two-stage post-training strategy: flow-based preference optimization using human feedback for open-ended editing, and Flow-GRPO with automatic rewards for content correctness and style consistency in generation.
Why does the model use a hybrid dual-stream and single-stream transformer architecture?
The dual-stream MMDiT blocks allow for bidirectional interaction between semantic and acoustic streams while preserving distinct residual pathways, while the subsequent single-stream DiT blocks jointly refine the fused sequence to predict the target latent velocity.
How does the model handle tasks that require different levels of acoustic control?
The model uses a shared VAE latent space for reference-audio conditioning; for tasks without reference audio, the acoustic stream contains only the noisy target latents, allowing the same backbone to support both text-only generation and reference-conditioned editing.
AuK demonstrates that a single, instruction-conditioned flow-matching backbone can replace fragmented task-specific audio models, providing a scalable path for unified speech synthesis and manipulation.
Introduction and Overview
We motivate a single model that handles speech generation, editing, and enhancement via natural‑language instructions.
Current speech systems treat generation, editing, and enhancement as separate modules, forcing users to juggle multiple models and interfaces. This fragmentation wastes modeling effort and creates a disjoint user experience, motivating a single foundation that can follow free‑form instructions to produce the desired speech.
Three obstacles make unification hard: (1) output constraints differ—generation creates new audio while editing modifies only selected regions; (2) conditioning varies—some tasks need text alone, others require joint reasoning over text and reference audio; (3) supervision is heterogeneous—objective metrics suffice for generation, but open‑ended editing demands subjective judgments of naturalness and attribute preservation. AuK addresses these by defining a shared instruction‑plus‑audio interface and a hybrid flow Transformer that fuses semantic and acoustic streams.
**Figure 2.** Versatile speech generation and editing capabilities of AuK. The model supports five task families: (1) speech generation, including instruction-based and zero-shot TTS; (2) acoustic editing of speaking rate, loudness, and pitch; (3) paralinguistic editing of emotion, accent, nonverbal vocalizations, timbre, and whisper style; (4) content editing of spoken words and song lyrics; and (5) enhancement and separation of speech and music. The examples illustrate how natural-language instructions and speech input are mapped to output speech.
AuK provides a unified, instruction‑based interface for speech generation, editing, and enhancement.
Pre-training Task Families
Describes the five task families and the speech‑generation data used for pre‑training.
We organize the pre‑training data into five task families, each exposing a common instruction‑plus‑audio interface.
**Figure 3.** Overview of the pre-training corpus. The inner ring groups tasks into five capability families, while the outer rings summarize their instruction-level operations. The surrounding panels illustrate representative tasks and their intended functions. Sector widths are adjusted for readability and do not indicate data volume.
All pre‑training tasks share a single interface: a textual instruction, an optional audio prompt, and the target waveform to be produced.
Why does a single model benefit from grouping such diverse tasks under one interface?
Because the shared instruction‑audio format lets the model reuse its latent representations across tasks, reducing the need for task‑specific heads and enabling zero‑shot transfer when a new instruction appears.
Instead of requiring a transcript for the reference utterance, we treat each pair of utterances from the same speaker as a prompt‑target duo, letting the model learn to clone voice from audio alone.
Form unordered pairs: (U1,U2), (U1,U3), (U2,U3).
For each pair create two instances: prompt = U1, target = U2 and prompt = U2, target = U1, etc., giving 3 × 2 = 6 training examples.
Each instance contains only the prompt audio and the target text; no transcript of the prompt is used.
This combinatorial pairing dramatically expands speaker‑level data while keeping the learning signal purely acoustic.
How does this differ from conventional zero‑shot TTS that uses a reference transcript?
Conventional approaches need the reference transcript to align text and audio, tying the model to an external ASR. Our formulation removes that dependency, learning the voice mapping directly from audio, which simplifies deployment.
Each training example pairs a free‑form natural‑language caption describing voice characteristics with the target text, teaching the model to generate speech that matches the described style.
Why can the model learn voice style without any reference audio?
Because the training captions explicitly enumerate acoustic attributes, the model learns a mapping from textual descriptors to latent voice embeddings, allowing it to synthesize the described style on demand.
Deterministic signal‑processing transforms generate paired examples that teach the model to modify speaking rate, pitch, or loudness while keeping content and speaker identity intact.
Apply a +2 semitone pitch shift, preserving duration; pitch becomes ≈1.12 kHz.
Apply a +10 dB loudness gain; RMS rises to 10 dB.
Run peak‑protection: if any sample exceeds 0 dBFS, scale the waveform down to the clipping threshold.
Pair the edited waveform with the original and the instruction “raise pitch by 2 semitones and increase loudness by 10 dB”.
The pipeline creates realistic edit pairs while guaranteeing audio safety through peak protection.
What would happen if peak‑protection were omitted after a loudness increase?
Without peak‑protection the waveform could clip, introducing distortion that the model would learn as part of the edit, degrading the quality of downstream synthesis.
Paralinguistic Editing Data
We build paired supervision for editing paralinguistic attributes while keeping the spoken content unchanged.
Paralinguistic aspects like emotion, timbre, accent, and non‑verbal cues affect how speech is perceived, yet most datasets only provide transcript‑level supervision. To train a model that can edit these attributes on demand, we need paired examples that isolate the desired change while keeping the spoken content fixed.
We create paired source–target audio samples where only the targeted paralinguistic attribute changes, and the transcript stays identical, so the model learns to modify that attribute on demand.
Qwen3‑TTS‑CustomVoice synthesizes a happy‑tone reference utterance of the same transcript using speaker S.
IndexTTS2 receives the original neutral audio to preserve speaker characteristics and the happy reference to inject the desired emotion.
IndexTTS2 outputs an edited waveform where the words “I am fine.” sound happy while the speaker’s voice remains unchanged.
The edited waveform is paired with the original audio and the instruction “make the utterance sound happy”.
This example shows that the model can learn to alter prosody and affective cues without touching the lexical content, because the only difference between source and target is the emotion‑specific acoustic pattern.
With these paired samples covering five paralinguistic dimensions, the model can be trained to follow natural‑language instructions that modify how speech sounds while preserving what is said.
Model Architecture
AuK fuses semantic and acoustic conditions via dual‑stream blocks then refines them with single‑stream blocks.
The AuK model must support both text‑only generation and audio‑conditioned editing, requiring a unified architecture that can ingest optional reference audio.
AuK treats the instruction and any reference audio as two parallel streams that are first processed separately and then merged—like a mail sorter that routes letters to language‑specific bins before a single courier delivers the combined package.
How does AuK differ from a conventional transformer that simply concatenates instruction and audio embeddings?
AuK preserves modality‑specific processing through dual‑stream MMDiT blocks with separate residual pathways, allowing each stream to attend to the other without overwriting its own information before the final concatenation.
Layer‑norm each hidden vector (no change here) and compute $c_{\text{sem}} = 0.6\cdot[1,0,0,0] + 0.4\cdot[0,1,0,0] = [0.6,0.4,0,0]$.
Reference audio yields VAE latent $z_{\text{ref}}=[0,0,1,0]$, so $c_{\text{ac}} = [0,0,1,0]$.
Project both conditions (identity projection for simplicity) to obtain $s^{(0)}=[0.6,0.4,0,0]$, $a^{(0)}=[0,0,1,0]$.
Dual‑stream MMDiT block attends cross‑modally, producing updated streams $s^{(1)}=[0.5,0.5,0,0]$, $a^{(1)}=[0,0,0.9,0.1]$ (illustrative values).
Concatenate $[s^{(1)};a^{(1)}] = [0.5,0.5,0,0,0,0,0.9,0.1]$ and feed through DiT block, yielding $v_t = [0.45,0.45,0.85,0.15]$.
The VAE decoder maps $v_t$ to a short waveform segment (e.g., a 4‑sample audio snippet).
The example shows how a tiny semantic vector and an acoustic latent are merged, refined, and turned back into audio, illustrating the flow of information through the dual‑stream and single‑stream stages.
Forward pass of AuK for a single flow step.
**Figure 4.** **Architecture of AuK.** (a) The framework maps a user instruction and optional input audio to complementary semantic and acoustic conditioning streams. A multimodal language model encodes the instruction and audio context, and a learnable weighted sum of its layer-wise hidden states forms the semantic condition. In parallel, the VAE maps input audio, when present, to reference latents, which are combined with noisy target latents to form the acoustic condition. The two streams exchange information through $M$ dual-stream MMDiT blocks before being concatenated and refined by $N$ single-stream DiT blocks. The predicted latent is decoded by the VAE to produce the output audio. (b) Each MMDiT block performs joint attention over semantic and acoustic tokens while preserving stream-specific residual pathways. (c) Each DiT block applies self-attention to the fused token sequence. The symbols $+$ and $C$ denote addition and concatenation, respectively; snowflakes denote frozen modules.
VAE Training and Latent Space
Training AuK combines a compact VAE with a unified rectified‑flow pre‑training.
Training AuK proceeds in two stages: first a VAE builds a compact latent space, then a unified pre‑training teaches the model to generate, edit, and enhance audio using a rectified‑flow objective.
The AuK‑VAE compresses 24 kHz waveforms into a 64‑dimensional latent that can be conditioned on reference audio and later decoded back to high‑fidelity waveforms.
How does AuK‑VAE differ from a standard VAE?
AuK‑VAE adds a normalizing flow $F$ that reshapes the posterior before KL regularization, and its decoder follows a BigVGAN architecture with causal, anti‑aliased convolutions, both of which are absent in a vanilla VAE.
Instead of training generation and editing separately, the model learns a velocity field that turns random noise into a clean latent, while hierarchical condition dropout forces it to succeed even when some conditioning signals are missing.
Compute the noisy latent: $z_{0.5} = (1-0.5)z_0 + 0.5 z_1 = (1, 2)$.
Target velocity is $v_{0.5} = z_1 - z_0 = (2, 4)$.
Model predicts $\hat{v}_{0.5} = (2.1, 3.9)$ (example output).
Apply the mask: only the first frame contributes, so $m \odot (v_{0.5} - \hat{v}_{0.5}) = ( -0.1, 0.1)$.
Loss $L_{\text{FM}} = \frac{(-0.1)^2 + (0.1)^2}{1} = 0.02$.
Masking ensures that padded frames do not generate spurious gradients, letting the model focus on real audio content.
Why does condition dropout differ from ordinary dropout?
Ordinary dropout randomly zeroes individual neurons, whereas hierarchical condition dropout removes whole conditioning modalities (acoustic, then both acoustic and semantic) to teach the model to operate under missing cues, which is essential for classifier‑free guidance.
Training data consists of 3 million hours of speech, music, and general audio sampled at a 6 : 3 : 1 ratio, split into 1.28‑second segments at 24 kHz.
Task‑family sampling in stage 2 uses the probabilities: Speech Generation 28.10 %, Content Editing 23.02 %, Enhancement 23.17 %, Paralinguistic Editing 21.75 %, Acoustic Editing 3.96 %.
Dynamic length‑bucketed batching groups utterances (1–35 s) so each accelerator processes at most 10 000 latent frames or 24 utterances, yielding a global batch of up to 6 144 utterances (~14 h of audio) per optimizer step.
Optimization runs on 256 GPUs with DeepSpeed ZeRO‑2 and bf16 precision, using fused AdamW ($\beta = (0.9, 0.95)$), a peak learning rate of $1\!\times\!10^{-4}$, gradient clipping to norm 1.0, and an EMA (decay 0.9999) after 100 updates.
Post-Training and Alignment
Fine‑tune the backbone to respect human editing preferences and improve generation quality.
Editing tasks cannot be judged by a single automatic metric because naturalness, transformation degree, and contextual fit are inherently subjective.
Human feedback is used to steer the model toward the kinds of edits that people deem successful, while keeping the underlying encoder and VAE frozen.
Form the preference pairs: $(2\succ1)$, $(2\succ0)$, $(1\succ0)$.
Compute logistic loss for each pair using $\ell(x)=\log(1+e^{-x})$ with label smoothing $\epsilon=0.05$.
Average the three losses to obtain $L_q$ for this relation.
Since only relation $2\succ0$ appears, set $\omega_{2\succ0}=1$ and $L_g = L_q$.
Back‑propagate the gradient to update only the Transformer backbone.
The loss depends solely on the relative ordering, not on how many candidates were generated, which keeps training stable even when some instructions yield many candidates.
How does this Editing Preference Optimization differ from standard DPO used for text generation?
Standard DPO compares a policy to a frozen reference on the same input sequence, but here the comparison is performed on a shared noise sample $z_0$ and flow time $t$ so that differences are attributable to the edited latent rather than stochastic sampling. This flow‑matching trick isolates the effect of the edit itself.
After aligning the model with human editing preferences, a second stage applies reinforcement learning to boost zero‑shot and instruction‑following speech generation.
Prompt Construction builds two frozen pools: 10 000 zero‑shot prompts (balanced Chinese/English) and 5 000 instruction‑following prompts, each stratified by duration and style difficulty.
Policy updates are driven by candidates generated from diverse noise trajectories, but gradient computation is confined to a low‑SNR window to reduce variance and cost.
Reward Design combines content correctness, speaker similarity, and style consistency into task‑specific scores.
Content correctness is measured by an ASR model, while speaker similarity is measured by cosine similarity of speaker embeddings; the two are fused with a temperature‑controlled interpolation.
A multimodal style‑consistency model judges whether a candidate matches the requested speaking style, and majority voting over multiple queries yields a binary style score.
Optimization runs for 500 updates with $G=16$ candidates per prompt, using classifier‑free guidance 2.0, a sway coefficient $-1.0$, and $\lambda=0.7$.
**Table 1.** Fixed per-batch sampling probabilities used in the second pre-training stage for the five task families introduced in Sec. 2.
Model Acceleration and Distillation
Accelerating AuK by distilling the post‑trained model into a fast, CFG‑free student.
The post‑trained AuK model requires many iterative flow steps and classifier‑free guidance (CFG), making inference slow and costly for both generation and editing.
AuK‑Flash compresses the full AuK model into a four‑step, CFG‑free student by first learning a stable few‑step initialization and then refining it with a task‑routed Decoupled DMD that preserves separation ability.
How does AuK‑Flash differ from standard diffusion distillation that simply copies the teacher’s weights?
Standard distillation copies the teacher’s parameters but retains the same iterative sampling and CFG, so inference remains slow. AuK‑Flash adds a consistency‑initialization step that teaches the student to jump directly to the endpoint, and then uses Decoupled DMD with APG to remove CFG entirely, yielding a four‑step student that is both fast and instruction‑aware.
The teacher advances $z_3$ to $z_2$ with one CFG‑guided velocity step.
The student predicts $z_1$ directly from $z_3$.
The teacher also advances $z_2$ to $z_1$ (single step).
The student’s prediction from $z_3$ is matched to the teacher’s $z_1$ prediction, and gradients flow only through the $z_3arrow z_1$ path.
By forcing the student to jump straight to the endpoint, it learns a stable few‑step flow that can later be refined without accumulating error.
Both stages share the same data distribution as the unified pre‑training, and training runs on 256 GPUs with a constant learning rate of 10^{-5} and global gradient clipping at 1.0.
For each sample $i$ with task label $\tau_i$, the routed objectives are $L^{\theta}_i = \| \hat{z}_{1,i} - z_{1,i} \|_2^2$ for separation tasks and $L^{\phi}_i$ equal to the Decoupled DMD loss for those tasks, while non‑separation samples receive the fake‑score loss.
Inference Pipeline
Inference turns free‑form requests into model‑ready instructions and samples audio.
Free‑form user requests differ wildly in wording and often omit required parameters, which makes direct feeding to AuK brittle.
The Prompt Enhancer acts like a concierge that translates a guest’s vague request into a precise service order—identifying the task, filling missing arguments, and normalizing audio context before handing the instruction to the model.
How does the Prompt Enhancer differ from a simple template‑filling script?
A script would merely substitute placeholders, assuming the request already matches a known pattern. The Prompt Enhancer first infers the task, validates arguments against the training distribution, and uses audio context to resolve ambiguities—steps that a naïve template filler cannot perform.
ASR transcribes the reference audio to “hello”. The request $q$ is parsed, yielding a TTS task with speed multiplier $s=1.5$.
Byte length of source text $B(\text{"hello"}) = 5$, byte length of target text $B(\text{"hello world"}) = 11$.
Duration estimate $T_{\text{out}} = \frac{2\text{s} \times 11}{5} = 4.4\text{s}$ (zero‑shot TTS case).
Parameters are validated (speed multiplier within trained range) and the instruction is rendered as “TTS speed=1.5 text=‘hello world’”.
Audio preprocessing trims silence, resamples to 24 kHz, and normalizes level before passing to the VAE.
The byte‑length ratio automatically scales duration for content‑changing edits, ensuring the generated speech matches the intended pacing.
If audio is present, convert it to mono and resample to 24 kHz.
Encode the resampled audio with the VAE to obtain an acoustic reference.
Feed the enhanced instruction (from the Prompt Enhancer) and the acoustic reference to the MLLM, which produces a semantic condition.
Determine the target latent length by multiplying the estimated duration $T_{\text{out}}$ by the VAE rate of 50 Hz.
Initialize a Gaussian noise tensor of that length and run the conditional flow sampler (AuK or AuK‑Flash) to generate latent audio.
Decode the latent tensor with the VAE decoder to obtain the final waveform.
**Table 2.** Inference configurations for AuK and AuK-Flash.
Performance Evaluation
AuK and AuK‑Flash are evaluated on reconstruction, generation, and editing benchmarks.
We assess AuK and its accelerated variant AuK‑Flash on three complementary fronts: reconstruction fidelity, speech generation, and speech editing.
Table 3 provides a compact comparison against representative benchmarks, retaining each metric’s native scale; the best result in each column is highlighted in bold.
VAE Reconstruction Results
AuK‑VAE outperforms baselines on reconstruction across speech, audio, and music.
AuK‑VAE outperforms all baselines on reconstruction quality across speech, audio, and music.
Highest PESQ, STOI, Mel Dist, and STFT scores in each domain (see table).
The table compares AuK‑VAE to four strong baselines on three domains, using four perceptual and spectral metrics.
Enhancement and Separation Results
AuK‑Flash dominates the enhancement benchmarks, while AuK preserves linguistic fidelity.
This section reports the quantitative evaluation of AuK and its Flash variant across three enhancement tasks: speech enhancement, speech separation, and speech super‑resolution.
AuK‑Flash attains the best result on six of the seven super‑resolution metrics, while AuK leads on speaker‑identity preservation.
Table 11 shows AuK‑Flash achieving OVRL 3.25, SIG 3.25, BAK 3.25, UTMOS 4.05, WER 2.92 %, PER 4.19 %; AuK records the highest SIM 0.97.
The full AuK model consistently yields lower word‑error rates, indicating stronger linguistic preservation, whereas the Flash variant trades a modest increase in WER for markedly higher perceptual scores.
**Table 14.** Speech super-resolution performance on VCTK-SR.
Conclusion
We summarize AuK's contributions, releases, and future directions.
AuK is an open‑source foundational model that unifies speech generation and editing through a shared instruction‑conditioned waveform interface, built on a multimodal language model, an audio VAE, and a flex‑style Transformer for latent diffusion.
The training corpus spans five task families, comprising roughly 3.03 billion instruction–audio pairs and 1.95 million hours of effective supervision.
We employ generation‑only warm‑up, joint generation‑editing pre‑training, human‑feedback preference optimization for open‑ended editing, and reward‑based reinforcement learning to align the model across this heterogeneous capability set.
Task‑routed distillation produces AuK‑Flash, retaining broad capability with a four‑step inference pipeline, eliminating classifier‑free guidance, and delivering a 4.5× wall‑clock speedup.
Experiments demonstrate leading performance on zero‑shot and voice‑design speech generation, as well as on instruction‑guided editing, while remaining competitive on speech restoration benchmarks.
Qualitative observations suggest that unified training facilitates useful transfer across utterances, tasks, and languages.
Nevertheless, native understanding of unconstrained editing requests is still incomplete, and the system continues to rely on explicit task routing and prompt enhancement.
Future work should improve native instruction grounding, compositional generalization, scalable alignment for open‑ended audio transformations, and further reduce inference cost.
We release the source code and model weights to facilitate future development and research.
Core contributors are Ziyang Ma, Zhikang Niu, Wenming Tu, Tianrui Wang, Ruiqi Yan, Junxi Liu, and Yanru Huo.