SwanTale: Unified Multi-Speaker Speech and Audio Generation for Instruct and Zero-Shot Tasks

Yu Zhang, Ruiqi Li, Changhao Pan, Ke Lei, Xiang Yin, Cheng Yang

SwanTale unifies zero-shot and instruct-based speech/audio generation using a sparse, task-aware Transformer.

How can a single unified model generate high-fidelity speech, music, and general audio for both instruction-based and zero-shot tasks?

Modern speech systems struggle to bridge the gap between zero-shot voice cloning and instruct-based audio design, often failing to maintain speaker identity while simultaneously generating complex environmental soundscapes and local audio effects. SwanTale addresses this by using a unified flow-based Transformer that routes inputs through task-specific and frame-level experts, conditioned on both natural language captions and reference audio. The model achieves state-of-the-art performance on both zero-shot and instruct benchmarks, supporting complex multi-speaker dialogue and integrated environmental audio generation.

Paper Primer

The core challenge is task compatibility: zero-shot synthesis relies on reference audio for speaker identity, while instruct synthesis relies on natural language captions to describe environment, persona, and fine-grained content. SwanTale solves this by using a Unified Mixture-of-Experts (MoE) architecture that dynamically allocates capacity based on the task type and the specific acoustic requirements of each latent frame.

SwanTale achieves superior expressiveness and instruction-following capability compared to existing specialized models.

Evaluation on SwanBench-Speech, InstructTTSEval, and SwanBench-Scene shows the model leads on key zero-shot and instruct metrics. Top-tier performance across heterogeneous generation tasks involving multi-speaker speech, environmental audio, and local effects.

The model uses a 48 kHz waveform-latent autoencoder (SwanVAE) to compress audio into a 25 Hz latent space, which is then processed by a non-causal diffusion Transformer. To ensure high-quality output without reinforcement learning, the model uses reward-conditioned quality control, where quality metrics are provided as a controllable input flag during inference.

Why use a Unified MoE instead of a standard dense Transformer?

Processing heterogeneous audio—such as speech, singing, and environmental effects—with shared weights forces these patterns to compete for capacity. Unified MoE allows the model to specialize experts for specific tasks and acoustic states, improving performance on complex, multi-modal generation.

How does the model handle the difference between zero-shot and instruct inputs?

The model uses a task-specific context mask and caption branch. Instruct samples use full-caption supervision over the entire sequence, while zero-shot samples use content-caption supervision with reference audio provided as context.

SwanTale demonstrates that a single, sparse-expert architecture can effectively handle both zero-shot cloning and complex, caption-driven audio design, setting a new standard for unified speech and audio generation.

Introduction and Motivation

Unifying instruct and zero‑shot audio generation requires new data and a versatile model.

Creating expressive audio for animation, games, podcasts, and short‑video production often starts without any reference recordings, yet existing TTS systems focus on zero‑shot voice cloning from a short audio clip. To serve creators who must design voices from scratch and also reuse them later, a unified system must handle both zero‑shot synthesis (using reference audio) and instruct synthesis (driven solely by natural‑language captions).

The problem presents three intertwined challenges: (1) data scarcity – high‑quality, richly annotated caption data are expensive to collect; (2) task compatibility – the model must reconcile speaker information coming from captions versus reference audio without degrading either pathway; and (3) multi‑audio‑modality complexity – the system must generate speech, environmental sounds, local effects, singing, and music within a single waveform while preserving intelligibility and identity.

SwanTale is a single Transformer‑based model that can take either a reference audio clip or a detailed natural‑language caption and produce expressive multi‑speaker speech together with accompanying audio effects.

The core need is a unified audio generation framework that seamlessly supports both instruct‑driven caption control and zero‑shot voice cloning.

The SwanData-Caption Pipeline

The Data Pipeline builds fine‑grained SwanData‑Caption through coverage design, preprocessing, annotation, and refinement.

Training a unified audio model from only transcripts leaves it blind to speaker dynamics, background ambience, and fine‑grained expressive cues. SwanData‑Caption supplies the missing structured supervision that lets the model generate expressive, multi‑speaker audio without task‑specific pipelines.

SwanData‑Caption is a four‑stage pipeline that turns raw speech recordings into richly structured captions, giving the model explicit cues about environment, speakers, and content.

Environment field is filled with “an antique courtyard interior with faint

Speakers field receives “Speaker 1: a young woman, calm, low pitch, cold tone” after the annotator parses the speaker’s voice characteristics.

Content field wraps the spoken line in a speaker tag and adds expressive cues: “Speaker 1 speaks in an icy and scrutinizing tone:

This concrete three‑field structure lets the downstream model condition on scene, speaker identity, and fine‑grained delivery separately, enabling controllable generation that matches the original expressive intent.

**Figure 1.** Overview of the four-stage SwanData-Caption data processing pipeline, including coverage design, SwanData-Speech preprocessing, caption annotation, and data refinement.

By chaining these stages, the pipeline yields a unified, fine‑grained caption corpus that powers the MoE flow‑matching Transformer downstream, eliminating the need for separate task‑specific data pipelines.

Waveform-Latent Autoencoding

Balancing fidelity, compactness, and learnability, SwanVAE turns 48 kHz audio into a short latent stream for downstream flow modeling.

Designing a latent space that is both expressive enough for high‑quality audio and short enough for efficient flow‑based generation creates a three‑way tension: reconstruction fidelity, representation compactness, and downstream learnability. A naïve high‑rate latent would preserve detail but explode sequence length, while an overly aggressive compression would starve the flow model of useful signal.

SwanVAE compresses 48 kHz mono audio into a 96‑dimensional latent vector every 40 ms (25 Hz), preserving local acoustic detail while delegating long‑range structure to the downstream flow model.

How does SwanVAE differ from a conventional VAE that uses a deep Transformer decoder?

Standard VAEs rely on a single, often deep, Transformer to generate the entire waveform from a latent, incurring quadratic cost in sample length. SwanVAE limits the decoder to a local Resampling Block that processes only six output tokens per latent frame, pushing long‑range modeling to the downstream flow network and dramatically reducing compute.

Stage 1 (×4): 1920 → 480 samples.

Stage 2 (×4): 480 → 120 samples.

Stage 3 (×4): 120 → 30 samples.

Stage 4 (×5): 30 → 6 samples.

Stage 5 (×6): 6 → 1 latent vector.

The single latent vector is 96‑dimensional and represents the entire 40 ms window.

This toy walk‑through shows how a modest 1920‑sample chunk collapses to one latent frame, explaining why the downstream flow model only needs to handle a 25 Hz sequence despite the original 48 kHz sampling.

**Figure 2.** Overview of SwanTale. Figure (a) shows the architecture of SwanTale, and Figure (b) shows Unified MoE. In (a), the zero-shot path supplies reference audio, while both tasks share text and caption. In (b), a task router selects experts at the sample level, while an audio router applies Top-P routing over frame-level audio and null experts.

Flow-Matching Transformer Backbone

Eliminate repetition in audio generation by a non‑causal flow‑matching Transformer.

Autoregressive audio generators often repeat sounds or become unstable when long‑range dependencies clash with local effects. The trick is to replace the causal denoiser with a non‑causal flow‑matching Transformer that can see the whole latent trajectory at once.

SwanVoice is a caption‑driven audio model that learns to map textual descriptions to waveform latent trajectories.

The model treats the entire latent audio trajectory as a fluid that can be reshaped in one shot, avoiding the step‑by‑step constraints that cause repetition.

Compute the noisy input $\tilde{x}^{(\text{zero})}_t = (1-m)\odot\big((1-t)\epsilon + t x^\star\big)$ with $t=0.5$: only frames 3‑4 receive $(0.5\epsilon + 0.5 x^\star)$.

Feed $\tilde{x}^{(\text{zero})}_t$ and the context latent $r(\text{zero}) = m\odot x^\star$ into the DiT backbone.

The backbone predicts a velocity $\hat{v}_\theta$ for frames 3‑4; assume it outputs [[0.06,0.07],[0.08,0.09]].

Apply the masked loss: only the error on frames 3‑4 contributes, i.e. $\big(\hat{v}_\theta - (x^\star-\epsilon)\big)^2$ for those frames.

Back‑propagation updates the model to reduce the error on the generated region while leaving the masked reference untouched.

The mask lets the same network learn to generate new audio while preserving existing reference frames, which is why a single non‑causal backbone can serve both instruct and zero‑shot modes.

How does this Flow‑based Transformer differ from a standard causal Transformer used for speech synthesis?

Standard causal Transformers attend only to past tokens, forcing generation to proceed step‑by‑step and often leading to repeated patterns. The Flow‑based version is non‑causal: it sees the entire latent sequence, injects conditioning everywhere via cross‑attention, and predicts a velocity field that can reshape the whole trajectory in one pass, eliminating the autoregressive bottleneck.

The conditioning stack splits caption information from generic text so that fine‑grained audio descriptors (e.g., “rainy street”) do not overwhelm the model’s language understanding.

Quality scores (STOI, PESQ, etc.) are turned into a controllable caption so the model learns how high‑quality audio looks, then at inference the highest‑quality caption is forced.

Engram acts like a fast‑lookup memory for recurring caption n‑grams, letting the model instantly recognize fixed patterns while still planning long‑range acoustic structure.

Unified Mixture-of-Experts

Unified MoE gives the Transformer a task‑aware, frame‑wise capacity boost.

Dense feed‑forward layers force heterogeneous speech, music, and environmental sounds to share a single set of weights, causing competition between very different acoustic patterns.

Unified MoE is a dynamic sparse feed‑forward module that hands out extra capacity where the audio signal needs it—first at the sample level (task router) and then at each latent frame (audio router).

How is Unified MoE different from a standard Mixture‑of‑Experts layer?

Standard MoE selects a fixed number of experts per token and usually shares the same routing logic across all layers. Unified MoE adds a two‑level routing scheme: a task‑level router that picks a shared expert set for the whole sample, and a frame‑level router that dynamically chooses a variable‑size subset of audio experts per latent frame using a Top‑P threshold that changes with diffusion time. This hierarchy lets the model allocate capacity where the acoustic content is most demanding while keeping overall compute low.

Compute the cumulative probability of the sorted logits: 0.8 + 0.3 = 1.1 → capped at 1.0, so the Top‑P threshold p(t) = 0.7 selects the first two routed experts.

Apply the load‑correction bias $b_i$ (assume b = 0) and obtain mixture weights \tilde{$\pi$} = softmax([0.8, 0.3]) ≈ [0.71, 0.29].

Pass the frame representation h through the two selected audio experts $E^{audio}_1$ and $E^{audio}_2$, weighting their outputs by 0.71 and 0.29 respectively.

Since the null expert was not selected, no skip path is added; the final MoE‑FFN output is the sum of the task‑shared expert output and the weighted audio‑expert output.

Top‑P routing lets quiet frames (e.g., silence) skip the expensive audio branch entirely, while frames with rapid speaker changes automatically pull in more experts.

Implementation and Training

Key results show SwanVAE outperforms codecs while scaling to massive training compute.

SwanVAE achieves the highest ViSQOL score (4.1269) among six audio models on the general‑audio benchmark.

Table 3 reports ViSQOL 4.1269 for SwanVAE, surpassing the next best EnCodec (4.09) and the lowest LSD (0.9358) for Stable Audio Open 1.0.

The model contains 407 M parameters (encoder 51.7 M, bottleneck 0.3 M, decoder 355 M) and processes 48 kHz mono waveforms into 96‑dimensional latents at $25$ Hz, yielding a nominal bitrate of 38.40 kbps.

**Table 1.** General Audio

Across downstream tasks, SwanVAE‑based SwanTale matches or exceeds strong baselines on zero‑shot and instruct benchmarks while using a unified MoE flow‑matching Transformer, confirming the paper’s claim of a single model handling diverse audio generation without task‑specific fragmentation.

Scaling to 100 k GPU‑hours and billions of training hours is the primary driver of SwanVAE’s quality gains.

Performance Evaluation

Key quantitative results on reconstruction, zero‑shot TTS, and instruction following.

SwanTale unifies instruct‑based and zero‑shot audio generation by merging a fine‑grained caption pipeline with a Mixture‑of‑Experts flow‑matching Transformer.

SwanVAE improves ViSQOL on general audio by +0.0129 over the runner‑up.

Table 4 shows SwanVAE 4.1269 versus Stable Audio 4.0355.

Zero‑shot TTS evaluation (Table 5) shows SwanTale surpassing all baselines in Timbre Consistency, Expressive Richness, and Hierarchy for both monologue and dialogue settings.

SwanTale raises Timbre Consistency to 0.95 and cuts Content Error to 0.086 in the dialogue setting.

Reported improvements over the SwanVoice checkpoint.

**Table 1.** Music

Instruct‑task evaluation (Table 6) reveals SwanTale achieving the highest Chinese APS (86.1) and tying for the best English APS (84.2), confirming strong acoustic control.

**Table 5.** Zero-shot monologue and dialogue TTS results on SwanBench-Speech. All scores are reported to two decimal places except Content Error, which is reported to three decimal places. Bold and underlined values indicate the best and second-best results among the compared systems within each setting, respectively.

SwanBench‑Scene (Table 7) demonstrates SwanTale’s overall superiority, with the top Mean MOS of 3.88 and the best scores on Overall Expressiveness, Prosodic Naturalness, and Audio Fullness.

**Table 6.** Instruct TTS results on InstructTTSEval. Results for all models other than SwanTale are taken from the VoxCPM2 paper [116]. Bold and underlined values indicate the best and second-best results.

Ablation on SwanBench‑Caption (Table 8) shows that removing Unified MoE drops Instruction Accuracy from 3.39 to 3.02, a +0.37 gain when MoE is present.

Unified MoE improves Instruction Accuracy by +0.37 (from 3.02 to 3.39).

Table 8 ablation results.

Scaling the caption encoder from 8 B to 32 B further lifts Instruction Accuracy to 3.70, Acoustic Quality to 4.34, and Overall Expressiveness to 3.98.

**Table 6.**

Ablations and Supplemental Data

Ablation study of MoE components and their impact on instruction accuracy and expressiveness.

SwanTale unifies instruct and zero‑shot audio generation via a fine‑grained caption pipeline and a MoE flow‑matching Transformer.

The table presents an ablation study comparing three model settings: "SwanTale w/o MoE", "SwanTale", and "SwanTale w/ 32B CE". The metrics evaluated are "Instruction Accuracy", "Acoustic Quality", and "Overall Expressiveness", with upward arrows indicating that higher values are better.

**Table 10.** Condensed style matrix for short-drama and film/TV-drama-style captions.

**Table 11.** Condensed style matrix for advertisement and digital-human-style captions.

**Table 12.** Utterance-level accuracy (%) of SwanVerifier on the held-out labeled split.

The MoE components are the primary driver of the observed gains in instruction accuracy and overall expressiveness; without them the model’s performance drops across all evaluated dimensions.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers