SEPGEN: Multi-Stem Audio-Video Separation and Generation in a Single Model

Aviad Dahan, Rajaei Khatib, Yonatan Bitton, Idan Szpektor, Lior Wolf, Raja Giryes

SepGen turns joint audio-video generators into multi-stem synthesizers that output individual waveforms for each captioned source.

How can a single model perform both audio-visual generation and source separation by treating stems as addressable, caption-conditioned components?

Joint audio-video generators synthesize a single audio-mix, making it impossible to move, remove, or rerender individual sound sources like a specific speaker or instrument. SepGen appends per-source stem channels to a pretrained generator's audio stream, using a low-rank adapter and block-diagonal attention to ensure stems remain caption-faithful without altering the backbone's original output. In a single sampling run, the model generates the video, the mixed soundtrack, and one waveform per captioned source, outperforming cascade baselines on both generation and separation tasks.

Paper Primer

The core mechanism hinges on a "protected" audio-mix channel that bypasses the adapter, ensuring the backbone's original soundtrack remains intact while stems attend to the mix and each other. To prevent cross-stem leakage, the model uses block-diagonal caption routing and Cross-Stem Attention Guidance, which steers each stem away from its sibling's caption.

SepGen produces stems that adhere to their captions with significantly less cross-stem leakage than cascade baselines.

In generation benchmarks, SepGen leads cascade baselines across all metrics, with the widest lead in J-swap (a measure of caption-source binding). SepGen achieves a J-swap of 2.07 compared to significantly lower scores for cascade baselines, which often return the same source twice when queried for two similar items.

The model maintains high-quality separation performance on real-world recordings while retaining its generative capabilities.

In user studies on real recordings (SAM-Audio-bench), listeners preferred SepGen stems in 64% of judgments compared to 20% for the strongest cascade baseline. SepGen consistently outperforms state-of-the-art language-queried separators (e.g., AudioSep, FlowSep) on both Judge ratings and swap margins.

Why is a joint generation approach superior to simply running a separator on the output of a standard audio-video generator?

Cascade separators process each query in isolation, which often leads to "collapse" where both queries return the same dominant source. SepGen predicts all stems jointly from the shared audio-mix, allowing them to be complementary and bound to their specific captions.

How does the model switch between generation and separation modes?

The modes are selected by the noise level of the audio-mix during sampling: separation keeps the observed audio-mix at zero noise, while generation denoises it together with the stems.

Introduction and Motivation

We expose the monolithic audio issue and motivate generating addressable stems.

Joint audio‑video generators synthesize a scene but emit only a single mixed soundtrack, leaving individual sources inaccessible for editing or rerendering.

Current generators bundle all sound sources into one waveform, so you cannot move, remove, or replace any single source after generation.

SepGen augments a pretrained generator to emit, in a single sampling run, the video, the mixed soundtrack, and one waveform per captioned source, enabling both generation of multi‑stem audio and separation of existing mixes.

**Figure 1.** SepGen overview and motivation. (a) Joint audio-video generation emits one soundtrack with no addressable source: the video is lifted to 4D and rerendered from a new viewpoint, but both ears hear the same mix. (b) SepGen emits one waveform per captioned source, each placed at its source in the lifted scene and rendered as spatial sound from the new viewpoint (illustration). A demonstration video is available on the project page.

The shift from a monolithic audio mix to addressable stems unlocks flexible scene manipulation and high‑fidelity audio‑visual synthesis.

Unified Architecture

SepGen extends joint generation to output per‑source audio stems via a token‑concatenated audio stream.

Existing audio‑visual generators emit a single mixed soundtrack, which prevents downstream control over individual sources. SepGen addresses this limitation by producing separate waveforms for each captioned stem while retaining a unified video output.

Current generators synthesize a video together with one mixed audio track, treating the soundtrack as an indivisible whole.

SepGen augments the audio stream with extra tokens that represent each source, concatenates them with the mixed‑audio token, and uses a low‑rank LoRA to adapt only those new tokens while keeping the pretrained backbone frozen.

Concatenate them: $z_a$ = [ (1,0,0,0) | (0,1,0,0) | (0,0,1,0) ] → a 12‑dim token sequence.

Stem 1 token attends to its own vector, the stem 2 vector, the mix vector, the video token, and its caption q₁.

Stem 2 token attends similarly, but uses its own caption q₂.

The mix token attends only to the scene caption c, never to the stem tokens.

During decoding, the model predicts velocities for each of the three channels, which are then transformed back to waveforms.

The concatenation preserves temporal alignment across stems and mix, enabling the cross‑attention mechanism to share timing information while keeping source‑specific content isolated.

How does SepGen differ from a standard joint audio‑video generator that also has an audio stream?

A standard generator treats the audio stream as a single token sequence representing the mixed soundtrack; SepGen inserts additional stem tokens, concatenates them with the mix, and restricts attention so only the stems receive caption information, while the mix and video remain unchanged.

**Figure 2.** **SepGen: a unified representation for multi-stem generation and separation.** (a) Generation: scene text and two source captions yield the video, audio-mix, and per-source waveforms (b) Separation: from an observed video and audio-mix, the same model extracts both captioned stems

Training Procedure

How the model learns to both separate and generate audio stems via a two‑stage training schedule.

The model must handle two opposing situations: (1) clean audio‑mixes for source separation, and (2) noisy mixes when generating new audio. A naïve single‑stage training would either over‑fit to clean mixes or struggle with noisy generation.

First teach the model to split a clean mix into its captioned sources, then continue training while gradually corrupting the mix so the same model can also generate from noise.

How does this two‑stage schedule differ from a standard diffusion‑based training that always adds noise to every token?

Standard diffusion treats all tokens uniformly, corrupting them at every step. Here only the stems receive the flow‑matching loss, while the audio‑mix token is sometimes left clean (Stage 1) and sometimes noisy but never noisier than the stems (Stage 2). This asymmetric noise schedule lets the model learn a clean‑mix separation skill first, then reuse it for generation without sacrificing separation quality.

Compute $\sigma_{m} = (1 - 0)\,\min(0.6, 0.4) = 0.4$ — the mix noise is capped to the stem noise.

Both stems and the mix are corrupted with the same noise level $0.4$, so the model sees a consistent noisy context.

If instead $b=1$, then $\sigma_{m}=0$ and the mix remains perfectly clean while the stems stay noisy, reproducing the Stage 1 separation setting.

The cap guarantees that during generation the model never has to attend to a mix that is noisier than the stems it must predict, preserving stability while still exposing the network to realistic noisy conditions.

Inference and Stem Routing

Describes how stems are routed and denoised across a decreasing noise schedule to enable both separation and generation.

When multiple stems share a single audio‑mix token, naïve attention lets information flow back into the mix, corrupting the pretrained base model. The section therefore introduces a routing trick that lets stems see the mix while keeping the mix isolated.

Each stem can attend to the audio‑mix, but the mix never receives any signal from the stems, preserving the original model’s behavior.

How does Stem Routing differ from a simple attention mask that only blocks stem‑to‑mix attention?

Standard masks usually prevent stems from seeing the mix, but Stem Routing does the opposite: stems can read the mix (so they know the global context) while the mix is forced to ignore stems, and the LoRA gate further guarantees that the mix’s projection stays frozen.

Query $s_1$ attends to $s_1$, $s_2$, and $m$ (all three entries in row 1 are 1).

Query $s_2$ attends to $s_1$, $s_2$, and $m$ (row 2 all 1).

Query $m$ attends only to $m$ (row 3 has zeros for $s_1$, $s_2$ and a 1 for $m$).

LoRA gate $g_j$ is set to 0 for the two mix tokens, so their updates use the frozen projection $W_0$; $g_j=1$ for the four stem tokens, allowing them to receive adapted updates.

The mask guarantees a one‑way flow of information: stems can incorporate the global mix context, but the mix never incorporates stem‑specific details, preserving the original audio‑mix quality.

During sampling we gradually reduce a noise level $\sigma$, updating latents with predicted velocities; depending on the mode, either only stems move (separation) or all latents move together (generation).

Why does the schedule switch to the estimated clean mix $\hat{z}_{0,m}$ only after the noise drops below $\sigma_g$?

At high noise the estimate $\hat{z}_{0,m}$ is dominated by the noisy velocity term and becomes unstable; keeping stems attached to the actual noisy mix preserves a reliable reference. Once the noise is low enough ($\sigma$ ≤ 0.97) the estimate is accurate enough to guide the stems without corrupting the final mix.

Step 0 → 1: $\Delta\sigma = -0.5$. Stem latents update: $z_{s1}^{(1)} = z_{s1}^{(0)} + (-0.5)\times0.2 = z_{s1}^{(0)} - 0.1$, $z_{s2}^{(1)} = z_{s2}^{(0)} - 0.05$. Mix token stays unchanged because $\sigma_0 > \sigma_g$.

Step 1 → 2: $\Delta\sigma = -0.5$. Since $\sigma_1=0.5 \le \sigma_g$, we compute $\hat{z}_{0,m}=z_{m}^{(1)} - 0.5\times0.15 = z_{m}^{(1)} - 0.075$ and let stems attend to this estimate. Stem updates: $z_{s1}^{(2)} = z_{s1}^{(1)} - 0.1$, $z_{s2}^{(2)} = z_{s2}^{(1)} - 0.05$. Mix token also updates: $z_{m}^{(2)} = z_{m}^{(1)} - 0.075$.

Final latents: stems have been denoised twice, the mix has been updated only in the last step using the clean estimate.

The schedule lets stems benefit from the global mix context early on, then gradually rely on a cleaner estimate as noise vanishes, ensuring coherent separation without contaminating the final mix.

**Figure 3.** Inference schedules in $\sigma$-space. Left to right: (a) Separation keeps the audio-mix $m$ clean. (b) Shared-noise generation, the naive approach, denoises the stems and the audio-mix at one shared noise level. (c) SepGen’s stems attend to noisy $m$ above $\sigma_g$, then its clean estimate $\hat{z}_{0,m}$ below it.

**Figure 4.** **Stem-to-video attention** in a generated frame. Each stem attends to the source named in its caption: bongo drum (left) and acoustic guitar (right); warmer is higher.

Experimental Setup

We evaluate SepGen on large generation and separation benchmarks, showing it beats cascade baselines.

SepGen joint‑run stems outperform all cascade baselines on every generation metric.

Table 1 shows SepGen leads each baseline on all metrics, with significance except for CLAP versus FlowSep and AudioSep.

We curate a diverse set of benchmarks: 194 generated two‑source scenes for generation, plus rendered and real‑world mixes for separation (149 Veo, 64 LTX‑2.3, 172 SAM‑Audio‑bench recordings). All methods, including both SepGen checkpoints, receive the same audio‑mix and extract the captioned sources.

These baselines first run a language‑queried separator on the audio‑mix, then feed the extracted stems to a second model, treating generation and separation as two independent steps.

**Table 1.** **Generation benchmark.** Comparison of SepGen with cascade baselines, each of which separates the same generated audio-mix given the same captions (sounds $n=134$, speech $n=60$). Both SepGen rows use the generation checkpoint.

The benchmarks span 194 generated scenes and 385 rendered or real mixes, providing a comprehensive testbed for addressable audio generation and separation.

Separation Results

SepGen beats every cascade baseline on both Veo and SAM‑Audio separation benchmarks.

SepGen outperforms all cascade baselines on the Veo and SAM‑Audio benchmarks, achieving the highest Judge scores.

Table 2 shows SepGen (generation checkpoint) reaches a Judge of 0.96, and Table 3 shows 0.95, both above the next best method.

We evaluate separation quality with Scale‑Invariant Signal‑to‑Distortion Ratio (SI‑SDR). The quoted lines in speech captions—text fragments enclosed in quotation marks—prevent cascade query encoders from exploiting the speech content, so each speech cell reports results both with and without those lines. Removing the lines silences 4–5 of the 90 stems, which explains why speaker‑cosine improves while word‑error rises.

**Figure b.** SepGen emits per-source waveforms that can be localized, lifted to 4D, and rerendered per-stem localization

Ablations and User Study

Ablation study isolates each component’s impact on separation and generation performance.

We evaluate each design choice on the LTX‑2.3 benchmark by disabling it while keeping all other components identical.

Removing the joint pass collapses the two stems into near‑copies, dropping the Judge score from 4.07 to 0.19.

Jointness ablation row in Table 4 shows a 0.19 Judge value versus the baseline 4.07.

Retraining with misaligned positions reduces Judge to 0.19 and raises WER to 0.66, a +0.47 increase in error.

Aligned Positions Retrained row shows Judge 0.19 and WER 0.66 compared with baseline 4.07 and 0.19.

All video‑ablation variants and the protected audio‑mix condition leave the primary metrics unchanged.

Video Blank, Unrelated, None rows and Protected audio‑mix row all retain Judge ≈ 4.07 and WER ≈ 0.01.

Generation Results

SepGen’s joint generation outperforms baselines across all metrics and scene categories.

SepGen (joint run) attains the highest Judge score among all methods.

Table S4 reports a Judge of 3.82 ± 0.01 for SepGen, exceeding the next best baseline (Audio‑mix twice) at 3.35 ± 0.19.

Seed spreads are larger for cascade baselines because their seed mean follows the regenerated audio‑mixes they receive (e.g., FlowSep reads 3.49, 3.43, and 3.14 Judge on the three seeds), whereas stems generated jointly with the audio‑mix do not inherit this difficulty. Across scene categories, SepGen’s Judge and J‑swap advantages persist, with the widest gaps in the music category where two instruments play throughout the clip and cascades cannot exploit silent spans. The joint‑run generation also outperforms independent caption passes, confirming that shared audio‑mix conditioning is essential.

Extended Separation Results

SepGen outperforms all cascade baselines across every benchmark column.

SepGen separation checkpoint outperforms every cascade baseline on every column of the Veo, LTX‑2.3, and SAM‑Audio benchmarks.

Tables S8, S9, S10 show SepGen leading all baselines across all metrics; only AudioSep ties on CLAP identity in Veo.

The tables also include the SAM Audio best‑of‑eight protocol applied to both SepGen and the baselines, ensuring that the reported superiority holds under a stronger inference regime.

Waveform Evaluation

Evaluates SepGen’s reference‑stem quality and explains why SI‑SDR is unsuitable for generative separators.

Reference‑stem evaluation holds out recordings that contain true stems, allowing a direct comparison between generated stems and their ground‑truth counterparts.

We test on 129 two‑source scenes, 60 URMP duets (including 26 same‑instrument pairs), 57 CelebV‑HQ dialogues, and 12 validation composites, totaling 258 stems; URMP and CelebV‑HQ are absent from the training lists of AudioSep and SAM Audio.

Each output stem is aligned to its reference via level matching and a ±50 ms lag search, then scored with 20 ms envelope correlation and 64‑band log‑spectral distance (LSD), the two measures that survive the audio VAE decoder.

SepGen outperforms all baselines on both metrics (Table S13): envelope r = 0.82 and LSD = 6.3 dB, beating the strongest cascade baseline (AudioSep) which attains 0.76 r and 11.0 dB LSD.

On the 57 dialogue clips, SepGen’s transcriptions achieve a word‑error rate of 0.05, comparable to the true stems, whereas the best cascade baseline records 0.29 and the raw audio‑mix 1.03.

Same‑instrument duets share captions on both stems; SepGen yields 11.7 dB LSD versus 6.3 dB on distinct‑instrument duets, and permutation‑invariant LSD (11.4 dB) remains close to the caption‑fixed value, indicating that the caption cannot resolve the split.

SI‑SDR, which compares a stem sample‑by‑sample, becomes negative for all generative separators after a single pass through the audio VAE decoder (Table S14), because the decoder reconstructs energy and spectrum while altering phase.

Restoring only the original short‑time phase makes SI‑SDR positive again, whereas restoring only magnitudes leaves SI‑SDR largely unchanged, confirming that phase loss dominates the degradation.

**Table S3.** User-study preference shares with 95% bootstrap intervals, resampling listeners and clips. The Sources column reports the number of sources on which SepGen won, tied, or lost by majority vote.

**Table S4.** Generation benchmark with seed standard deviations, degenerate controls, and the paired-test significance of SepGen (joint run) against each baseline.

Training Details

This appendix details the training regimen, inference settings, and the cross‑stem attention guidance used in SepGen.

Training draws from four families: CelebV‑HQ talking‑head pairs (1,658 segments), URMP two‑instrument scenes (1,437), generated two‑source clips (749), and cross‑video composites (999) from VGGSound and MUSIC‑21. Each mix is the sum of two known sources, providing ground‑truth stems for every segment.

Captions are sampled uniformly from three registers—semantic, positional, and short. Positional captions define a source box, and a soft additive bias forces each stem’s audio queries to attend only to video tokens inside that box; the bias is zeroed on the audio‑mix channel and removed at inference.

Training proceeds in two stages. First, 12 k separation updates train the model to recover stems from a clean audio‑mix ($\sigma\!=\!0$). Then, 3 k generation updates expose the model to noisy mixes (30 % clean, the rest at a noise level clamped to the stems) to learn joint video‑and‑stem generation.

LoRA adapters (rank = 128, scale = 128) are applied to audio self‑attention, the query and output projections of text cross‑attention, video‑to‑audio attention, and the audio feed‑forward block, yielding 327 M trainable parameters while the 22 B LTX‑2.5 backbone remains frozen and int8‑quantized.

At inference, video frames are encoded at 768 × 512 and each clip retains its original length (e.g., 113 frames at 25 fps for the generation benchmark). Guidance follows the LTX‑2 pipeline: classifier‑free guidance (CFG) scales of 3.0 for video and 7.0 for audio, cross‑modal guidance 3.0 on video and audio‑mix tracks, and stem rescale 0.7. Separation uses 30 Euler steps with CFG rescale 0.7; generation uses a 15‑step sampler with video/audio‑mix rescales 0.45/1.0.

Cross‑Stem Attention Guidance (CSAG) mitigates stem leakage by combining a positive branch $Z^{+}_{i}= \text{Attn}(h_{i},q_{i})$ with a negative branch $Z^{-}_{i}= \text{Attn}(h_{i},q_{j})$ using $G=\lambda Z^{+}-(\lambda-1)Z^{-}$, normalizing $\bar{G}=G/\min(r,\tau)$, and blending $Z^{\text{CSAG}}=\alpha\bar{G}+(1-\alpha)Z^{+}$ with $\lambda\!=\!2$, $\tau\!=\!2.5$, $\alpha\!=\!0.5$.

Table S1 reports an ablation over $\lambda$: increasing $\lambda$ to 3 or 5 degrades the Judge metric while leaving CLAP swap margin unchanged; the configuration $\lambda\!=\!2$ (the default) yields the best overall separation performance.

Adapter contribution analysis shows that disabling all LoRA updates leaves the audio‑mix and video outputs bit‑identical across transformer layers, while stem outputs differ, confirming that adapters affect only the stem pathways.

Two checkpoints are saved: the separation checkpoint (after the first stage) and the generation checkpoint (after the second stage). The separation checkpoint is used for pure separation benchmarks, whereas the generation checkpoint powers joint video‑and‑stem generation and separation experiments.

Benchmark Details

Evaluation datasets, metrics, and statistical protocols for SepGen are detailed.

The evaluation uses four benchmarks: a synthetic generation set, two rendered sets (LTX‑2.3 and Veo), and a real‑world SAM‑Audio‑bench.

The generation benchmark contains 194 two‑source scene prompts (speech, foley, music, speech‑with‑background) with no prompt overlap with training data; each prompt yields a 4.5 s clip, and the audio‑mix is transcribed with a word error of 0.05 against the scripted lines.

The LTX‑2.3 benchmark provides 64 two‑source clips (5 s each) rendered by a model of the same family as our backbone, while the Veo benchmark supplies 149 curated clips (average 5.7 s) after duplicate removal and reviewer screening.

SAM‑Audio‑bench consists of 172 in‑the‑wild recordings from multiple sources, split into instrument‑wild, sound‑effects, and speaker subsets, each evaluated with benchmark‑provided captions adapted to our style.

Evaluation protocol defines identity scores $J(s,c)$ (SAM Audio Judge) and $C(s,c)$ (CLAP cosine) for stem $s$ and description $c$, with swap margins computed as the difference between matching and mismatching pairs, averaged over both stems.

Results are reported as mean ± standard deviation over seeds (three seeds for separation tables, five for SAM‑Audio‑bench); statistical significance uses Holm‑corrected paired Wilcoxon tests, and word‑error variability is accounted for by fixing the recognizer pass.

Metric controls verify that decoder choice, sampling rate, and speaker embedding do not drive the reported gaps: re‑encoding baselines through our audio VAE changes scores by at most 27 % of the Judge margin, and band‑limiting to 16 kHz preserves the majority of the performance gap.

Table S2 reports an audio VAE requantization control on the sounds lane (n = 134, seed 0), showing the change $\Delta$ between requantized and published scores.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers