The Attention Triangle in Audio-Video Models

Sagi Polaczek, Noa Kraicer, Gal Metzer, Zhuo Ning, Ali Mahdavi-Amiri, Daniel Cohen-Or, Raja Giryes

Audio-video diffusion models suffer from semantic leakage; steering the full "attention triangle" restores correct source attribution.

How can we prevent audio-video diffusion models from incorrectly leaking semantic information (like lip-syncing or appearance) between unrelated sound sources and visual entities?

Audio-video diffusion models often misattribute sounds to the wrong visual entity, such as assigning a parrot's speech to a pirate because the model's learned priors favor human-like speakers. This leakage occurs because cross-modal attention acts as a soft, global mixer that allows semantic information to bleed between modalities along unintended pathways. The authors model these interactions as an "attention triangle" connecting text, audio, and video streams. They introduce a training-free steering method that applies additive logit biases to all three edges of this triangle—text-to-video, text-to-audio, and audio-to-video—to suppress conflicting associations and reinforce intended ones. This joint intervention significantly improves source attribution and reduces visual leakage compared to baseline models, while preserving generation quality across diverse open-domain scenarios.

Paper Primer

The core mechanism hinges on the "attention triangle," a coupled system where semantic information flows bidirectionally between text, audio, and video. The authors' steering algorithm acts as a traffic controller: it identifies intended and conflicting token regions, then applies pre-softmax logit biases to force attention toward prompt-specified pairings and away from statistically canonical but incorrect ones.

Joint steering of all three triangle edges is necessary to resolve both appearance leakage and source misattribution.

Ablation studies show that steering only text edges corrects appearance but leaves attribution broken, while steering only the audio-video edge corrects attribution but introduces appearance artifacts. Ours-Full achieves a source-attribution score of 0.1349 compared to 0.1216 for the native baseline.

Why does the model struggle to bind sound to the correct visual source in the first place?

The model's training distribution encodes strong priors—such as speech correlating with human-shaped patches—that override prompt-specified bindings. Additionally, the dimensional mismatch between 1D audio temporal embeddings and 3D spatiotemporal video patches leaves source localization under-constrained.

How does this approach differ from simply restricting attention to specific regions?

Unlike methods that only bound text-to-video attention, this framework addresses the indirect, audio-mediated pathways. By steering the audio-video edge alongside the text edges, it prevents the "second-order" leakage where audio tokens act as a bridge to misroute visual attention.

The Problem of Semantic Leakage

We expose semantic leakage in audio‑video diffusion and propose inference‑time steering to curb it.

Audio‑video diffusion models rely on cross‑modal attention to bind text, sound, and visual streams. This mechanism, while powerful, can inadvertently mix semantics across modalities, leading to systematic semantic leakage where audio cues drive unrelated visual entities.

When audio signals unintentionally steer visual generation, the model produces visual elements that incorrectly reflect the sound source.

We probe this phenomenon by examining the attention triangle—the three cross‑attention edges among text, audio, and video. Our analysis shows the audio‑video edge is bidirectional and, when biased, becomes the dominant pathway for leakage, especially under prompts that clash with learned priors.

**Figure.** A weathered pirate with a black tricorn hat and thick grey beard stands on a wooden dock at sunset. A bright green-and-red macaw parrot perches on his right shoulder. The parrot is the only character that speaks: ‘We are lost at sea.’ The pirate does not move or speak.

Extracting attention‑derived signals lets us diagnose and deliberately induce leakage, providing a controlled lens on cross‑modal routing. Guided by these signals we devise inference‑time steering interventions that suppress the offending audio‑video paths while preserving overall generation quality.

Audio‑video models inadvertently link unrelated sound and visual sources, but inference‑time steering of the attention triangle can restore correct cross‑modal alignment.

Attention in Diffusion Models

Background on cross‑modal attention, its role, and known leakage issues in audio‑video diffusion.

Cross‑attention couples two token sequences—typically a noisy latent and a conditioning embedding—allowing one modality to selectively attend to another. This mechanism underpins most modern diffusion models that generate images, video, or audio‑video jointly.

Cross‑modal attention is the process by which tokens from one modality (e.g., audio) query and aggregate information from tokens of another modality (e.g., video) via learned key‑value pairs.

Leakage arises because the soft, global mixing of attention does not enforce exclusivity; entities that should remain separate can inadvertently share attention mass, producing blended visual or auditory artifacts.

Early works on image diffusion treated cross‑attention as a steerable signal for spatial layout and attribute binding, while later video diffusion models extended the same mechanism across time. Recent audio‑video diffusion systems inherit this pipeline, adding a third modality edge that further compounds leakage.

Numerous inference‑time methods have been proposed to edit cross‑attention maps—boosting under‑attended tokens, aligning maps with syntactic structure, or enforcing user‑specified layouts—demonstrating that the attention matrix is a practical control surface without retraining.

The Attention Triangle Framework

We introduce the Attention Triangle to isolate and suppress cross‑modal audio‑video leakage.

Audio‑video diffusion models inherit a systematic semantic leakage: audio signals unintentionally steer visual generation toward unrelated entities. The core difficulty lies in the cross‑modal attention edges that silently reroute textual attributes.

The three modalities—text, audio, and video—form a triangle of cross‑attention edges, each edge equipped with a bias that can reinforce the intended pairing (green) or suppress a conflicting one (red).

How does the Attention Triangle differ from a simple pairwise cross‑attention graph?

A pairwise graph only adds a bias on a single edge; the Triangle adds coordinated biases on all three edges and explicitly separates intended (reinforcing) and conflicting (suppressing) components, enabling the model to steer attention jointly across the three modalities rather than treating each edge in isolation.

Compute the uniform baseline $\mathbf{1}\,N_V \times N_A$ which yields a $2\times2$ matrix of ones.

Subtract $G_{VA}$: $\begin{bmatrix}1 & 1\\ 1 & 1\end{bmatrix} - \begin{bmatrix}0.2 & 0.1\\ 0.1 & 0.2\end{bmatrix}= \begin{bmatrix}0.8 & 0.9\\ 0.9 & 0.8\end{bmatrix}$.

Apply the negative scalar: $B_{A\to V}= -1 \times \begin{bmatrix}0.8 & 0.9\\ 0.9 & 0.8\end{bmatrix}= \begin{bmatrix}-0.8 & -0.9\\ -0.9 & -0.8\end{bmatrix}$.

Intended mask $M_{\text{int}}$ selects the diagonal entries (strengthening self‑aligned pairs), while conflicting mask $M_{\text{conf}}$ selects the off‑diagonal entries (weakening cross‑pairs).

After adding the bias to the raw attention scores, the diagonal (intended) scores increase, and the off‑diagonal (conflicting) scores decrease, steering the video patches toward the audio tokens they are meant to accompany.

The bias transforms a uniform attention landscape into one that explicitly favors prompt‑designated pairings and penalizes unintended cross‑modal bindings.

**Fig. 2. The Attention Triangle.** Edge-specific biases promote attention to the intended speaker (green) and suppress leakage to the competing entity (red). *Intended* denotes amplified attention for prompt-designated pairings: source↔source, sound↔sound, or source↔sound. *Conflicting* denotes attenuated pairings between these signals and the competing source. The agreement matrix $G_{VA}$ is defined in Eq. 2; the intended and conflicting text-edge masks $M_{VT}$ and $M_{AT}$ are defined in Eqs. 4 and 5, respectively.

**Fig. 3.** Cross-modal leakage in T2V vs T2AV generation. Left: T2V. Middle: T2AV. Right: prompt. Adding audio induces both appearance and speaking leakage: in the puppet example, both the performer and puppet move their lips; in the dog and cat example, the dog shifts toward a cat-like appearance and both animals appear to speak.

Inference-Time Steering Algorithm

We present a training‑free steering algorithm that reweights attention edges to eliminate semantic leakage.

The core of our method is a training‑free steering algorithm that adjusts three cross‑attention edges of the attention triangle to suppress semantic leakage.

We compute masks for the intended source, the sound/action, and any competing source, then inject additive logit biases on the three attention edges so that intended pairs are reinforced and mismatched pairs are penalized.

How does this steering differ from simply zero‑masking the conflicting cells?

Zero‑masking would hard‑remove those logits, breaking gradient flow and discarding useful context. Our additive bias softly penalizes mismatched pairs while still allowing the model to attend if the evidence is strong, preserving training‑time dynamics.

Compute $G_{VA}=m_V^{src}(m_A^{snd})^\top + (1-m_V^{src})(1-m_A^{snd})^\top$, yielding a $4\times3$ matrix with high values (≈1) on the parrot‑speech pair and low values (≈0) elsewhere.

Form the audio→video bias $B_{Aarrow V}=-\lambda(1-G_{VA})$ with $\lambda$=10, producing large negative logits for mismatched pairs.

Build the text‑video mask $M_{\text{int}}$ that marks the parrot‑“parrot” token pair as intended and $M_{\text{conf}}$ as empty (no competing source).

Apply $B_{Tarrow V}=βM_{\text{int}}-γM_{\text{conf}}$ with $\beta$=0.5, $\gamma$=2.0, which simply adds +0.5 to the intended cell.

During each denoising step, add these biases to the logits; the softmax now concentrates attention on the parrot‑speech pair.

The soft agreement matrix lets the algorithm distinguish “both intended” from “both unintended” pairs, enabling a nuanced suppression that a hard mask cannot achieve.

LTX‑2 is the underlying text‑to‑audio‑video diffusion model on which the steering algorithm is instantiated.

**Fig. 4. Mitigating audio-to-video attention leakage.** Using the same prompt and random seed as in Figure 1, we sum, for each video query, its attention weights over the *speech-active* audio keys identified from the waveform shown at the bottom, then overlay the resulting spatial score on a representative frame. **Left (baseline):** high active-audio scores concentrate across the pirate’s face and chest, the visually canonical but prompt-incorrect human source. **Right (ours):** after attention-triangle steering, high scores concentrate sharply on the parrot, the entity designated by the prompt. The right panel is desaturated outside the high-scoring region to highlight the post-steering localization.

**Fig. 5.** **Audio-mediated video-to-video leakage.** Using the same prompt as in Figure 1, we apply the $V arrow A arrow V$ row rollout in Eq. 1, using $P_{VA}^{(\ell)}$ for the $V arrow A$ rollout hop followed by $P_{AV}^{(\ell+1)}$ for the $A arrow V$ rollout hop. This exposes an indirect attention association between visual regions that is not represented by either individual cross-attention matrix alone. **Left:** seed mass on the parrot, the prompt-designated source. **Middle:** the resulting audio-token distribution after the $v arrow a$ hop is diffuse rather than peaked, reflecting the 1D-to-2D dimensional mismatch (Sec. 3). **Right:** after the $a arrow v$ hop, mass has migrated onto the pirate, exposing the audio-mediated misrouting behind source-attribution leakage.

Empirical Validation

We evaluate steering, showing reduced semantic leakage and higher human preference.

Audio‑video diffusion models often let audio cues leak into visual generation, causing mismatched entities. Our steering method suppresses those cross‑modal attention paths.

5.1 Attention‑Leakage Analysis visualizes how audio‑video cross‑attention routes leakage. First‑order maps show high attention on the pirate in the baseline but on the parrot after steering; second‑order maps expose the audio‑mediated round‑trip that moves mass from the correct source to the wrong entity.

5.2 Manipulation in the Wild applies steering to diverse scenarios (humans, animals, animated characters). We observe four leakage modes—source‑attribution, appearance, voice‑characteristic, and generation suppression—and find that Ours‑Full mitigates all of them.

5.3 Comparisons and Ablations benchmark Ours‑Full against the native LTX‑2 baseline, Bounded Attention, and two partial variants (Ours‑Text, Ours‑AV). Qualitative examples (Fig. 6) illustrate that only Ours‑Full simultaneously fixes appearance and source attribution.

Ours‑Full achieves the highest Qwen source‑attribution score of $0.1349$, surpassing the native LTX‑2 baseline ($0.1216$).

Table 1

Ovi is an external audio‑video model that aligns sound and visual content via a learned cross‑modal embedding.

**Fig. 8. VA-Judger pairwise audio-video preference.** Ours-Full is compared with each reference on the same 400 prompt/seed pairs, each shown in both presentation orders (800 decisions per row). Left: mean pair score after averaging the two orders (win = 1, tie = 0.5, loss = 0), expressed as a percentage; bars are pointwise 95% bootstrap confidence intervals over pairs, and the dashed line marks chance. Right: mean Ours-Full-minus-reference differences on the five judge dimensions scored from 1 to 10 (prompt fidelity, A/V consistency, audio quality, video quality, and completeness), averaged across both orders; darker green indicates a larger gain. The gray column reports order agreement separately; because agreement ranges from 51% to 69%, we interpret only the aggregated, counterbalanced scores. All five pair-score intervals lie above 50%, and all displayed dimension differences are positive.

**Fig. 9. Pairwise user-study preference.** Annotators compare Ours-Full against each comparison method across sound-source attribution, visual leakage, and overall quality. Bars show percentages over all votes: preference for Ours-Full (green), no preference/same (gray), and preference for the comparison method (red). “Same” responses are treated as ties and are excluded from decided-vote significance tests.

**Fig. 10.** User-study interface. (a) Onboarding screen shown before each session, anchoring the annotators’ criteria with the parrot/pirate leading example: a side-by-side baseline-vs-steered pair illustrating the source-attribution and visual-leakage failures the study targets. (b) Per-trial annotation screen presenting two anonymized videos (Video A, Video B) generated from the same prompt and seed. The target sound and the intended source/silent-entity annotations are shown explicitly above the videos; annotators then answer three forced-choice questions (source attribution, visual leakage, and overall quality), each with the options Video A, Same, and Video B.

Extended Analysis and Robustness

Ablation studies reveal that jointly steering all attention edges yields the strongest reduction of semantic leakage.

We run ablations that remove one or more edges of the attention triangle to measure how each component contributes to leakage mitigation.

Removing the audio‑video edge (Ours‑AV) improves attribution over the baseline but still injects appearance leakage, while removing only the text edges (Ours‑Text) sharpens textual binding yet cannot overcome the strong audio‑video prior.

Restricting text‑to‑video attention (Bounded Attention) partially relocates the sound source but fails to consistently suppress leakage, highlighting that steering a single edge is insufficient.

The VA‑Judger pairwise evaluator confirms that Ours‑Full is preferred over every alternative, with all five‑dimensional confidence intervals above 0.5, indicating robust improvement across prompt fidelity, audiovisual consistency, and overall quality.

Human pairwise studies further show that annotators favor Ours‑Full on sound‑source attribution, visual leakage reduction, and overall generation quality, with the largest margins on the first two criteria.

**Fig. 11.** Cross-modal attention routing in LTX-2. Each panel compares the unmodified baseline (upper internal row) with Ours-Full (lower internal row). Columns show the intended-source visual seed distribution $\pi_V$, the audio-bridge attention $\pi_A$, and the returned visual mass $\pi'_V$. Labels T, S, and R denote the percentage of mass assigned to the intended source, competing source, and residual patches. Top: with the golden retriever as the intended source and the orange cat as the competing source, Ours-Full increases intended-source mass from 46.1% to 90.3% and reduces competing-source mass from 39.5% to 6.6%. Bottom: with the polar bear as the intended source and the expedition leader as the competing source, intended-source mass increases from 21.2% to 88.6%, while competing-source mass falls from 27.4% to 2.3%.

**Fig. 12. Representative Ovi comparisons.** For each example, columns compare the Ovi baseline with Ours-Full, and rows show three matched timestamps. Left: the prompt assigns a woman’s voice to the man while the woman remains silent. Right: the prompt assigns horse neighs to the pig while the horse remains silent. The corresponding videos provide the audio comparison; these frames show the visual context and temporal consistency of each output.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers