The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation

Yichen Liu, Quanwei Zhang, Haozhe Wang, Donghao Zhou, Xiaojie Li, Yang Shi, Jiaming Liu, Ruihua Huang, Yingtian Zou, Daquan Zhou

Temporal Context Routing aligns script-specified shot and dialogue timing with video-audio generation via additive cross-attention bias.

How can we force generative models to respect precise, script-defined timing for shot transitions and dialogue?

Joint audio-video generators align modalities with each other, but they fail to follow the specific timing of shots and dialogue defined in a structured script. This causes narrative transitions and speech to occur at the wrong moments, even when the video and audio remain synchronized. Temporal Context Routing (TCR) maps script timing onto the shared temporal axis and injects it as a duration-normalized bias into the cross-attention logits. This allows each prompt to independently guide both modalities without modifying the underlying model architecture. On a test set of 200 scripts, TCR reduces shot boundary error by 96% and increases dialogue timing accuracy from 28.3% to 84.1% compared to the strongest baseline.

Paper Primer

Script-driven generation requires precise temporal control, yet current models treat script timing as implicit text rather than explicit control signals. TCR solves this by treating each prompt's time interval as a routing score: it is like a traffic controller that reads the schedule on each envelope and directs the flow of information to the exact temporal window where it belongs.

TCR significantly improves temporal adherence to script timelines.

Shot Boundary Mean Absolute Error (MAE) dropped from 1.11s to 0.042s, and Dialogue Accuracy at 0.5s improved from 28.3% to 84.1%.

Temporal accuracy gains do not degrade visual or audio quality.

Imaging Quality (IQ) and Aesthetic Quality (AES) scores remained within 2.5% of baseline performance across all ablations.

Why is this approach better than simply including timing information in the text prompt?

Including timing in text forces the model to learn temporal alignment implicitly, which often fails to synchronize with the actual generation timeline. TCR provides an explicit, additive bias to cross-attention logits, ensuring the model is mathematically steered toward the requested temporal intervals.

Does TCR require retraining the entire audio-video generation model?

No. TCR is applied during LoRA adaptation, meaning it optimizes only the adapters while keeping the backbone model frozen, preserving the original visual and audio-visual synchronization capabilities.

The Temporal Control Problem

Generative models struggle to align video‑audio output with scripted timing.

Current joint audio‑video generators keep video and audio synchronized, yet they treat script timing as ordinary text tokens. As a result, shot transitions and spoken dialogue often drift from the intended script timeline, breaking narrative coherence.

Instead of a single prompt, the input is a structured script that lists each shot and dialogue line together with its desired start‑and‑end times.

Three concrete challenges arise. First, shot and dialogue prompts can overlap, so they must be controllable independently. Second, existing annotations provide only coarse timestamps, yet the model needs fine‑grained supervision. Third, any temporal control mechanism must preserve the high visual fidelity and audio‑visual sync that current generators already achieve.

To supply the required supervision we build a coarse‑to‑fine pipeline: a large‑language model first produces a rough script with approximate timestamps, then visual cut detection and word‑level speech alignment refine those timestamps onto a 0.1 s grid.

The core gap is that script intent is not enforced on the generated timeline, leading to mismatched cuts and dialogue.

Prior Approaches to Multimodal Generation

We survey prior audio‑visual generation and temporal‑control methods, highlighting how our approach differs.

Joint audio‑visual generators build on diffusion models, diffusion transformers, and flow‑matching techniques, extending them to synchronize video and audio streams. Approaches such as MM‑Diffusion couple modality‑specific denoisers via cross‑modal attention, while AV‑DiT shares a lightly adapted DiT backbone across modalities. More recent works (JavisDiT, Harmony, UniAVGen, VideoPoet, LTX‑2) explore hierarchical priors, unified optimization, and asymmetric interactions, yet they do not enforce script‑driven timing.

Structured and local prompting methods move beyond monolithic prompts by associating latent segments with sub‑captions or shot‑specific tokens. Techniques like Presto, ShotAdapter, MultiShotMaster, KeyVID, and Audio‑Sync Video Generation provide shot‑aware attention masks or rotary encodings, while MTSS factorizes descriptions into Reference, Shot, Event, and Global streams with explicit temporal links. Our approach extends this line by routing each prompt’s timing through both video and audio conditioning pathways, preserving independent control over shots and dialogues.

Temporal control methods differ in where timing enters the model. Access‑based approaches use masks to expose prompt tokens only within designated temporal windows, whereas representation‑based methods encode temporal structure directly into query–key interactions via RoPE variants. Cross‑Modal Context Learning combines aligned RoPE with dynamic routing, and score‑based steering steers attention, latent states, or logits toward target regions; Gaussian logit priors have also been used to model attention locality. In contrast, Temporal Context Routing (TCR) computes prompt‑specific, duration‑normalized routing scores and applies them to both video‑text and audio‑text cross‑attention, enabling overlapping shot and dialogue prompts to control both modalities independently.

Temporal Context Routing

Core mechanism that injects script timing into attention via Temporal Context Routing.

Generative models often miss script‑defined timing because they treat temporal cues as ordinary text tokens. To enforce precise alignment we route the script’s timing directly into the attention logits, letting each prompt guide the video and audio streams at its own moments.

TCR converts a prompt’s timing interval into a smooth routing profile that biases the attention scores toward the right temporal positions without altering the token embeddings.

Compute the squared distance: $(t^{v} - c)^2 = (2.5 - 3)^2 = 0.25$.

Plug into the routing formula: $B^{v} = -5 \times \frac{0.25}{2 \times 1^2} = -5 \times 0.125 = -0.625$.

For a token without an interval, $B^{v}=0$ by definition.

When the attention logits are exponentiated, the token with $B^{v}=0$ receives a higher weight than the one with $B^{v}=-0.625$, biasing the model toward the prompt’s temporal center.

The routing score is zero at the interval center (no penalty) and becomes increasingly negative toward the edges, so attention naturally concentrates on the intended time window without any learned parameters.

How does Temporal Context Routing differ from traditional positional encodings?

Positional encodings add a fixed bias to token embeddings before any attention is computed, applying the same pattern to every token. TCR, by contrast, injects a prompt‑specific bias directly into the attention logits based on the actual script timing, allowing each prompt to have its own temporal profile and to affect video and audio streams independently.

**Figure 2.** Overview of Temporal Context Routing (TCR). (a) A structured script organizes reference, Shot, and dialogue prompts together with their timing. (b) The shared text encoding conditions the video and audio towers, while prompt timing bypasses the text encoder and is supplied separately as $B(t)$ to both text cross-attention modules. (c) Prompt temporal routing converts each prompt's timing into a duration-normalized routing profile that peaks at its center and reaches $-\beta/2$ at its boundaries. The resulting profiles route Shot (blue) and dialogue (green) guidance along the video-audio timeline.

Empirical Performance and Data Pipeline

We evaluate temporal precision and overall quality of TCR against strong baselines.

The experimental suite measures how well TCR aligns generated video and audio to scripted shot and dialogue timings, and compares it to four strong baselines.

TCR reduces Shot Boundary MAE by 96 % compared to the strongest baseline.

MAE drops from 1.11 s to $0.042$ s on the 200‑script test set.

**Figure 6.** **Shot-timing performance across training checkpoints.** (a) Shot Boundary MAE and (b) Shot IoU on the same 200 test scripts. Shaded regions denote 95% prompt-bootstrap intervals, and the dashed line marks the 7k checkpoint reported in the main tables. TCR performs best on both metrics at every checkpoint.

**Figure 1.** Overview of the data processing pipeline. (a) Clip construction uses audio silence detection to segment source episodes into clips. (b) Structured annotation uses a multimodal model to extract references, shots, events, and global context from clips. (c) Temporal refinement aligns proposed timing with visual scene cuts and speech boundaries to produce a refined script.

**Figure 4.** Qualitative comparison on a four-shot script. The requested Shot and dialogue timing is shown at the top. (a) Each baseline misses or delays at least one of the three requested shot transitions; Wan2.2 generates no audio. (b) TCR produces all three cuts near their requested times and follows the specified four-shot structure. Speech-energy traces show dialogue activity on the same timeline.

**Figure 5.** **Pairwise human evaluation.** Tie-inclusive vote shares for TCR against LTX-2.3 (left) and JoyAI-Echo (right) across five dimensions, based on 16 randomly sampled cases and 28 participants.

**Table 1.** End-to-end comparison on 200 test scripts. Table 2 reports controlled comparisons using the same backbone and training setup.

Ablation Studies and Temporal Operators

We isolate each component’s impact on temporal precision and overall quality.

The central premise is that script‑defined timing must be routed directly into attention; $TCR$ does this, enabling precise alignment of shots and dialogue.

It injects a smooth, Gaussian‑shaped positional bias into attention windows, encouraging the model to focus around a central time interval.

How does Gaussian Interval RoPE differ from standard sinusoidal RoPE?

Standard RoPE rotates query and key vectors uniformly with time, preserving relative positions. Gaussian Interval RoPE instead multiplies attention scores by a Gaussian window centered on a target time, which biases the model toward a specific interval but loses the exact periodic structure.

It applies a binary mask that completely blocks attention outside a prescribed time window, forcing the model to attend only within that interval.

Why does the hard mask still generate extra cuts despite its binary enforcement?

Because the mask is defined on discretized time steps; if the scripted cut falls between two steps, the nearest step becomes a hard boundary, creating an unintended cut.

**Figure 7.** Qualitative comparison of temporal operators. Requested shot and dialogue timing is shown above each example. (a) Gaussian Interval RoPE and the hard mask produce two extra cuts, whereas TCR matches all four requested shots. (b) Gaussian Interval RoPE places all three cuts early, while the hard mask and TCR align them closely with the requested boundaries. Triangles mark produced cuts; the audio lanes show detected speech and energy.

$TCR$ outperforms the matched temporal operators on every shot‑timing metric.

Shot Boundary MAE = 0.042 s (60 % lower than both baselines), Shot IoU = 0.957, Shot Count Accuracy = 93.0 %, Dialogue Acc@0.5 s = 84.1 % (hard mask 83.7 %).

Coarse temporal supervision dramatically harms control.

Shot Boundary MAE rises to 0.375 s and Dialogue Acc@0.5 s drops to 37.6 % when using coarse annotations.

Separating audio and video prompts reduces joint temporal coordination.

Dialogue Acc@0.5 s, WER, Sync‑C, and offset accuracy all decrease relative to the full‑prompt configuration.

Implementation Details

Implementation and script compilation details for reproducibility.

We compute video query times as the midpoints of temporal cells and audio query times from patch positions, converting both to seconds before applying the temporal operator. The epsilon tolerance is set to $\epsilon = 10^{-4}$ s (Equation 2) and global prompts span $[0,\,T]$, with no routing score assigned to sentinel tokens.

After tokenization the timing map is repacked to match the text connector, padded to shape $B \times N \times 2$, while the padding mask has shape $B \times 1 \times 1 \times N$ and the routing score $B_m$ has shape $B \times 1 \times Q_m \times N$, shared across all attention heads. Training uses LoRA rank 128, learning rate $10^{-4}$, 500 warm‑up steps, cosine decay, and a first‑frame conditioning probability of 0.5; inference runs 30 sampling steps with guidance 4.0, spatiotemporal guidance 1.0 at block 29, seed 42, resolution $704 \times 1280$, and 24 fps.

**Figure 8.** Human-evaluation interface. Videos are presented anonymously as A and B with balanced ordering. Raters view the target shot and dialogue timeline and make independent A/Tie/B choices for shot timing, dialogue timing, script fidelity, audio-visual synchronization, and overall execution.

The structured script record supplies each model with shot IDs, time ranges, visual descriptions, and dialogue events; the compiler strips numeric `time_range` fields before text encoding and feeds them separately through the timing map, ensuring all variants receive identical prompt content and clip duration.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers