Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning

Wenzheng Zeng, Siyi Jiao, Chen Gao, Hwee Tou Ng, Mike Zheng Shou

PadCaptioner enables parallelized, event-aware dense video captioning by restructuring causal dependencies via latent global planning.

How can we accelerate dense video captioning by replacing strictly sequential token generation with a parallelized, model-inferred dependency graph?

Dense video captioning models rely on strictly sequential, token-by-token decoding, which creates massive inference latency as video length and event density increase. The authors introduce PadCaptioner, a framework that uses latent global planning to identify event boundaries and then decodes these event-specific subchains in parallel. This approach achieves a 3.8× wall-time speedup over standard autoregressive models while simultaneously improving event grounding and caption quality.

Paper Primer

Standard autoregressive video large language models (Video-LLMs) treat every token as strictly dependent on the previous one, forcing a serial bottleneck. PadCaptioner breaks this chain by generating compact "global event tokens" that act as structural anchors, allowing the model to decode independent event subchains simultaneously.

PadCaptioner delivers significant inference speedups without sacrificing captioning accuracy.

Actual wall-clock time measurements on LongVALE and ChronusAV benchmarks. 3.8× total wall-time decoding speedup and 3× speedup per token.

The framework improves both temporal grounding and semantic caption quality compared to sequential baselines.

Outperforms state-of-the-art models on LongVALE by 6.7% in F1 (grounding) and 6.1% in Sim (captioning). Consistent SOTA performance across multiple omni-modal benchmarks.

Why is parallel decoding difficult for dense video captioning compared to standard image generation?

Unlike image generation, which often uses a fixed spatial layout, dense video captioning requires producing variable-length textual outputs tied to dynamically evolving, temporally localized events, necessitating a structure-aware approach rather than simple spatial parallelization.

How does the model maintain global coherence if it decodes events in parallel?

The model uses an event-factorized attention mechanism where each parallel branch is masked from other local tokens but retains full access to the shared global event tokens, ensuring that cross-event relationships are preserved through compact latent representations.

The Latency Bottleneck in Video Captioning

Why dense video captioning needs a shift from serial to parallel decoding.

Dense video captioning (DVC) requires generating temporally grounded descriptions for many events within a long video. Autoregressive Video‑LLMs produce these captions token‑by‑token, which creates a sequential decoding bottleneck that scales linearly with video length and event density. This bottleneck is like waiting for each person in a line to finish speaking before the next can start, whereas parallel decoding lets everyone speak simultaneously into separate channels.

DVC asks a model to locate multiple events in an untrimmed video and produce a separate caption for each event, rather than a single global description.

In standard autoregressive decoding each token waits for the previous token’s output before it can be predicted, forming a strict chain of dependencies.

**Fig. 1:** Left: Comparison of token dependency modeling and decoding strategies between existing Video-LLM-based captioning models and ours. We achieve lossless parallel autoregressive decoding under a model-inferred restructured dependency graph. Right: Our method surpasses the previous SOTA [8] in both grounded captioning accuracy and decoding efficiency on the LongVALE benchmark [13].

To overcome the bottleneck we introduce PadCaptioner, a parallelized autoregressive framework. The key insight is that tokens belonging to different events exhibit weak cross‑event dependencies, allowing them to be generated simultaneously while preserving strong intra‑event sequential coherence. Two components make this possible: a latent global planning mechanism that learns event‑level structure, and an event‑factorized parallel decoding mechanism that respects that structure during generation.

The shift from serial to parallel decoding is necessary for real-time dense video analysis.

Prior Approaches to Video Captioning

We survey prior dense video captioning approaches and parallel generation techniques, highlighting gaps our method addresses.

Early dense video captioning (DVC) methods followed a proposal‑and‑caption pipeline: first they localized temporal event segments, then described each segment with a sequence decoder.

Subsequent query‑based approaches replaced explicit proposal generation with learnable event queries, enabling joint event localization and caption prediction within a unified detection‑style framework.

More recently, DVC has adopted video large language models (LLMs) as the dominant paradigm, benefitting from powerful pretrained backbones, strong language priors, and cross‑modal reasoning that yield superior performance and broader generalization.

However, the strictly sequential decoding process of these LLMs incurs substantial inference latency, especially because dense video captioning requires generating multiple sentences for many events.

Parallel prediction has been explored in other domains. Diffusion‑based LLMs generate tokens through iterative refinement rather than left‑to‑right decoding, but they need multiple refinement steps and cannot fully exploit KV‑cache acceleration, limiting practical speedup.

Visual autoregressive generation methods achieve parallel decoding by reorganizing dependencies over image tokens, assuming a fixed spatial token layout and a predetermined output cardinality. This structural assumption does not hold for DVC, which must produce variable‑length textual outputs organized around dynamically evolving events.

Conventional query‑based methods such as PDVC support parallel event‑level caption generation, yet their architectures differ markedly from the autoregressive transformer paradigm of modern pretrained multimodal LLMs, preventing them from inheriting rich world knowledge and vision‑language priors.

Latent reasoning augments sequence generation with intermediate latent tokens that enable structured planning before producing final outputs; prior work has used latent visual representations for grounded image/video perception and long‑horizon reasoning.

Our work employs latent planning to reason about event‑level structure, exploiting weak inter‑event dependencies to restructure the conditional dependency graph and achieve structured, lossless parallel decoding.

Temporally grounded multimodal LLMs aim to improve time awareness and event‑grounded perception, but most existing approaches are visual‑only; recent audio‑visual (omni) LLMs add acoustic cues yet still fall short on temporal grounding.

We introduce an omni‑modal LLM that natively supports both visual and audio perception, improving both accuracy and decoding efficiency for dense video captioning while generalizing to other temporally grounded video understanding tasks.

PadCaptioner: Parallelized Autoregressive Decoding

Restructure token dependencies via event‑level planning to enable parallel decoding without loss.

The sequential token chain in standard Omni‑LLM decoders forces every caption token to wait for its predecessor, inflating latency even when many tokens are semantically independent.

A two‑phase pipeline: first a latent planner discovers event‑level anchors, then a parallel decoder generates all local tokens conditioned on those anchors.

How does PadCaptioner differ from a naïve “run the decoder twice” approach?

PadCaptioner inserts a learned event‑level abstraction that conditions each parallel branch, whereas a naïve two‑pass scheme would still generate all tokens sequentially and lack the shared global anchors that guarantee inter‑event coherence.

The decoder’s token graph is factorized: tokens belonging to the same event form a tightly coupled chain, while tokens from different events are conditionally independent given the global event tokens.

Why is it safe to drop cross‑event token dependencies?

Because the global event tokens already encode the high‑level semantics that govern inter‑event relations; conditioning on them preserves the necessary context while removing redundant low‑level causal links.

The model autoregressively emits a compact sequence of event tokens that summarize temporally coherent segments of the video.

How does the model decide how many events $K$ to generate?

Generation stops when the special switch token $S$ is emitted; thus $K$ is implicitly learned from the data and adapts to each video’s temporal structure.

Step 1: Compute similarity $\text{sim}_1^{t}$ for each segment $t$; the binary mask $y_1^{t}$ is 1 for $t\in\{1,2\}$, 0 otherwise.

Step 2: BCE loss pushes $\text{sim}_1^{t}$ high for $t=1,2$ and low for $t=3,4$.

Step 3: After training, at inference the model predicts $G_1$ with peak similarity on segments 1‑2.

Step 4: The same process yields $G_2$ aligned with segments 3‑4.

Step 5: The switch token $S$ terminates planning, yielding the event token sequence $[G_1, G_2, S]$.

The grounding loss guarantees that each event token is anchored to a distinct temporal region, enabling reliable event‑wise decoding later.

Before emitting $G_i$, the planner condenses the multimodal prefix tokens belonging to the event into a single vector using one of three learned pooling schemes.

Why might attention‑guided aggregation be preferable to a learned scoring head?

It reuses the model’s existing attention distribution, avoiding extra parameters and ensuring that the importance scores are already aligned with the query’s semantic focus.

Each event token $G_i$ seeds an independent decoding branch; at every step all branches emit tokens simultaneously, conditioned on shared context.

How does the decoder handle branches of different lengths?

All branches share the same positional index at each step; when a branch emits an EOS token it is padded with additional EOS tokens so the joint distribution remains well‑defined, and the loss encourages those pads to be predicted correctly.

Step 1 (j=1): Sample $L_1^1$ and $L_2^1$ in parallel.

Step 2 (j=2): Sample $L_1^2$ and $L_2^2$ in parallel.

Step 3 (j=3): Event 1 samples $L_1^3$; Event 2 has already emitted EOS, so it outputs a padded EOS token.

Step 4 (j=4): Both branches output EOS padding; decoding terminates.

Because branches are synchronized by step index, the decoder can exploit hardware parallelism while still allowing each event to finish at its natural length.

**Fig. 2:** The overall pipeline of the proposed parallelized autoregressive decoding framework, where we set the model-inferred event count $K = 3$ for illustration.

**Fig. 3:** Comparison of the different decoding strategies and attention mechanisms.

Performance and Efficiency Evaluation

PadCaptioner delivers multi‑fold speedups with accuracy on par with state‑of‑the‑art video captioners.

Video‑LLMs suffer from a sequential decoding bottleneck; PadCaptioner restructures generation into a parallel graph, eliminating token‑dependency delays while preserving output quality.

Video‑SALMONN 2+ is a 3 B‑parameter omni‑modal LLM that jointly processes visual, audio, and textual streams for dense video captioning.

How does Video‑SALMONN 2+ differ from the earlier Video‑SALMONN model?

Video‑SALMONN 2+ expands the visual encoder to Qwen2.5‑VL and adds a dedicated audio‑aligner (Q‑Former), increasing modality coverage while keeping the parameter count at 3 B, unlike the original which relied on a smaller vision encoder and no explicit audio pathway.

PadCaptioner speeds up total decoding by 3.7× and per‑token decoding by 3× compared with the strongest prior omni‑modal LLM.

Table 2 shows a 3.7× reduction in wall‑clock time per video and a 3× reduction in time per generated token on both LongVALE and ChronusAV.

On LongVALE, PadCaptioner surpasses prior SOTA by at least 6.7 % F1 and 6.1 % Sim while remaining lightweight; on ChronusAV it retains comparable scores, confirming robustness across datasets.

**Table 2.** Efficiency analysis on the dense video captioning task on LongVALE and ChronusAV. T/video and T/token denote the average decoding latency per video and per token, respectively, measured by actual wall-clock time (in milliseconds).

**Table 3.** Zero-shot evaluation on YouCook2. C: CIDEr. S: `SODA_c`.

Beyond dense captioning, PadCaptioner excels on LongVALE’s Omni‑TVG and Omni‑SC tasks, delivering competitive BLEU‑4, ROUGE‑L, CIDEr, and METEOR scores (Table 4), evidencing its generality to diverse temporally grounded video‑audio problems.

**Table 5.** ChronusAV. Here we evaluate the six tasks defined in ChronusAV, which emphasize both time awareness and cross-modal reasoning [8]. From Tab. 5, our method also demonstrates competitive performance across all tasks, further validating the robustness and generalization capability of our framework for omni-modal temporally grounded video understanding.

Ablation studies (Table 6) reveal that latent planning yields the largest boost in both grounding (F1) and caption quality (Sim), while pure textual planning offers modest gains.

**Table 8.** Effect of attention mechanism.

**Table 9.** Aggregation strategy ablation.

**Table A2.** Effect of similarity threshold on ChronusAV.

**Table A1.** Effect of similarity threshold on LongVALE.

**Fig. 4.** Qualitative example of a video caption generated by PadCaptioner.

**Fig. A1:** Speedup ratio over serial decoding across subchain count.

**Figure.** A visual comparison of video segmentation results. The top section displays a series of 15 keyframes from a cake decorating video, each labeled with its timestamp in seconds. The bottom section shows a timeline comparison between "Ground truth," "ChronusOmni 7B," and "Ours 3B," where colored bars represent different segments of the video content over a 395-second duration.

PadCaptioner achieves significant latency reduction while maintaining competitive captioning accuracy.

Ablations and Qualitative Analysis

Ablation studies quantify how each design choice affects efficiency and caption quality.

This appendix reports the ablation experiments that answer the core question “does removing this component hurt?” for each of our design choices.

Naïve fine‑tuning inflates generated captions by 2.5× without improving any evaluation metric.

Section B.1 reports a 2.5× increase in average token count while BLEU/F1 scores remain unchanged.

Our grounding and adaptive aggregation pipeline produces substantially shorter captions (549 tokens) than the baseline (685 tokens), a reduction of 136 tokens.

Section B.1 compares average caption lengths across the two systems.

All ablated components retain their gains when the backbone is swapped to an alternative video‑LLM.

Section B.2 shows consistent metric improvements across a second base model.

Increasing the predicted number of event subchains (average 13.2) yields a logarithmic speed‑up over serial decoding, while caption quality stays flat.

Section B.3 plots speed‑up versus subchain count and observes a log‑like trend.

Without FlashAttention‑2, our method consumes 0.3× more GPU memory during training.

Section B.4 reports a memory overhead of roughly 30 % relative to the baseline.

Training time increases between 1.4× and 1.9× on H200 GPUs when FlashAttention‑2 is unavailable.

Section B.4 measures wall‑clock time under the same 12K‑sample budget.

Parallel decoding does not enlarge the KV‑cache; peak GPU memory stays under 30 GB.

Section B.5 monitors memory usage during inference with full‑length captions.

**Table A4.** Training cost on 4 H200 GPUs (bs: batch size).

**Fig. A3:** A qualitative example (YouTube ID: pEkr5UJlrk8) for the V2T (video-to-time) task [2].

**Fig. A4:** A qualitative example (YouTube ID: 0Tmpt63T79M) for the A2T (audio-to-time) task [2].

**Fig. A5:** A qualitative example (YouTube ID: i9jTOj4Bqo4) for the V2A (video-to-audio) task [2]. Red text indicates incorrect/hallucinated descriptions.

**Figure.** A sequence of five video frames from a convention setting, with a red box highlighting the middle three frames (86s, 91s, 93s). The frames show people interacting at a booth, including a man speaking, individuals looking at objects, and a close-up of a product box on a table.

**Fig. A7:** A qualitative example (YouTube ID: d8okJ2yYyEc) for the T2V (time-to-video) task [2]. Red text indicates incorrect/hallucinated descriptions.

**Fig. A8:** A qualitative example (YouTube ID: JuS41kVKnF0) for the T2A (time-to-audio) task [2]. Red text indicates incorrect/hallucinated descriptions.

The figure displays a sequence of four video frames at timestamps 35s, 40s, 69s, and 94s, showing police officers approaching and entering a building. Below the frames, a timeline compares "Ground truth" and "Ours 3B" annotations. The ground truth segment is marked from approximately 33s to 96s, while the "Ours 3B" segment extends from 33s to 133s, indicating a longer duration of the detected event.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers