Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs

Prakhar Khatri

A controlled study shows that query-aware frame selection is the primary lever for long-video MLLM performance.

Which of the three visual-token allocation decisions—frame selection, spatial resolution, or frame count—most significantly impacts long-video MLLM performance?

Long-video models cannot process every frame, so they rely on selection rules to pick a small, fixed subset. Because existing papers change the scorer, resolution, and answering model simultaneously, it is unclear which part of the pipeline actually drives performance gains. The authors isolate these decisions by holding the scorer, prompt, and answering model constant while varying selection, spatial compression, and token reinvestment one at a time. Selection is the dominant lever: a simple, decades-old pursuit algorithm outperforms uniform sampling by up to 11.8 points, often while using half the frames.

Paper Primer

The paper uses Orthogonal Matching Pursuit (OMP) as a baseline selector. OMP is a greedy algorithm that iteratively selects the frame most correlated with the remaining query residual, then projects that direction away to ensure subsequent picks cover new information.

Selection substitutes for frame count.

On hour-long videos, OMP with 8 frames outperforms uniform sampling with 16 frames by 6.9 points.

Spatial compression is nearly free but only useful when reinvested.

Halving the per-frame spatial budget changes accuracy by at most 0.44 points, but spending those saved tokens on additional frames yields a 2–3 point gain.

Why use a 1993 algorithm like OMP instead of modern, purpose-built selectors?

OMP serves as a "no-machinery" baseline with no tuned hyperparameters. Its strong performance demonstrates that much of the gain in recent, complex selectors comes from the basic logic of query-relevance and diversity rather than specialized video-specific heuristics.

Does the choice of scorer (the encoder) change the ranking of these selection rules?

No. Replacing LongCLIP with SigLIP changed 67–84% of the selected frames, yet the relative performance ranking of the selection rules remained intact.

The Token Allocation Problem

Long‑video MLLMs must pick a few frames, and we expose which selection choices truly drive performance.

Long‑video language models cannot ingest every frame; a ten‑minute clip yields 600 images, an hour‑long one 3,600, yet only a tiny fixed slice reaches the model. This slice is the first and tightest bottleneck, so the rule that picks frames is not a peripheral detail—it determines what the model ever sees.

The core question is whether performance gains come from *which* frames are kept, *how many* are kept, or *how much* detail each frame retains.

**Table 8.** OMP minus uniform on official LongVideoBench categories. Percentage-point gains differ by duration, and the category interaction is not significant in either long bin.

The key trade‑off is between covering more of the video timeline (temporal coverage) and preserving finer visual detail (spatial detail) within a fixed token budget.

Prior Approaches to Frame Selection

Survey of prior frame selection and visual‑token reduction techniques.

Recursively partitions the video timeline to preserve coverage while selecting frames.

Treats temporal clips as arms in a budgeted bandit to allocate frame budget.

Models relevance, list‑wise diversity, and sequentiality with a DPP‑based dynamic program.

Pursues the relevance–diversity trade‑off using an adaptive key‑selection heuristic.

Greedy algorithm that adaptively adjusts relevance and diversity weights during selection.

Assigns per‑segment quotas based on relevance scores and content deviation.

Forms event‑like segments and refines query‑relevant anchors with adaptive maximal marginal relevance.

Introduces uncertainty‑triggered computation and adaptive temporal suppression.

Learns a frame‑level estimate of conditional evidence conditioned on the query.

Combines CLIP‑based query‑aware selection with multi‑resolution scaling to fit more frames under a compute limit.

Linear‑DPP selector combined with Group‑DPP importance for frame retention and dynamic resolution.

Extracts relevance from MLLM attention and jointly optimizes candidate‑pool size and per‑frame token budget.

Learns retain, compress, and drop decisions for visual tokens after encoding.

Assigns group budgets based on answerer uncertainty to guide token allocation.

Prunes visual tokens with query guidance before inference.

Uses scene structure and motion cues to retain informative visual tokens.

Allocates a fixed token budget between global context and high‑resolution local evidence via a temporal‑grounding pipeline.

Controlled Experimental Protocol

We detail the controlled experimental setup, defining selection, compression, and reinvestment interventions.

All experiments compare two policies on the same benchmark question, differing only in the controlled intervention.

Pick a benchmark question and its video.

Encode the video frames and the question stem once with LongCLIP; cache the embeddings.

Apply a selection rule (e.g., OMP or Uniform Sampling) to choose timestamps.

Resize the selected frames according to the spatial‑budget policy (fixed or reinvested).

Feed the resized frames to the MLLM and record the answer.

Repeat the same question with a second policy while keeping everything else identical; count which policy yields the correct answer.

OMP greedily picks frames that best explain the remaining query vector, iteratively removing already‑covered directions.

Iteration 1: compute dot products $e_i^{\top} q_0$ → $[0.8, 0.5, 0.1]$; pick $b_1=1$ (frame 1).

Project $q_0$ onto $e_1$: $\operatorname{Proj}_{e_1}(q_0)=\frac{q_0^{\top}e_1}{\|e_1\|^2}e_1 = (0.8,0.2)$. Residual $q_1 = q_0 - (0.8,0.2) = (0.2,-0.2)$.

Iteration 2: dot products with residual → $e_1^{\top} q_1 = 0.0$, $e_2^{\top} q_1 = 0.0$, $e_3^{\top} q_1 = -0.7$; pick $b_2=3$ (frame 3) because absolute value is largest.

Project $q_0$ onto span$\{e_1,e_3\}$ and recompute residual (omitted for brevity); the algorithm stops after two selections.

OMP first grabs the most aligned frame, then forces the second pick to cover a different direction, avoiding the redundancy that top‑k would incur.

OMP selection pseudocode – greedy orthogonal matching pursuit.

How does OMP differ from simple cosine top‑k selection?

Top‑k picks the $k$ frames with highest raw similarity to the original query $q_0$, ignoring overlap between them. OMP, by contrast, recomputes a residual after each pick and explicitly removes directions already covered, so later picks are forced to explain new components of $q_0$.

Uniform sampling draws frames at regular intervals, guaranteeing no bias toward any particular visual content.

Why would Uniform Sampling ever be a competitive baseline?

Even without learned relevance, uniformly spaced frames can capture the temporal structure of a video, and many downstream tasks benefit from covering the entire duration rather than focusing on a few high‑similarity frames that may be redundant.

**Figure a.** Score once. Every rule reads the same LongCLIP stem vector.

By holding the scorer constant and auditing token usage, we guarantee that any observed performance differences stem from the selection, compression, or reinvestment policies themselves.

Benchmarks and Metrics

We evaluate three long‑video benchmarks using a fixed visual‑token budget.

Evaluation uses visual‑token allocation with $k\!=\!8$ selected frames and $k\!=\!16$ reinvested frames.

Table 1 lists the frame budgets applied across all benchmark experiments.

LongVideoBench measures multimodal model performance on long videos, offering 1,337 validation videos ranging from 15 s to 3600 s.

Video‑MME evaluates multimodal models on 2,700 videos, focusing on open‑ended question answering across diverse content.

LVBench contains 1,549 long‑video samples and measures model performance on a distinct set of tasks.

Selection vs. Frame Count

Selecting frames yields higher accuracy than simply using more frames, even with half the visual tokens.

OMP with eight frames reaches .5461 versus uniform sampling with sixteen frames at .4770 on 3600 s LongVideoBench videos.

Result is a 6.9‑point accuracy gain (p=.0011).

OMP with sixteen frames reaches .6578 versus uniform sampling with thirty‑two frames at .6044 on 600 s videos.

Result is a 5.3‑point accuracy gain (p=.018).

Both contrasts compare a cheaper visual‑token allocation against a more expensive one, and the cheaper policy wins—providing stronger evidence than equal‑budget comparisons.

Scorer Independence

Scorer swap shows the selector ranking is robust across encoders.

The selector ranking (Uniform < Top‑k < OMP) remains unchanged when swapping the frozen scorer from LongCLIP to SigLIP.

Table 3 shows identical ordering under both scorers, with OMP still outperforming Uniform and Top‑k.

All other variables—prompt boundary, frame budget, answerer, and evaluation harness—were held constant across the swap.

Spatial Resolution Impact

How compression and allocation choices affect long‑video MLLM accuracy.

This ablation isolates the effect of pixel‑level compression versus frame‑selection strategies on long‑video MLLM performance.

The budget limits how many pixels (visual detail) each selected frame can contribute to the fixed token budget.

**Figure 2.** OMP’s gain over uniform sampling appears only once the 1 fps candidate pool outgrows the eight-frame budget. At 15 s, uniform, top-k, and OMP have identical accuracy (.7249); the gain is 7.8 points at 600 s and 7.5 at 3600 s. The effect switches on with pool size rather than growing with duration.

Compression is cheap, yet the allocation shape still matters: a residual‑proportional schedule does not beat a flat split at the same mean budget, and OMP’s ranking adds no detectable accuracy when used to distribute pixels.

Reinvesting Tokens

Spending saved token budget on more frames yields consistent accuracy gains.

Reinvesting the saved token budget into additional compressed frames improves accuracy across all evaluated benchmarks.

Table 5 shows that using sixteen compressed frames instead of eight full‑resolution frames yields +2.24 pp on LongVideoBench, +3.04 pp on LVBench, +1.56 pp on Video‑MME, and +2.39 pp on GPT‑5‑mini LVBench.

All comparisons keep the total visual‑token budget constant: the compressed arm uses the same or fewer tokens than the full‑resolution baseline, and the same resize rule is applied to Video‑MME and LVBench.

**Figure 3.** LongVideoBench 3600 s accuracy on fixed OMP-8 timestamps across spatial budgets. The flat 50% control and the residual-proportional schedule are indistinguishable, so OMP rank alone does not provide a useful resolution policy.

**Table 5.** Reinvestment at equal or lower visual-token cost on the audited LongVideoBench bins. The compressed arm approximately doubles temporal coverage; Video-MME and LVBench use the same resize rule without a separate token audit.

Residual Trace Analysis

Later OMP picks add relevance without further shrinking the residual norm.

OMP’s residual trace does not decay as signal‑processing theory would predict: after a single pick the residual norm remains $0.972$ of its initial value, and after eight picks it is $0.967$, with no further reduction even at sixteen picks. Meanwhile, the residual’s correlation with the frame being selected drops sharply—from $0.233$ at the first pick to $0.004$ at the eighth and essentially zero by the sixteenth.

Despite the stagnating norm, later frames are not useless: they keep a cosine of about $0.20$ with the original query. This reflects that the text query sits almost orthogonal to the dominant directions of the frame‑embedding cone—a modality gap visible in Figure 5b—so eight greedy OMP picks only capture those dominant directions, leaving little room for further residual reduction regardless of the selected frames.

An exploratory audit of 93 OMP failures at $k=8$ across the two long bins revealed a dominant pattern of off‑topic novelty (20 of 41 cases for 600 s and 35 of 52 for 3600 s), where several picks landed on visually distinctive but question‑irrelevant material such as title cards or dark transitions. The mirror‑image failure appears on sequence‑of‑scenes questions: dense top‑$k$ sampling clusters on one relevant moment and misses another, exactly where OMP’s spread proved beneficial.

**Figure 4.** LongVideoBench 600s and 3600s pooled: sixteen compressed OMP frames outperform eight full-resolution OMP frames at a measured token ratio no greater than one.

Model Generalization

Identical frame allocations affect models differently, revealing answerer‑benchmark interactions.

Table 6 isolates the effect of swapping the answerer while keeping the visual‑token allocation identical, allowing a direct comparison of OMP versus uniform sampling across benchmarks.

InternVL3‑2B gains on Video‑MME across all durations, whereas InternVL3‑8B shows gains only on short videos.

Table 6 reports +6.00 pp (p=1.2 × 10⁻⁴) for short videos, +3.33 pp (p=.046) for medium, and +4.22 pp (p=.0062) for long videos for the 2B model; the 8B model shows no improvement on medium and long videos.

**Figure 5.** LongCLIP geometry on LongVideoBench-600 s ($n=411$). (a) Residual correlation vanishes while residual norm changes little; the line connects logged picks 1, 8, and 16 and is not a dense trace. (b) Within-video frames occupy a coherent cone, while the text query is nearly orthogonal to the frame dictionary. In this space, OMP mainly suppresses directions already represented by earlier selections.

Synthesis of Allocation Levers

Selection dominates performance under a fixed visual‑token budget.

Under a fixed visual‑token budget, frame‑selection rules consistently provide the largest accuracy gains.

Table 2 and Table 7 show OMP, AKS, and FOCUS outperform uniform sampling across all three benchmarks.

Compression of selected frames is essentially free but does not improve accuracy unless the saved tokens are reinvested in temporal coverage; thus resolution acts as a slack variable.

**Table 7.** Two matched-condition harnesses running the same rules. Published columns are Chen et al. (2026a), Table 1, Qwen3-VL-8B at eight frames with a Long-CLIP encoder—the same answerer family, budget, and encoder as ours—where LDDR-select is their stage-1 row. The first line gives absolute uniform-sampling accuracy; all other cells are gains over the uniform row of the same column. $^\dagger$ marks rules we run on a substituted scorer (both score with BLIP-ITM natively), and $^*$ marks the FOCUS schedule replay of Appendix A. OMP has no published counterpart, and MDP3 and Q-Frame appear in their table but not ours (§7).

**Figure 6.** Coverage-gated selector variants against OMP at $k=8$ on the 600 and 3600 s LongVideoBench bins. The shaded $\pm 1.5$-point band is a visual aid, not a statistical criterion.

Caveats and System Constraints

Key constraints and caveats of the reported experiments.

This section enumerates the principal caveats that limit the interpretability and generality of the reported results.

FOCUS* replays the original frame‑selection schedule using dense LongCLIP similarity scores instead of the budgeted ITM scorer, so it evaluates the schedule under a matched scorer rather than the full FOCUS pipeline.

LDDR‑select is limited to stage 1; the stage‑2 allocation contrast is reconstructed from the paper rather than executed from the authors’ code, so those rows should not be read as reproductions of the original method.

Equivalence margins were set after inspecting the data, e.g., the Qwen result meets a ±3‑point margin but fails a tighter ±2‑point bound, making the claim descriptive rather than a pre‑specified confirmatory test.

Statistical tests are uncorrected for multiplicity, and Video‑MME compression lacks a paired comparison; LVBench differences are likewise reported descriptively, limiting inferential strength.

Mechanism evidence relies on encoder‑specific statistics (residual geometry and Gram matrices) from frozen LongCLIP embeddings on LongVideoBench‑600 s; other encoders and components such as BLIP‑ITM heads or subtitle‑aware selection remain untested.

Secondary experiments were run on a different hardware stack (L40S GPUs) than Tables 2–5 (RTX PRO 4500 GPUs), revealing baseline drift (e.g., the uniform eight‑frame arm is +0.36 points, the OMP arm –0.71 points) that prevents direct comparison of absolute accuracies across sections.

The evaluation scope is incomplete: subtitles are disabled, GPT‑5‑mini compression uses pixel accounting, and several baselines (MDP3, Q‑Frame) lack validated numbers, so the reported gaps may reflect missing components rather than true performance differences.

Summary of Findings

Key findings, ethical stance, and data release are summarized.

Eight carefully chosen frames outperform sixteen uniformly spaced frames in the hour‑long LongVideoBench bin. Running OMP unchanged beats uniform sampling by 5.7–11.8 points and stays within one point of a purpose‑built modern selector. Removing roughly half of the per‑frame spatial budget changes performance by at most ±3 points, while reallocating those tokens to additional keyframes yields another two to three points gain.

These results suggest a simple recipe: select relevant frames first, compress them second, and use the saved tokens to view more of the video, though the benefit varies with video length, benchmark, and model.

The experiments rely on public video‑QA benchmarks and frozen open or API models, without collecting new human subjects, personal data, videos, or annotations. Only question stems (not answer options) are fed to the selector, and benchmark videos remain under their original licenses and are not redistributed.

All evaluation code, selector implementations, analysis scripts, selected frame indices, and per‑item predictions (subject to benchmark licenses) are released at the provided GitHub repository. The release includes the scorer‑swap pipeline (§5.2), fused‑query arms (§3), the script generating Table 7, and pre‑registration documents, while excluding benchmark video files and API credentials.

Appendix: FOCUS* Schedule Replay

The appendix details extra ablations, failure audits, and selector‑variant sweeps supporting the main analysis.

Every row in Table 2, including the starred FOCUS⋆ entry, consumes the same dense 1 fps LongCLIP stem‑score vector. The original FOCUS method treats scoring as a budgeted online process that pulls ITM evidence, whereas our replay preserves the temporal clip schedule and feeds pre‑computed LongCLIP scores to the bandit, isolating the schedule under a matched scorer without assessing the full FOCUS pipeline’s efficiency or accuracy.

The replay uses 16 s clips, three coarse pulls per clip, $\alpha = 0.25$, a pull budget equal to half the candidate pool, and approximately $k/4$ surviving clips. Unobserved rewards are interpolated from the nearest observed frame, and the random seed is fixed per item; this boundary explains the star annotation and why no conclusion about native FOCUS ITM protocol can be drawn.

LongVideoBench question‑category tags invite the hypothesis that temporally‑referred questions need a different frame‑selection regime. We compare OMP with uniform sampling on the temporally‑referred T* group versus all other categories, without relabeling the data, and find equivalent results, indicating that within this scorer geometry and budget range, swapping relevance‑diversity trade‑offs yields little change.

Two independent model‑assisted visual passes inspected OMP failures at $k = 8$ through a frame viewer: 41 cases (16 categories) in the 600 s bin and 52 cases (17 categories) in the 3600 s bin. The sample, chosen to cover categories and include both OMP‑only and shared failures, is not random; reviewers compared timestamps, frames, questions, gold answers, and model predictions, tagging missed evidence, temporal offset, answerer error despite adequate frames, and unanswerable questions.

This sweep changes only the subset rule on the same LongCLIP cache. MMR underperforms; query‑blind DPP ($\beta = 0$) collapses toward uniform sampling, while query‑weighted DPP, residual floors, and partial orthogonalization cluster around OMP, with no DPP variant significantly exceeding OMP in either long bin.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers