Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

Yunheng Li, Guohong Mu, Hao Li, Shengsheng Qian, Dingwen Zhang, Qibin Hou, Ming-Ming Cheng

OraRL integrates ground-truth annotations as oracle rollouts to improve video MLLM training efficiency and precision.

How can we improve reinforcement learning for video MLLMs by using ground-truth annotations as oracle rollouts rather than just rewards?

Multimodal Large Language Models (MLLMs) struggle with fine-grained video perception because supervised fine-tuning lacks task-level supervision, and existing reinforcement learning methods rely on sparse, high-quality on-policy rollouts. OraRL treats each ground-truth annotation as an "oracle rollout" appended to the training group, using a decoupled advantage estimator to prevent the oracle from skewing the baseline and inverting policy advantages. This approach achieves state-of-the-art results across seven video task families while reducing training time and eliminating the need for costly chain-of-thought reasoning.

Paper Primer

The core mechanism, annotation-as-rollout, serializes annotations into the model's response format and injects them directly into the reinforcement learning group. To prevent "advantage inversion"—where the high-reward oracle artificially raises the baseline and penalizes strong on-policy rollouts—OraRL computes advantages using only on-policy rewards and encodes the oracle-policy gap through separate directional gain and detached advantage terms.

Efficiency is maintained via sign-balanced pruning: the system retains the oracle and a subset of the strongest positive and negative rollouts, ensuring the model learns from both successful and failed explorations while reducing backward computation.

OraRL significantly outperforms existing RL paradigms in both accuracy and computational efficiency.

Video-ORA-9B improves temporal mIoU from 62.5 to 66.0 and tracking AO from 73.0 to 78.2, while reducing step time to 2.2× that of SFT (compared to 4.9× for GRPO with CoT). 33.5% reduction in training step time and 130ms inference latency vs 4,780ms for CoT-enabled models.

The method scales consistently across model sizes and data budgets.

Performance improves monotonically from 0.8B to 9B parameters, and OraRL consistently outperforms GRPO across all evaluated data budgets up to 100k prompts.

Why does naive oracle injection fail in existing RL frameworks?

Including a high-reward oracle in the group normalization baseline raises the threshold for a positive advantage, causing above-average on-policy rollouts to be incorrectly penalized (advantage inversion).

Does this approach require chain-of-thought reasoning to achieve high precision?

No; OraRL achieves superior performance using answer-only generation, which significantly reduces inference latency and training costs compared to CoT-based methods.

By treating annotations as direct optimization targets rather than mere reward references, OraRL provides a scalable path to fine-grained video perception that bypasses the latency and cost bottlenecks of reasoning-heavy training.

Introduction and Motivation

We expose why standard RL struggles for video MLLMs and why oracle‑guided RL offers a principled fix.

Standard reinforcement‑learning (RL) post‑training for video MLLMs relies on noisy reward signals that rarely provide a clear learning direction, and chain‑of‑thought (CoT) generation makes each rollout expensive without delivering proportional gains.

A Video MLLM is a large language model that can understand and generate video‑level content, handling tasks that require precise temporal and spatial reasoning rather than just producing a single caption.

We propose annotation‑as‑rollout: each ground‑truth annotation is inserted into the on‑policy group as an “oracle rollout,” giving the policy a reliable positive target without relying on CoT. Naïvely mixing this high‑reward oracle with on‑policy rollouts creates advantage inversion— the oracle lifts the baseline so that many otherwise good rollouts receive negative advantages, suppressing useful exploration.

**Figure 1.** **Video-ORA: one model, trained by a single OraRL recipe, for unified video perception.** Each column presents a representative input and a family-level score computed over common benchmark coverage. Video-ORA-9B, evaluated without chain-of-thought decoding, outperforms every displayed baseline [71, 118, 21, 2, 96, 1] across seven task families. See Sec. 4 for detailed metrics.

The key shift is moving from reward‑only RL to oracle‑guided RL, which supplies a stable, high‑quality anchor for policy optimization.

Related Work

Related work surveys video MLLM foundations, fine‑grained perception, and RL baselines.

Multimodal foundation models for video understanding connect a visual encoder to a language model via lightweight projection or cross‑attention modules. Prominent examples include Video‑ChatGPT, the Video‑LLaMA series, Chat‑UniVi, the VideoChat series, and the LLaVA‑OneVision and LLaVA‑Video families. Large‑scale video‑text pretraining has also yielded dedicated foundations such as InternVideo and InternVideo3, while open MLLMs like Qwen3‑VL, Qwen3.5, InternVL3, InternVL3.5, Molmo2, Keye‑VL, MiniCPM‑V, MiMo‑VL, and Eagle2.5 extend these ideas to unified image‑video understanding.

Fine‑grained video perception splits into temporal grounding/highlight detection and spatial reasoning. Temporal grounding has been tackled by TimeChat, VTimeLLM, Grounding‑GPT, VTG‑LLM, Grounded‑VideoLLM, LLaVA‑ST, and the TimeLens generalists, while spatial tasks rely on reasoning segmentation, referring video object segmentation, and visual tracking, with transformer specialists still dominant. Recent work such as OneThinker unifies many of these tasks but depends on chain‑of‑thought supervision, unlike the answer‑only rollouts used in this paper.

GRPO estimates advantages by normalizing rewards within a group of rollouts that share the same query, avoiding a learned critic.

Reinforcement learning adapts multimodal models beyond supervised imitation, with PPO providing the standard policy‑optimization backbone. GRPO, as described above, normalizes advantages within query groups, while LUFFY injects teacher traces via regularized importance sampling to strengthen weak groups. CPPO prunes rollouts with low absolute advantages, and reward shaping enriches a fixed budget. All these methods rely on on‑policy rollouts; OraRL instead retains the oracle annotation as a dedicated rollout, excluding it from the baseline to avoid advantage inversion.

The OraRL Mechanism

Describes the OraRL mechanism that injects oracle rollouts to stabilize policy updates.

Standard RL for video MLLMs suffers from noisy rewards that can invert advantages, penalizing good on‑policy rollouts.

OraRL (Oracle Reinforcement Learning) treats each ground‑truth annotation as an extra “oracle rollout” appended to the on‑policy group, giving a stable high‑quality target while keeping all original rollouts for comparison.

How does OraRL differ from simply replacing low‑reward rollouts with the oracle?

OraRL appends the oracle rollout instead of replacing any on‑policy rollout, preserving the full set of $n$ on‑policy samples for relative comparison while still providing a strong positive anchor.

An Oracle Rollout is the ground‑truth annotation transformed into the same sequence format as a model response, so it can be treated as a regular rollout during RL.

Why not treat the oracle rollout as a fixed target and exclude it from the RL loss?

Including the oracle in the loss lets the policy learn to reproduce the high‑quality behavior directly, and the sign‑balanced pruning ensures the oracle’s gradient is weighted alongside informative on‑policy rollouts.

Sample $n$ on‑policy rollouts $\mathcal{O}_{\text{op}}$ for query $q$.

Serialize the ground‑truth annotation into $o_{\text{gt}}$ and form $\mathcal{O}_{\text{aug}}$.

Compute on‑policy advantages $A^{(0)}_i = r_i - \mu_{\text{op}}$.

Calculate directional gain $g_q$ and apply it to above‑mean rollouts to obtain $U_i$.

Derive detached oracle advantage $A_{\text{gt}}$ using weight $w_q$.

Perform sign‑balanced pruning to keep the oracle and top‑$K^{+}$ positive and $K^{-}$ negative rollouts.

Center and rescale the retained advantages with $\lambda_q$.

Update the policy using the clipped objective $J_{\text{ours}}(\theta)$.

Compute on‑policy mean $\mu_{\text{op}} = (2+3+1+4)/4 = 2.5$.

Standard deviations: $\sigma_{\text{op}} \approx 1.12$, $\sigma_{\text{aug}} \approx 1.71$ (including $r_{\text{gt}}$).

Directional gain $g_q = \operatorname{clip}(1.71 / (1.12+ \epsilon), 1, 4) \approx 1.53$.

Raw advantages $A^{(0)} = [ -0.5, 0.5, -1.5, 1.5 ]$.

Apply $g_q$ to positive entries: $U =

Experimental Results

Video‑ORA‑9B achieves state‑of‑the‑art performance across all evaluated video tasks.

Video‑ORA‑9B sets new state‑of‑the‑art across all evaluated video tasks, improving the best prior metric by up to 5.0 % mIoU on temporal grounding.

Table 1 shows Video‑ORA‑9B leading every temporal‑grounding metric; the mIoU gap over the specialist TimeLens2‑8B reaches 5.0 %.

Tables 1–5 compile the full set of results, confirming that Video‑ORA‑9B consistently outperforms both proprietary and open‑source baselines across temporal grounding, spatial grounding, video question answering, tracking, and spatial‑temporal grounding.

Extended Evaluations

Video‑ORA‑9B sets new state‑of‑the‑art scores across all evaluated benchmarks.

Video‑ORA‑9B achieves the highest VSI‑Bench average of 73.1, beating the strongest proprietary model by 18.0 points.

Table 6 reports an average of 73.1 for Video‑ORA‑9B, while the best proprietary baseline reaches 55.1.

On STVG, Video‑ORA‑9B ranks first on all four metrics, gaining 3.0 points on tIoU@0.5 and a striking 15.5‑point boost on sIoU@0.5 versus Qwen3.5‑9B. Segmentation results show a 29.2 J&F improvement on MeViS and 42.2 J&F on ReasonVOS, confirming that the backbone alone cannot generate usable mask prompts.

**Table 8.** Results on MMSI-Bench [95] and MindCube-Tiny [75]. Both report accuracy; Avg. is their mean.

**Table 7.** Segmentation results. RefCOCO/+/g [101, 54] use cIoU, whereas MeViS [17] and ReasonVOS [3] use J&F.

Scaling and Ablations

We evaluate how each component and scaling choice impacts performance and efficiency.

Macro‑average performance rises from 51.8 pts at 0.8 B to 66.2 pts at 9 B, a 14.4‑point gain.

Figure 4 shows consistent improvement across all seven task families under the same OraRL configuration.

Tracking margin expands from 11.3 pts (0.8 B) to 32.2 pts (9 B), a 20.9‑point increase.

Figure 4 panel for tracking demonstrates the widening gap as model size grows.

OraRL beats GRPO by 2.6 pts on the video‑perception aggregate when scaling prompts from 6.4k to 100k.

Figure 5 (panels a & b) reports the widening margins of 4.1 pts and 2.6 pts respectively.

Adding 9 k video prompts to a spatial‑intelligence RL run raises the three‑benchmark mean by 1.2 pts.

Table 9 shows averages of 49.2 pts (spatial only) versus 50.3 pts (augmented with video).

OraRL improves the average by 2.7 pts over answer‑only GRPO while reducing step time by 33.5 %.

Table 11 reports 58.7 pts (OraRL) vs 56.0 pts (answer‑only GRPO) and step‑time 93.9 s vs 135.6 s.

Compared to the strongest on‑policy group method, OraRL achieves a 2.0‑point higher average.

Table 12 shows the best on‑policy variant at 58.8 pts, OraRL at 60.5 pts.

OraRL reduces advantage‑inversion flip rate to 1.9 % versus 22.4 % for naive oracle mixing, a 20.5 % reduction.

Figure 6 and Table 14 quantify the flip‑rate drop.

Removing the directional‑gain component lowers the overall average by 1.5 pts.

Table 15 entry “Without directional gain” reports 60.5 pts vs 62.0 pts (full OraRL).

Removing the detached‑oracle‑advantage component lowers the average by 1.4 pts.

Table 15 entry “Without detached oracle advantage” (not shown explicitly) is inferred from the reported 1.4‑point drop.

Removing the reward‑gap weight reduces the average by 1.1 pts.

Table 15 entry “Without reward‑gap weight” shows a 1.1‑point decrement.

Pruning rollouts to $\kappa$ = 0.5 cuts step time by 30.1 s while incurring only a 0.4‑point average loss.

Table 16 reports 92.5 s → 62.4 s step time and average 63.1 pts → 62.7 pts.

**Figure 4. Model scaling from 0.8B to 9B.** Under the same OraRL recipe, Video-ORA improves across all task families with model size (x-axis, in billions of parameters, log scale) and outperforms its Qwen3.5 backbone at every scale. The final panel reports the macro-average; Spatial Grounding averages RefCOCO-family R@0.5, while ST-Grounding averages tIoU and sIoU.

**Figure 5.** Task-group data scaling and reward dynamics under the controlled 9B protocol. (a) The video-perception aggregate is the macro average over temporal grounding, tracking, segmentation, and video question answering. (b) The spatial-intelligence aggregate averages VSI-Bench, MMSI-Bench, and MindCube. (c) Total training reward over one epoch for OraRL and GRPO.

**Figure 6.** Advantage inversion. Shading marks inverted rollouts; labels give full-data rates among GRPO-positive samples.

**Figure 7. Training and inference efficiency.** (a) Accuracy against training cost, where OraRL gives the strongest trade-off and CoT substantially raises cost. (b) End-to-end single-request latency on one H20 in BF16. All models see the same 2-fps input of about 120K tokens per video. Solid bars give median latency split at the first token, pale tails run to P90, and the numbers report medians.

Qualitative Analysis

OraRL’s predictions align almost perfectly with ground‑truth annotations across all tasks.

OraRL achieves perfect qualitative agreement on all held‑out examples across the seven task families.

Figures 9–15 show GT (amber dashed) and PRED (green) markings overlapping exactly for every presented case.

The qualitative suite covers temporal grounding, spatial grounding, mask‑aware segmentation, visual tracking, spatial‑temporal grounding, video QA, and spatial intelligence, each demonstrating exact or near‑exact matches between model output and annotation.

The figure illustrates a temporal action localization task. It shows a sequence of four video frames from a kitchen setting, with timestamps at 90.6s, 93s, 100s, and 102.5s. A timeline spanning 0s to 156.6s displays a predicted interval [93, 100]s in orange and a ground truth interval [93, 100]s in green. Below the timeline, the ground truth (GT) and prediction (PRED) are explicitly stated as "93 to 100" within answer tags.

The image displays a comparison between predicted hints and a ground truth (GT) mask for an object detection or segmentation task. The left panel shows "predicted hints" overlaid on a street scene, including a bounding box around a car, several green dots within the box, and green 'x' markers outside the box. The right panel shows the "GT mask" with a yellow outline highlighting the same car. Below the images, the text "GT run-length mask, 960x540; tight box [812, 135, 917, 252]" provides technical metadata for the ground truth.

Training Data Details

We detail the data construction, RL setup, and reward formulas that underpin the experiments.

SFT and RL both cover seven task families, but their sampling distributions differ markedly. SFT holds 284,779 prompts with 18 % structured and 82 % answer‑only examples, while RL uses 100,032 prompts with 63 % structured and 37 % answer‑only.

**Figure 8.** Composition of the SFT and RL training data. Open and filled markers indicate each task's mixture proportion during SFT and RL, respectively. Tasks are grouped by output format, and the rightmost columns report exact example counts.

Source datasets span a wide range: temporal grounding from TimeLens‑Sources 100K; tracking from GOT‑10k, TrackingNet, and ElysiumTrack; segmentation from the One‑Thinker pack (ReVOS, Ref‑YouTube‑VOS, Ref‑SAV, MeViS, DAVIS‑17, COCO referring); spatial grounding from RefCOCO/+/g; spatial‑temporal grounding from LLaVA‑ST‑STVG; video QA from LLaVA‑Video‑178K, LongVILA, STAR, CLEVRER, Video‑Holmes; and spatial‑intelligence from VSI‑590K, ScanNet, ScanNet++, ARKitScenes, SenseNova‑SI‑800K.

Prompts are sampled from these pools according to the task proportions in Fig. 8, with a hard cap of two prompts per video. For RL, each candidate’s difficulty is estimated by running the SFT checkpoint and scoring its prediction; only candidates scoring at least 0.80 are allowed to occupy 15 % of the RL subset, emphasizing intermediate‑difficulty examples.

Supervised fine‑tuning (SFT) and reinforcement learning (RL) configurations are summarized in Table 17. During SFT the vision tower and multimodal aligner remain frozen, while the language‑model parameters are initialized by interpolation between the pretrained base and the SFT‑trained model.

In RL each prompt yields eight on‑policy rollouts generated by the current policy, plus a ninth oracle rollout that contains the ground‑truth annotation. All rollouts follow the task‑specific answer format without any chain‑of‑thought generation.

Mixture construction first normalizes all source datasets to a common schema, discards examples with inaccessible media or unrecoverable annotations, and removes duplicates identified by content similarity.

Optimization uses a pruning ratio $\kappa$ = 0.5; sign‑balanced pruning keeps the oracle rollout together with one positive and two negative on‑policy rollouts, and only these four rollouts contribute to gradient computation.

The default video pipeline samples at 2 fps, keeps at most 128 frames, and enforces a per‑clip pixel budget of 8.4 million. Tracking tasks use 32 uniformly sampled frames, while video‑QA, segmentation, and spatial‑intelligence tasks receive higher budgets (10.5–16.8 million pixels). The 4 B experiments run on four nodes with eight NVIDIA H20 GPUs each; the full 9 B RL run uses eight such nodes with full parameter sharding.

**Table 19.** Complete results on MMSI-Bench [95] and MindCube-Tiny [75]. Scores are accuracies; C/O/R denote the MMSI-Bench camera/object/region categories.

**Table 20.** Results on ReVSI [111]. Numerical and multiple-choice tasks use MRA and accuracy, respectively; Avg. is the reported aggregate over seven tasks, and the last column gives the original VSI-Bench average. All baseline rows, including their VSI-Bench averages, are quoted from [111], which evaluates each model at its native frame setting and restricts proprietary models to a 1,093-question subset, so these values differ from our own measurements in Tab. 6. The backbone and Video-ORA rows are evaluated by us.

Relative‑direction analysis reveals a pronounced asymmetry: Video‑ORA reaches 91.5 % forward accuracy but only 8.3 % backward accuracy at 128 frames, a pattern shared by other models except Gemini‑3‑Pro, which remains roughly symmetric.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers