ChronoReplay: One Answer Interface Is Not Enough for Streaming Spatial Intelligence
Yihao Hu, Yitong Lin, Pan Wang, Zhihao Wen, Hang Wang, Yiwen Zhao, Deyuan Liu, Yutong Li, Mingda Wan, Yuchen Mao, Tao Lin
ChronoReplay balances heterogeneous spatial supervision to prevent interface lock-in in streaming VLMs.
How can we train spatial VLMs to handle both categorical spatial relations and metric trajectory reasoning without one task degrading the other?
Streaming spatial models often struggle to answer diverse query types—like categorical options, metric distances, and temporal timestamps—over the same causal history. Training on one answer format often degrades performance on others, a phenomenon the authors call "interface lock-in." ChronoReplay addresses this by replaying supervision as chronologically ordered episodes that retain native answer formats, while using a balanced objective to regulate the influence of different task families. This approach improves performance across three major spatial benchmarks simultaneously, including a 13.41-point gain on OST-Bench under identical observation access.
Paper Primer
The core mechanism hinges on two stages: Causal Replay construction and family-balanced learning. Causal Replay aligns every target with its valid chronological prefix, ensuring the model learns how accumulated motion and state changes produce different answers from the same recent observations. Family-balanced learning then normalizes the loss across different task types, preventing high-frequency categorical data from drowning out metric or temporal trajectories.
ChronoReplay achieves consistent cross-interface transfer without new memory modules or annotations.
Performance gains across OVO-S-Bench, OST-Bench, and UCS-Bench compared to single-family baselines. +13.41 points on OST-Bench, +7.44 on UCS-Bench, and +4.91 on OVO-S-Bench.
The authors isolate the contribution of chronological alignment using a "SHIFTED" control, which rotates episode turn blocks while keeping content constant. This intervention confirms that preserving the original causal order provides a repeatable ~2.41-point gain on OST-Bench, validating the importance of chronological alignment over isolated snapshot training.
Why does training on one spatial interface (e.g., categorical) often hurt performance on another (e.g., metric)?
The authors observe "interface lock-in," where the model specializes its representation to satisfy the dominant readout's requirements, effectively erasing the finer-grained distinctions needed for other formats.
Does this method require new memory architectures or specialized scene graphs?
No. ChronoReplay is a post-training recipe that uses the existing autoregressive decoder and visual features, relying on the organization of training data rather than structural modifications to the model.
The paper assumes that spatial histories are inherently ambiguous if detached from their origin; therefore, the "causal prefix" is the load-bearing unit of supervision, not the individual question-answer pair.
Researchers can improve multi-task spatial reasoning by treating training data as chronologically aligned episodes rather than independent snapshots, effectively forcing the model to learn state accumulation as a unified skill.
Introduction
Introducing the gap: spatial VLMs excel at categorical queries but lose metric displacement needed for trajectory reasoning.
Training spatial VLMs on a single supervision family creates an “interface lock‑in” effect: the model becomes proficient at the trained readout (e.g., categorical side‑of‑object) while its ability to answer other native queries (metric distance or temporal ordering) degrades or remains stagnant.
Spatial VLMs must retain both categorical relations (which side an object lies on) and metric displacement (how far it is) to answer the full spectrum of trajectory queries; losing either side breaks downstream reasoning.
**Figure 1.** Why reusable spatial state. Categorical preservation and trajectory supervision constrain complementary spatial distinctions. Causal Replay combines them in shared parameters. The illustrated readouts span categorical, metric, temporal, and reference-frame queries.
**Figure 4.** Historical context in spatial reasoning. The first question retrieves an earlier state in which the bathtub and folded towel lie on the observer's left. The second evaluates a later location relative to the facing direction in Round 2. Teal indicates annotated states, and red shows the control's incorrect retrieval or displacement. Both questions depend on a reference established earlier in the episode.
The core trade‑off is preserving categorical side‑of‑object information while also retaining metric displacement; ChronoReplay’s dual‑supervision balances these to enable reliable trajectory reasoning.
Problem Formulation
Survey of streaming spatial benchmarks and the gaps they expose for causal VLMs.
Streaming spatial intelligence has been explored through causal video models that compress history into memories, hierarchical summaries, or key‑value caches. Benchmarks such as StreamingBench, OVO‑Bench, and SimpleStream formalize timestamped evaluation, while memory designs like L2‑VMAS and AtlasVA propose latent‑memory or exemplar‑based stores. Recent causal‑stream suites—OVO‑S‑Bench, OST‑Bench, and UCS‑Bench—extend these ideas to categorical, continuous, and timestamp‑dependent queries, exposing the need for representations that support diverse native answer formats.
A causal streaming benchmark that asks purely categorical spatial queries—e.g., “which region contains the object?”—and expects a discrete region label as the answer.
A benchmark that mixes categorical and metric queries, requiring the model to output precise distances or angles alongside region labels within a causal video stream.
A benchmark that adds timestamp‑dependent queries, asking the model to report the exact time an event occurred while processing a continuous observation stream.
**Figure 3.** One origin, two native readouts on an OST-Bench episode. Left: the five observation rounds and two questions. Q1 (Round 4): when were both the bathtub and the folded towel on the observer's left? Q2 (Round 5): using the Round-2 facing as forward, did the observer move left or right? Right: ChronoReplay answers Round 2 and Left, both correct, whereas the origin-shifted control (SHIFTED) returns Round 4 and Right under identical observations. Teal marks annotated states, red the control's errors; the full-size episode is in Appendix F.
Causal Replay Construction
Build training data that keeps each answer in its original format and preserves the order of observations.
When a model must answer categorical, metric, and temporal questions, converting all answers to a single format strips away units, timestamps, and the relational structure that make those answers meaningful.
Each answer is kept in the exact value space of the interface that produced it, so a distance stays in meters, a class stays as a discrete label, and a timestamp stays as a time value.
For query 1 the native target is the class label “chair” – no conversion needed.
For query 2 the model must output the number 2.3 together with the unit “m”; a naïve conversion to a unit‑less scalar would lose the meter information.
For query 3 the target is the timestamp 12.5 s; discarding “seconds” would make the answer ambiguous for downstream reasoning.
Keeping each answer in its original format lets the downstream reasoning module treat distances, categories, and times uniformly without having to infer missing units or ordering.
Why can’t we simply map every answer to a floating‑point number and let the model learn the missing semantics?
Because the mapping would conflate fundamentally different quantities (e.g., “red” vs. 2.3 m), making the loss function ambiguous and forcing the model to guess units or categories that are not present in the data.
ChronoReplay builds two families of training examples—single‑turn “preservation” cases and multi‑turn “trajectory” cases—by replaying each episode in its original chronological order while keeping every target in its native format.
Encode $c_{i1}$ → $z_{i1}=F_{\theta}(c_{i1})$; store categorical target “yes”.
Extend prefix to $c_{i2}$, encode → $z_{i2}=F_{\theta}(c_{i2})$; attach metric target 3.5 m to $z_{i2}$.
Extend to $c_{i3}$, encode → $z_{i3}=F_{\theta}(c_{i3})$; attach temporal target 7.2 s to $z_{i3}$.
The three representations $z_{i1},z_{i2},z_{i3}$ form a trajectory that the model learns to predict at each step, preserving both the categorical decision and the evolving metric/temporal information.
This toy episode shows how ChronoReplay supplies the model with a growing context, so it learns not only isolated answers but also how those answers evolve as the scene changes.
How does ChronoReplay differ from simply stacking independent QA pairs in the training set?
ChronoReplay keeps the original temporal order and attaches each target to the exact causal prefix that generated it; independent QA pairs lose that ordering, making metric and temporal targets ambiguous because the model no longer sees the displacement that gave rise to them.
Preservation examples are single‑turn queries that keep categorical targets exactly as they appear in the source interface, ensuring the model learns the original class distribution without any transformation.
Why bother keeping a separate “preservation” family if the metric/temporal families already contain categorical queries?
Preservation isolates the pure classification signal, preventing it from being diluted by the longer trajectories where the model also has to predict distances or timestamps. This ensures the categorical decision boundary is learned as sharply as in the original dataset.
Trajectory examples are multi‑turn episodes where each metric or temporal target is tied to the growing causal prefix, letting the model see how a displacement or event order changes the answer over time.
What would happen if we trained on metric targets without attaching them to their causal prefixes?
The model would receive isolated distance values without knowing which prior movements produced them, leading to ambiguous gradients and poor generalization to unseen trajectories.
**Figure 2.** Overview of ChronoReplay. Stage I (Causal Replay construction) builds ordered trajectory supervision $D_T$ and single-turn preservation examples $D_P$. Stage II (family-balanced learning) combines their normalized native-format risks with weights $1 - \lambda$ and $\lambda$. *Out of scope* marks observations beyond a query's causal frontier.
Family-Balanced Learning
Balancing loss contributions from preservation and trajectory families to prevent domination.
Because the preservation family contains far more examples than the trajectory family, naïve joint training would let the larger family dominate the shared parameters, reproducing the same behavior despite the added trajectory data.
We prevent one family from overwhelming the other by normalizing each family’s risk and then explicitly mixing them with a controllable ratio.
How is Family‑Balanced Learning different from simply scaling one family’s loss by a constant factor?
Simple scaling would still be sensitive to the number of examples because the unscaled loss averages over raw counts. Family‑Balanced Learning first normalizes each family to an average loss (removing count bias) and then mixes them with a coefficient derived from a separate hyperparameter, guaranteeing the intended mass allocation regardless of dataset size.
Compute $L_P = (1\cdot0.2 + 2\cdot0.5 + 1\cdot0.3)/(1+2+1) = 1.5/4 = 0.375$.
Compute $L_T = (0.4+0.6+0.5+0.7)/4 = 2.2/4 = 0.55$.
Derive $\lambda = \alpha/(1+\alpha) = 0.5/1.5 \approx 0.333$.
Blend: $L_{\text{replay}} = (1-0.333)\cdot0.375 + 0.333\cdot0.55 \approx 0.250 + 0.183 = 0.433$.
This toy calculation shows that despite the preservation family having three times as many raw examples, the final loss mass is split one‑third to trajectory and two‑thirds to preservation, exactly as prescribed by $\alpha$.
Cross-Interface Transfer
ChronoReplay boosts performance across OVO‑S, OST, and UCS task families.
ChronoReplay raises the average accuracy across OVO‑S, OST, and UCS to 56.86 %, surpassing ordinary token SFT (54.10 %) and natural turn weighting (55.57 %).
Table 2 reports mean accuracies of 56.86 % for ChronoReplay versus 54.10 % for ordinary token SFT and 55.57 % for natural turn weighting. +2.76 % over token SFT, +1.29 % over turn weighting
**Table.** Comparison of joint-training objectives. With the joint data, chronological prefixes, native targets, and training budget held fixed, ordinary token SFT obtains 51.8/60.7/49.8 on OVO-S/OST/UCS, and natural turn weighting obtains 52.9/62.3/51.5 (Table 9). ChronoReplay improves all three endpoints, with a mean of 56.86 versus 54.10 and 55.57. This comparison supports the balanced objective beyond joint data exposure under the tested setting. Its OST advantage is 3.23 points over ordinary token SFT and 1.63 over natural turn weighting; the corresponding OVO-S/UCS gains are 2.09/2.97 and 0.99/1.27. The advantage over natural turn weighting repeats on every benchmark in each of three paired seeds (Table 10). Mean paired gains are 0.92 ± 0.27 on OVO-S, 1.43 ± 0.19 on OST, and 1.17 ± 0.25 on UCS (points, mean ± sample standard deviation).
ChronoReplay consistently improves cross‑interface transfer, especially for tasks that require multi‑turn state accumulation.
Benchmark Results: OVO-S and OST
ChronoReplay yields sizable gains on OST‑Bench, surpassing base models across all metrics.
ChronoReplay improves OST‑Bench performance across all evaluated models, delivering an average gain of +13.4 % over the official base.
Table 2 shows $\Delta$ values ranging from +2.9 % to +18.3 % with a mean improvement of +13.4 %.
Benchmark Results: UCS-Bench
ChronoReplay yields consistent gains on UCS‑Bench, averaging +7.65% over baselines.
ChronoReplay improves UCS‑Bench performance by an average of +7.65% over official VLM baselines.
Table 3 reports the $\Delta$ values across all 8,114 timestamp‑causal questions, with the overall $\Delta$ of +7.65%.
Diagnostic Analysis
Ablation studies reveal how each component of ChronoReplay contributes to performance gains.
ChronoReplay preserves metric displacement information by jointly supervising categorical relations and ordered trajectory episodes. This dual‑supervision objective enables spatial VLMs to reason about trajectories.
Full ChronoReplay attains a three‑benchmark mean of 56.86, surpassing the single‑family CATEGORICAL‑ONLY and TRAJECTORY‑ONLY variants (means 50.74 and 50.76).
CATEGORICAL‑ONLY: 58.03 OVO‑S, 47.48 OST; TRAJECTORY‑ONLY: 52.03/51.03/49.22 OVO‑S/OST/UCS; ChronoReplay: 53.89/63.93/52.77 OVO‑S/OST/UCS.
ChronoReplay improves OST by 3.23 points over ordinary token SFT.
Ordinary token SFT OST = 60.7; ChronoReplay OST = 63.93.
Using native targets raises UCS by 2.92 points compared to the canonical wrapper.
Native UCS = 58.75; Canonical UCS = 55.83 (implied 2.92‑point gap).
ChronoReplay outperforms SHIFTED by 2.42 OST points.
SHIFTED OST lower by 2.42 points relative to ChronoReplay.
Setting $\alpha$ = 0.5 yields the highest OVO‑S–OST mean of 61.20, exceeding the $\alpha$ = 1.0 setting (60.92).
$\alpha$ = 0.5 mean = 61.20; $\alpha$ = 1.0 mean = 60.92.
Observation Access and Conclusion
We assess how observation access independently boosts performance beyond training.
We examine observation access as an orthogonal factor (RQ4), asking how much performance it adds beyond standard training gains.
Adaptive access lets the model query observations at test time with a learned policy, separate from the static training signal.
**Table 5.** Training–access decomposition on OVO-S (%). Levels average their three family accuracies; Overall averages all twelve. Summary rows separate training and access effects, using unrounded scores.
Adaptive access contributes an additional $+4.09$ points beyond the $+4.91$‑point training gain.
Table 5 shows the “Access effect” column where the Adaptive row exceeds the Uniform‑128 baseline by $+4.09$ points overall.
Theoretical Analysis
Formalizes the theoretical foundations underlying ChronoReplay.
This appendix formalizes the three arguments that motivate ChronoReplay (Sections 4.2.1, 4.2.2, 4.3). The results are stated at the population level with an unrestricted readout, describing what the objective demands of the representation.
For any representation $F_{\theta}$ and any interface $\kappa$, the population risk satisfies $$ R_{\kappa}(F_{\theta},G_{\kappa}) = H(Y_{\kappa}\mid Z,Q) = H(Y_{\kappa}\mid C,Q) + I(Y_{\kappa};C\mid Z,Q), \tag{7} $$ where the infimum is over all conditional distributions of $Y_{\kappa}$ given $(Z,Q)$. Moreover, for each query $q$ in the support of $Q$, $H(Y_{\kappa}\mid Z,Q=q)=0$ implies perfect prediction from $(Z,q)$ on that conditional support.
The expected log‑loss equals conditional entropy plus a KL term.
The KL term vanishes when $G_{\kappa}$ matches the true conditional.
Applying the chain rule yields the second equality.
Let $\tilde{Y}=g(Y_{\kappa})$ with $g$ invertible on the target support. Then for every representation, $$ H(\tilde{Y}\mid Z,Q)=H(Y_{\kappa}\mid Z,Q)\quad\text{and}\quad I(\tilde{Y};C\mid Z,Q)=I(Y_{\kappa};C\mid Z,Q). $$
Conditional entropy is invariant under a bijective deterministic transform.
Mutual information is also invariant.
Let $Y=A(C_{j},Q)$ be a deterministic native target. With an unrestricted readout, the population log‑loss floor under shifted conditioning is $$ \mathbb{E}\big[-\log G(Y\mid C'_{j},Q)\big] = H(Y\mid C'_{j},Q) = I(Y;O\mid C'_{j},Q) + H(Y\mid C'_{j},O,Q), \tag{8} $$ where $O$ is the identity of the episode’s first turn block. This floor exceeds $H(Y\mid C_{j},Q)=0$ whenever the shifted inputs induce at least two distinct targets with positive conditional probability.
Expected log‑loss equals conditional entropy plus KL divergence.
Vanishing KL gives the first equality.
Chain rule yields the decomposition.
Let $L_{P}$ and $L_{T}$ be the family risks of Eq. 5 and $$ L_{\text{replay}} = (1-\lambda)L_{P} + \lambda L_{T}, \qquad \lambda = \frac{\alpha}{1+\alpha}. $$ Then: 1. Replacing $D_{P}$ or $D_{T}$ by a family with the same within‑family empirical distribution but a different number of sequences or supervised turns leaves $\nabla_{\theta}L_{\text{replay}}$ unchanged; $\alpha$ controls the explicit ratio of the two family weights. 2. $L_{T}$ is invariant to whether supervised transitions of an episode are grouped into one sequence or presented as separate examples, provided each transition keeps its causal prefix. 3. Any common positive rescaling of all example weights (e.g., setting the mean weight to one) leaves the minimizers of $L_{\text{replay}}$ and the direction of its gradient unchanged.
Family risk depends only on the empirical distribution of $(\ell,w)$ within the family.
Grouping transitions does not alter the set of loss terms.
Uniform rescaling of example weights cancels in normalized risks.
Section B.1 introduces the attribution framework that separates model performance $J(\theta,\pi)$ into contributions from the trained model $\theta$ and the observation policy $\pi$.
Evaluation Metrics
We detail the evaluation protocol and the controlled contrasts used to isolate training and access effects.
This appendix formalizes the evaluation of ChronoReplay by separating the contribution of training from that of the adaptive observation policy.
The full system performance is measured as $J(\theta^{*},\pi_{1})$, while the two controlled contrasts above attribute improvements either to the training procedure or to the adaptive access mechanism.
We probe design choices with matched training interventions: the origin‑shift intervention keeps the multiset of turn blocks and supervision weights identical but permutes episode‑level chronology; the target‑interface experiment swaps the answer representation while preserving causal histories and training settings.
OVO‑S‑Bench evaluates only observations available up to the query boundary. Uniform access ($\pi_{0}$) uses 128 frames, while adaptive access ($\pi_{1}$) selects a causal set per query. Family accuracy is computed as $af = 100\,k_{f}/N_{f}$ and overall accuracy as $1_{f}=1/af$, averaging over the three families before rounding.
**Table 1.** The complete data contain 39,561 serialized sequences. The trajectory family consists of 6,381 episodes and 45,006 supervised states. An episode may also contain observation turns that establish the spatial context for later targets. These turns enter the causal prefix and receive no direct loss.
OST‑Bench replays conversations in their original turn order; the evaluation split contains 8 351 turns, of which 7 814 are scored by the official evaluator, covering agent state, visible information, spatial relations, judgement, temporal localization, counting, and estimation.
UCS‑Bench evaluates 8 114 questions across four categories—Trajectory & Movement, Position & Orientation, Proximity & Reachability, and Category & Quantity—each with two subcategories, yielding twelve displayed scores and four aggregated scores.
Published systems are compared using matched observation settings; post‑training results use fixed observation access, adaptive access results keep model parameters fixed, and component ablations use matched training interventions to isolate the effect of each design choice.
Replay Supervision Details
Details of the dual‑supervision replay construction, loss weighting, and scene separation.
ChronoReplay draws from two supervision families. The preservation pool holds 30,635 categorical and 2,545 numerical examples, while the trajectory pool supplies ordered episodes with metric, counting, judgement, and temporal targets.
At the balancing coefficient $\alpha$ = 0.5, loss mass is normalized so that each family contributes a total weight of 13,187. Consequently the mean serialized‑example weight equals one, but these coefficients do not imply equal gradient norms across families.
The causal prefix for turn j in episode i is defined as $c_{ij}$ = ($h_{ij}$; $q_{i1}$, $y_{i1}$, … , $q_{i,j-1}$, $y_{i,j-1}$) (Eq. 10). This prefix contains all observations and responses available up to that point, preserving object identity, observer pose, and reference frame.
The full replay dataset comprises 47,096 turns: 45,006 supervised states and 2,090 observation turns that provide context but incur no direct loss.
Iterate over each episode i and its turns j = 1…$m_i$, building the causal prefix $c_{ij}$ as defined above.
Record the native target $y_{ij}$ and its interface $\kappa_{ij}$ without alteration.
Collect all ($c_{ij}$, $q_{ij}$, $y_{ij}$, $\kappa_{ij}$) pairs into a dataset DT.
Compute the preservation loss $L_P$ on DT using the weighted pool $w_r$.
Set $\lambda$ = $\alpha$/(1 + $\alpha$) and minimize the combined objective (1 − $\lambda$) $L_P$ + $\lambda$ $L_T$ starting from the base VLM parameters $\theta$₀.
Return the trained parameters $\theta$.
Training and evaluation are separated by physical scene identity: a scene is defined by its scan regardless of frame rate, crop, or encoding. Our OVO‑S and UCS evaluations use scenes that never appear in the training set, and for OST we withhold 243 overlapping ScanNet scenes while keeping all question types.
Optimization and Training Settings
Training settings, loss normalization, and objective comparisons for ChronoReplay.
The experiments use the Qwen3‑VL‑8B‑Instruct model with its language backbone and visual merger fine‑tuned for a single epoch; Table 8 lists the common hyper‑parameters such as a learning rate of $2 \times 10^{-6}$, cosine decay with 3 % warmup, and a maximum sequence length of 32,768 tokens.
**Table 8.** Shared optimization settings for ChronoReplay and its matched controls.
Loss is scaled so that each answer token, each turn, and each family contributes proportionally, preventing longer episodes from dominating the training signal.
**Table 9.** Joint-training objective comparison (%). OVO-S uses Uniform-128; OST and UCS use the same fixed protocols as the main results. Mean is the unweighted average of the three benchmark scores.
**Table 10.** Three-seed replication of the objective comparison (%). Natural: natural turn weighting; Full: ChronoReplay; $\Delta$: Full minus Natural in percentage points. Pair indices enumerate runs. Differences and summaries use the displayed scores; summary values are rounded after aggregation.
Adaptive Access Details
We analyze how trajectory supervision weight, target formatting, and chronological order affect adaptive access performance.
In the family‑mass sweep we vary the relative weight $\alpha$ of trajectory supervision while keeping all other factors fixed, allowing us to isolate the impact of this trade‑off on adaptive access.
Chronological replay is ahead by 2.42 and 2.41 points in the two comparisons, a mean advantage of 2.41 points. Because the shift also changes the context preceding each target, the comparison tests chronological alignment as a whole rather than the episode origin alone.