Scal3R: Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction

Chin-Yang Lin, Yang-Che Sun, Cheng Sun, Fu-En Yang, Min-Hung Chen, Yen-Yu Lin, Wei-Chen Chiu, Yu-Lun Liu

Scal3R replaces unstable global pose regression with multi-reference relative querying on frozen 3D backbones.

How can we perform stable, long-sequence 3D reconstruction without retraining the backbone or suffering from the drift inherent in absolute pose regression?

Online 3D reconstruction models typically regress camera poses relative to the first frame, causing small feature drifts to amplify into catastrophic geometric collapse over long sequences. Scal3R reformulates this as a multi-reference relative pose query task, injecting lightweight learnable tokens into a frozen backbone via asymmetric attention to predict relative transformations between the current frame and multiple past keyframes. This approach reduces average trajectory error by over 60% on KITTI and enables globally consistent reconstruction on kilometer-scale sequences with only 8 hours of training on a single GPU.

Paper Primer

The core move is to decouple local geometry from global pose estimation. By using asymmetric attention, the system allows pose query tokens to extract geometric relationships from the frozen backbone's features without modifying the backbone's internal representation space, ensuring point-map quality remains intact.

The system functions like a mail sorter: the pose tokens act as queries that read the "address" (geometric relationship) of the current frame relative to multiple stored reference frames, which are then aggregated by an online pose-graph optimization (PGO) module to suppress drift.

Scal3R significantly improves long-sequence trajectory accuracy compared to global-anchor baselines.

Average Absolute Trajectory Error (ATE) on the KITTI dataset. Reduction of over 60% compared to the online baseline.

The system maintains high throughput despite adding pose-graph optimization and loop closure.

Inference speed on KITTI sequences. Retains >90% of the original backbone's throughput (14.4 FPS vs 15.9 FPS).

Why does the paper use asymmetric attention instead of standard prompt tuning?

Symmetric attention allows pose tokens to influence image features, which dilutes geometric cues and degrades point-map quality. Asymmetric injection restricts information flow so pose tokens only read from image features, preserving the frozen backbone's representation.

Does this method require retraining the entire 3D reconstruction model?

No. Scal3R keeps the backbone completely frozen and only trains a set of lightweight tokens that account for approximately 1% of the total parameter count.

The Problem of Absolute Pose Drift

Scal3R tackles drift by replacing absolute pose regression with multi‑reference relative queries.

Online 3D reconstruction models that regress absolute poses from a fixed first‑frame anchor quickly drift on long video streams because the pose head must extrapolate far beyond its training distribution, while the frozen backbone still produces reliable per‑frame depth.

**Fig. 1: Scal3R enables scalable online 3D reconstruction on long video streams.** (a) Existing feed-forward models, such as CUT3R [82] and SStream3R [40], regress absolute global poses ($P_t$) relative to a fixed first frame. This forces extrapolation far beyond its training distribution, resulting in catastrophic drift and geometric collapse on kilometer-scale sequences. (b) Scal3R reformulates the problem into a local multi-reference relative pose query ($\hat{T}_{t \leftarrow r_k}$). By injecting lightweight learnable tokens into a frozen backbone and aggregating relative constraints via online Pose-Graph Optimization (PGO), Scal3R suppresses long-range drift and recovers globally consistent geometry. It requires only 8 hours of fine-tuning on a single GPU.

**Fig. 2:** Global pose regression fails under out-of-distribution sequences, while local geometry remains reliable. (a) Feed-forward models like CUT3R [82] produce accurate reconstructions for in-distribution frames (blue). However, they suffer from severe geometric collapse when extrapolating to unseen long-range trajectories (orange). (b) Our error correlation analysis reveals a critical decoupling. The global position error diverges catastrophically in the out-of-distribution region (red), while per-frame depth remains stable (blue). This finding suggests that the backbone's local geometric representations are intact. This motivates Scal3R to freeze the base model and replace fragile global regression with stable multi-reference relative pose querying.

Absolute pose regression collapses because the pose head is forced to extrapolate beyond its training distribution, whereas the frozen backbone’s local geometry remains reliable.

Prior Approaches to 3D Reconstruction

We survey prior 3D reconstruction approaches and highlight streaming pose methods.

Early pipelines relied on offline Structure-from-Motion and Multi-View Stereo, while recent feed‑forward models predict geometry in a single pass. Online systems such as recurrent TSDF fusion and differentiable bundle adjustment process video frame‑by‑frame, but most still regress poses relative to the first frame, leading to drift. Long‑sequence streaming approaches address kilometer‑scale drift via test‑time updates, memory tricks, and token pools. Prompt‑tuning techniques adapt frozen 3D backbones with minimal parameters, a trend extended to reconstruction.

CUT3R equips a frozen 3D foundation model with a persistent state that updates causally as new frames arrive, enabling continuous reconstruction without retraining.

STream3R extends a frozen 3D transformer with a streaming module that incrementally refines geometry and pose as each frame is processed.

LongStream tackles kilometer‑scale streaming by decoupling short‑term token pools from a long‑term spatial memory, preserving global consistency over very long sequences.

First work to apply prompt tuning on a frozen CUT3R model for joint human‑scene reconstruction, demonstrating that lightweight adapters can control both geometry and pose.

Asymmetric Attention Injection

We inject pose queries asymmetrically into a frozen backbone, enabling multi‑reference pose prediction with minimal parameters.

Adding new tokens to a frozen decoder normally perturbs the attention distribution of image tokens, degrading point‑map reconstruction. To avoid this, we inject pose queries asymmetrically so they influence image features without being influenced themselves.

Pose query tokens drop their “letters” into a one‑way mailbox of image tokens: they can read image features, but image tokens never read back, so the frozen image representation stays unchanged.

How does this differ from standard self‑attention where all tokens attend to each other?

In standard self‑attention every token is both a query and a key/value, so inserting pose tokens would alter the keys/values of image tokens and change their representation. Our asymmetric injection restricts pose tokens to query‑only, leaving image keys/values untouched and preserving the frozen backbone.

Asymmetric attention injection – one decoder layer.

We keep a small buffer of past camera tokens and combine each with a shared base query $q$ via a lightweight MLP, producing a distinct query for every reference frame.

Compute $\text{MLP}(c_{r1}) = [0.5, 0.1, 0.0, 0.2]$.

Add to the base query: $\tilde{q}_1 = q + \text{MLP}(c_{r1}) = [0.6, 0.3, 0.3, 0.6]$.

The resulting pose token $\tilde{q}_1$ now carries both the shared query information and the specific pose hint from the reference frame.

Even with a trivial MLP, the injection adds the reference‑specific signal while keeping the dimensionality and parameter count tiny.

Why can we increase the number of reference frames at inference without retraining?

Each new reference simply creates another pose token $\tilde{q}_k$ using the same base $q$ and the stored camera token; no new weights are learned, so the model can handle any $K$ on the fly.

Enforcing Global Consistency

Online Pose‑Graph Optimization refines camera poses incrementally to suppress drift.

Online Pose‑Graph Optimization (PGO) continuously refines a graph of camera poses as new frames stream in, preventing the drift that plagues pure absolute‑pose regression.

PGO treats each incoming frame as a node in a graph and repeatedly solves a small optimization problem that aligns the node with its recent neighbours and any detected loop closures, keeping the whole trajectory globally consistent.

How does this online PGO differ from traditional bundle adjustment?

Traditional bundle adjustment optimizes all poses in a batch after the entire trajectory is collected, incurring $O(N^3)$ cost. Online PGO, by contrast, maintains a sliding window of recent nodes and solves a small sparse problem at each step, yielding $O(1)$ per‑frame cost while still enforcing global consistency through loop‑closure edges.

Step 1: $F_0$ becomes the first node; no edges exist yet.

Step 2: $F_1$ arrives → add node $F_1$ and a sequential edge $F_0\!arrow\!F_1$.

Step 3: $F_2$ arrives → add node $F_2$, sequential edge $F_1\!arrow\!F_2$, and multi‑reference edges $F_2\!arrow\!F_1$, $F_2\!arrow\!F_0$ (since $K=2$).

Step 4: $F_3$ arrives → add node $F_3$, sequential edge $F_2\!arrow\!F_3$, multi‑reference edges $F_3\!arrow\!F_2$, $F_3\!arrow\!F_1$, and a loop‑closure edge $F_3\!arrow\!F_0$ (detected by visual overlap).

After each addition, a small least‑squares solve adjusts the poses of the active nodes to satisfy all edge constraints.

The sliding‑window graph ensures that each new pose is anchored to multiple recent frames, so a single noisy measurement cannot cause unbounded drift.

**Fig. 5: Inference pipeline and pose graph structure.** To suppress accumulative drift in long sequences, incoming frames are registered via keyframe selection and pose-graph optimization over sequential, multi-reference ($K=3$), and loop closure edges before being committed to the pose token buffer.

Performance Benchmarks

Scal3R dramatically lowers pose drift on long‑sequence benchmarks.

Recall that Scal3R queries poses relative to recent frames, avoiding the drift that plagues absolute‑pose regression.

Scal3R achieves an average ATE of 2109 on the KITTI benchmark, outperforming all prior online methods.

Table 1 lists per‑sequence ATE values; every entry for our variants is lower than the best competing online baseline.

**Fig. 6:** Qualitative 3D reconstruction on Virtual KITTI. Comparison against CUT3R and STream3R on Scene 01 (332 m) and Scene 02 (113 m). While both baselines produce distorted or collapsed point clouds, Scal3R recovers scene geometry closely matching the ground truth across both sequences.

**Fig. 7:** Qualitative trajectory comparison on KITTI long sequences. We visualize estimated camera trajectories on Seq. 00 and Seq. 05 against CUT3R, WinT3R, and SStream3R. All baselines suffer from catastrophic drift, while Scal3R faithfully recovers the full loop structure with metric accuracy.

**Fig. 8: Per-frame runtime breakdown on KITTI.** On both the CUT3R (left) and SStream3R (right) backbones, the frozen forward pass dominates latency; keyframe selection, PGO, and loop detection together add only a small fraction.

**Fig. 9: Robustness Analysis on KITTI.** Scal3R maintains the lowest error across the full evaluation range, while competing methods suffer catastrophic divergence at moderate sequence lengths.

**Fig. 10: Qualitative effect of loop closure on KITTI 07.** Without loop closure, accumulated drift causes the trajectory to deviate from the ground-truth loop structure. With loop closure, Scal3R recovers a globally consistent trajectory.

**Fig. 11: More qualitative comparison on vKITTI.** Scal3R produces globally consistent reconstructions with minimal trajectory drift, while baselines exhibit elongated or distorted point clouds.

**Fig. 12: More qualitative comparison on TUM-Dynamic.** Scal3R produces cleaner reconstructions with fewer ghosting artifacts from dynamic pedestrians on both backbones.

Scal3R’s relative‑pose querying yields up to 30 % lower drift on long‑sequence benchmarks, enabling reliable online 3‑D reconstruction.

Ablation Studies and Visualizations

Ablation experiments quantify the impact of each system component on pose accuracy and runtime.

We now isolate each design choice to see how it contributes to the overall drift‑free reconstruction.

Scal3R improves geometric consistency over the frozen backbones.

On 7‑Scenes, the Normal Consistency (NC) mean reaches 0.579 and median 0.622, surpassing both CUT3R and STream3R.

Removing reference‑frame supervision harms pose quality.

When reference supervision is omitted, the translational relative pose error (RPEtrans) spikes to 3.336 m.

Keyframe selection is essential for accurate pose estimation.

Without keyframe selection the Absolute Trajectory Error (ATE) climbs to 38.258 m, far above the full system’s 5.632 m.

Pose‑Graph Optimization (PGO) contributes a sizable accuracy boost.

Disabling PGO raises ATE to 15.748 m compared with 5.632 m when PGO is active.

Increasing the number of reference frames at inference time steadily reduces error.

Using K = 4 yields ATE = 7.362 m, while the full configuration K = 12 achieves ATE = 5.632 m.

Loop closure dramatically improves global consistency.

On KITTI, enabling loop closure cuts average ATE from 143.45 m to 75.01 m—a 48 % reduction.

System‑level components add negligible overhead.

Model forward accounts for 86.3 % of runtime; the remaining components together contribute under 10 %.

**Fig. 13:** Ablation visualizations on vKITTI. Removing keyframe selection or PGO each leads to distinct trajectory degradation.

**Fig. 14:** Ablation visualizations on KITTI. Disabling loop closure leaves visible gaps at revisited regions; the full system produces a globally consistent reconstruction.

**Fig. 15: Scal3R (CUT3R) runtime breakdown on KITTI.** The model forward pass dominates latency in both configurations; all system-level components together add less than 10% overhead.

**Fig. 16: Scal3R (STream3R) runtime breakdown on KITTI.** The latency profile closely mirrors the CUT3R variant; system-level components remain lightweight across both backbones.

**Fig. 18: Attention maps of relative pose query tokens on TUM-Dynamic.** The pose query tokens attend primarily to static structures while implicitly down-weighting dynamic pedestrians, without any explicit motion segmentation.

Summary and Future Directions

We conclude with Scal3R’s results, discuss its limits, and note funding support.

Scal3R reformulates camera localization as multi‑reference relative pose querying on a frozen backbone, using lightweight tokens that comprise about 1 % of the model parameters to query relative poses. Trained in eight hours on a single GPU, it delivers accurate online reconstruction of long video streams.

Limitations arise from the frozen backbone, which degrades under occlusion or textureless regions, and from the online backend, whose appearance‑based loop closure can miss revisits under extreme viewpoint or illumination changes. Moreover, keyframe selection and loop detection depend on hand‑set thresholds, leaving room for future improvement.

We thank NVIDIA Taiwan AI Research & Development Center (TRDC) for support and acknowledge funding from the National Science and Technology Council, Taiwan (grants NSTC 112‑2222‑E‑A49‑004‑MY2, 113‑2628‑E‑A49‑023‑, 115‑2628‑E‑A49‑024‑, 111‑2628‑E‑A49‑018‑MY4). Yu‑Lun Liu also thanks the Yushan Young Fellow Program by the MOE in Taiwan, and the paper is authored by C.-Y. Lin et al.

Multi-Reference Relative Pose Querying

Scal3R replaces absolute pose regression with multi‑reference relative queries to stay stable online.

Online 3D reconstruction with a frozen backbone suffers from drift because the model must regress absolute poses relative to the first frame, and errors accumulate as the sequence grows.

Instead of asking “where am I in a distant global map?”, the system asks “where am I relative to several nearby reference frames” – like navigating by nearby landmarks rather than a far‑away city center.

Query $c_{r_1}$: the MLP produces a 0.5 m shift along the X‑axis.

Query $c_{r_2}$: the MLP produces a 0.4 m shift along the Y‑axis.

Query $c_{r_3}$: the MLP produces a –0.3 m shift along the X‑axis.

Fuse the three relative poses (e.g., average) → final estimate $(0.07,0.13,0)$ m.

Aggregating several short‑range relative poses yields a stable estimate without ever needing a global coordinate origin.

How does this differ from standard absolute pose regression?

Absolute regression predicts a single pose in a fixed world frame, requiring the network to extrapolate far beyond its training distribution as the sequence grows. The multi‑reference query instead predicts several short‑range relative transforms to nearby frames, keeping each prediction within a familiar local context and preventing drift.

Scal3R builds on frozen pretrained backbones such as CUT3R’s persistent‑state model and STream3R’s causal‑Transformer model. Both keep their image feature tokens $F_t$ and camera token $c_t$, but we discard their original global‑pose heads and inject our lightweight query tokens instead.

**Fig. 3: Overview of the Scal3R framework.** For an incoming frame `Image_t`, a frozen encoder extracts dense image tokens. At the same time, historical camera tokens from selected reference frames ($r_k$, $r_{k-1}$, ...) are projected using lightweight trainable MLPs to generate relative pose tokens. These tokens are concatenated and sent into a completely frozen 3D reconstruction decoder (e.g., CUT3R [82] or SStream3R [40]). Our Asymmetric Attention Injection mechanism is crucial as it ensures that relative pose tokens serve only as queries to extract geometric cues, while image tokens perform self-attention exclusively among themselves. This method preserves the original high-quality point cloud generation ($X_t$) through the frozen Point Head, while the trainable Relative Pose Head predicts robust multi-reference relative transformations ($T_{t \leftarrow r_k}$). Finally, an online inference backend (PGO and loop closure) aggregates these local constraints to produce a globally consistent trajectory.

The shift from global to relative coordinate frames eliminates long‑range extrapolation, enabling drift‑free online reconstruction.

Implementation Details and Extended Analysis

This section details implementation, extra experiments, and performance analyses supporting Scal3R.

The supplementary material expands on the core paper by providing full implementation details, extra qualitative results, and extensive performance analyses.

Section A describes how Scal3R is built on frozen CUT3R and STream3R backbones, the training regime, and the pose‑graph optimization (PGO) setup.

Section A.1 presents Algorithm 1, the per‑frame inference pipeline: reference selection, loop‑closure detection, asymmetric attention injection, and incremental iSAM2 optimization.

Section A.2 explains the evaluation protocol, including Sim(3) alignment for metric‑scale recovery and the metric‑scale variant where the scale factor is fixed to 1.

Section A.3 lists the PGO hyper‑parameters: keyframe selection thresholds, gap‑dependent noise model, robust Huber kernel, and loop‑closure filtering criteria.

Section B adds visual evidence, showing that CUT3R and STream3R drift on long vKITTI/KITTI sequences while Scal3R maintains compact, globally consistent reconstructions.

Section B.1 provides qualitative comparisons on vKITTI, KITTI, and TUM‑Dynamic, highlighting Scal3R’s ability to suppress drift and handle dynamic occlusions.

Section B.2 visualizes ablations, demonstrating that removing keyframe selection, PGO, or loop closure each degrades trajectory quality.

Section C reports runtime and memory profiling, confirming that the frozen backbone dominates latency while auxiliary components add modest overhead.

Section C.1 shows per‑frame latency on KITTI, with the forward pass accounting for ~86‑90 % of total time and loop closure incurring only a 2.4 ms penalty.

Section C.2 studies scalability with reference count K, revealing that K = 12 balances accuracy (ATE ≈ 5.6) and throughput (~15 FPS) before performance degrades at larger K.

Section D contains further experiments that probe scale handling, baseline comparisons, and robustness.

Section D.1 evaluates metric‑scale pose estimation, showing that multi‑reference querying improves both relative pose accuracy and global scale consistency.

Section D.2 compares Scal3R against classic SLAM pipelines on KITTI, demonstrating competitive ATE without requiring camera intrinsics.

Section D.3 contrasts Scal3R with LongStream, highlighting that Scal3R achieves a comparable regime with a frozen backbone and ~1 % trainable parameters versus LongStream’s billion‑parameter retraining.

Section D.4 ablates asymmetric versus symmetric attention injection, confirming that the asymmetric design dramatically reduces outdoor ATE.

Section D.5 demonstrates that zero‑shot test‑time training (TTT3R) consistently lowers ATE when applied on top of Scal3R.

Section D.6 analyzes attention maps on TUM‑Dynamic, revealing that pose query tokens naturally down‑weight moving objects, providing implicit robustness to dynamics.

The table compares two methods, CUT3R and Scal3R, across different datasets (Sintel, TUM, ScanNet, vKITTI, KITTI) with and without SA (Self-Attention/Spatial Attention). The values represent performance metrics, with bold text indicating the best results for Scal3R.

The table compares two methods, LongStream [16] and Scal3R (CUT3R), across several metrics including pose formulation, drift mitigation, KITTI ATE, backbone characteristics, training data/views, and compute requirements.

**Table 11:** Comparison with the concurrent LongStream [16] on KITTI (ATE $\downarrow$). LongStream attains lower absolute ATE by retraining a 1.3B backbone on large-scale data, while Scal3R reaches a comparable regime by adapting a frozen backbone with ~1% parameters and yields pairwise constraints that LongStream's single-reference design cannot provide.

**Table 12.** Asymmetric vs. Symmetric attention injection (ATE $\downarrow$). Asymmetric injection is critical for outdoor sequences, reducing ATE by up to 10$\times$.

**Table 13.** Compatibility with zero-shot test-time training (ATE $\downarrow$). TTT3R consistently improves ATE when applied on top of Scal3R's globally optimized poses.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers