Vision Pretraining for Dense Spatial Perception

Zelin Fu, Bin Tan, Changjiang Sun, Shaohui Liu, Kecheng Zheng, Yinghao Xu, Xing Zhu, Yujun Shen, Nan Xue

LingBot-Vision uses boundary-aware masked modeling to learn spatially structured visual representations from raw images.

How can we improve dense spatial perception in vision models by making image boundaries a first-class citizen of self-supervised pretraining?

Modern visual foundation models prioritize semantic recognition, often failing to recover the fine-grained spatial structure required for physical tasks like depth estimation or navigation. The authors introduce masked boundary modeling: a self-distillation framework that forces the model to reconstruct boundary-bearing tokens, turning geometric discontinuities into native learning signals. This approach allows a 1B-parameter model to match or outperform foundation models up to seven times larger on dense spatial perception tasks, while enabling a 0.3B student to match the depth accuracy of a 7B-parameter baseline.

Paper Primer

The core mechanism is a dual-supervision loop where the teacher network identifies boundary-bearing tokens and forces them into the student's masked set. By reparameterizing these boundaries into categorical distributions, the model treats boundary prediction as a stable classification problem, allowing geometric and semantic representations to co-emerge rather than compete.

Boundary-forced pretraining significantly improves dense spatial perception efficiency.

A 0.3B-parameter student model achieves NYU-Depth v2 accuracy comparable to a 7B-parameter DINOv3 model. ~23x reduction in parameter count for equivalent depth estimation performance.

Boundary modeling acts as a scalable pretraining principle that rivals larger foundation models.

LingBot-Vision (1B parameters) matches or surpasses models up to 7x larger on dense spatial tasks. Superior performance on 14 depth completion benchmarks compared to standard semantic-only pretraining.

Why does this approach use boundary-forced masking instead of standard random masking?

Random masking is content-agnostic and often hides redundant information, whereas boundary tokens are information-dense and cannot be inferred from local context, forcing the model to learn actual geometric structure.

How does the model avoid the "chicken-and-egg" problem of needing boundaries to learn boundaries?

The model uses a lightweight, frozen corner-point detector to seed initial boundary fields; these fields are over-parameterized and redundant, allowing coherent line segments to emerge and sharpen even from an untrained backbone.

Introduction: The Spatial Perception Gap

We expose why current vision models miss dense geometry and propose boundary‑centric masked modeling as a self‑supervised solution.

Dense spatial perception—recovering metric, structured representations from pixels—is crucial for physical intelligence. Yet modern visual foundation models prioritize semantic invariance, leaving fine‑grained geometry under‑represented.

Standard masked modeling treats images as bags of content, ignoring the geometric boundaries that define shape and depth.

**Figure 1.** LingBot-Vision learns dense representations via boundary-centric masked modeling. Each row, from left to right: the input image; the PCA projection of the frozen teacher's patch tokens; boundary tokens (pink) obtained by a-contrario validation of dense line proposals decoded from the model's own boundary-field prediction, overlaid on the accumulated response of the validated proposals; and cosine-similarity maps between nine boundary-token queries (red crosses, selected by farthest-point sampling in feature space) and all patch tokens. The learned representations carry both semantic grouping and geometric structures. Input images are at 1024px for the short side.

We therefore propose a boundary‑centric self‑supervised pretraining paradigm: masked boundary modeling forces teacher‑discovered boundary tokens into the student’s masked set, and a categorical reparameterization stabilizes the dense teacher–student loop. Scaling this framework yields LingBot‑Vision, a 1 B‑parameter Vision Transformer that matches or surpasses models up to seven times larger on dense spatial tasks.

Current vision models falter on dense spatial tasks because they lack native boundary signals during pretraining.

Related Work

Survey of self‑supervised vision pretraining and boundary‑centric modeling.

Self‑supervised learning (SSL) has become the dominant paradigm for pretraining visual models, progressing from contrastive objectives to masked reconstruction and joint‑embedding predictive architectures (JEPA).

Recent SSL work can be grouped into three families: self‑distillation (e.g., DINO, iBOT), masked image modeling, and language‑guided contrastive learning (e.g., CLIP, SigLIP 2).

The baseline trains a student network to mimic an exponential‑moving‑average teacher on either whole‑image tokens (DINO) or masked patches (iBOT), yielding representations that incidentally capture object boundaries.

Masked image modeling reconstructs pixel‑level content, guaranteeing spatial fidelity but yielding weaker semantic features when frozen.

Language‑supervised methods such as CLIP and SigLIP 2 excel at recognition because text describes whole images, yet they underperform on dense prediction tasks; patch‑text alignment narrows the gap but does not close it.

Predictive JEPA families (e.g., LeJEPA) avoid pixel‑level reconstruction by predicting latent representations, a design that raises the stakes for spatial quality when encoders are later used in world models or robot policies.

Boundary‑centric modeling has a long history, from classic edge detectors (Sobel, Canny) to modern deep approaches that learn edges or contours in a supervised or self‑supervised manner.

The a‑contrario theory underlies LSD, declaring a line meaningful only if it would be unlikely under a structureless null model, providing parameter‑free control of false detections.

The attraction‑field line representation encodes each line segment densely: every nearby pixel stores enough geometry to reconstruct the whole segment, turning sparse detection into a dense prediction field.

Boundary-centric Masked Representation Modeling

Boundary-centric masking forces the model to reconstruct structural edges, coupling semantic and geometric learning.

Random masking treats every patch as equally informative, but interior patches are trivially predictable while boundary patches encode the scene’s geometry.

Instead of hiding patches at random, we force the model to hide exactly those patches that intersect predicted image boundaries, then ask it to reconstruct both their semantic content and their geometry.

Run the teacher ViT on a global view to obtain a dense boundary field.

Rasterize the field to a pixel‑level map and max‑pool onto the $P\!\times\!P$ token grid, producing the boundary token set $B$.

Sample a random mask $M$ with target ratio $r$ (block‑wise uniform).

Form the forced mask $M^{+}=M\cup B$.

Feed the masked view to the student; compute iBOT loss on all tokens in $M^{+}$ and boundary loss on tokens in $B$.

Boundary‑forcing mask construction (pseudocode)

Compute $M^{+}=M\cup B=\{3,5,6,7,12\}$.

The student receives a view where those five patches are replaced by a shared mask token.

iBOT loss is applied to all five masked patches, encouraging reconstruction of their visual content.

Boundary loss $L_{\text{bnd}}$ is additionally applied to patches 5 and 6, forcing the student to predict the discretized distance‑orientation tuple for each.

After back‑propagation the student updates; the teacher EMA incorporates the new $\theta$.

Masking only the boundary patches removes the only information that cannot be inferred from neighboring context, making the reconstruction task genuinely informative.

**Figure 2.** Boundary-centric masked modeling on a toy scene. (a) An input image and its patch grid. (b) Random masking is content-agnostic: most hidden patches are flat and recoverable from context, while the boundary-bearing patches largely escape the mask. (c) Boundary-forcing masking adds every boundary-bearing patch to the random mask, so the structure of the scene is exactly what the student must reconstruct. (d) The boundary field that supervises boundary tokens, encoded as per-pixel categorical distributions over discretized distance and orientation bins. During pretraining the boundary field is predicted online by the teacher itself, without human labels or external detectors; every masked token follows the semantic self-distillation objective, and boundary tokens additionally match the categorical boundary target.

**Figure 3.** Boundaries emerge from corner points (Finding 1). For two images, we draw five random boundary fields per image, fix the corner points, and decode line segments from each draw (columns). In the bottom two rows, every field channel is sampled uniformly at random: the decoded segments still anchor on the corner points, but stay short and fragmented. In the top two rows, only the direction channel is guided by a parameter-free estimate of the image's level-line orientation [47], rendered into a boundary field, and every draw yields coherent, near-identical segments. In either case the fields carry no learned information. Sampling and decoding details are in Appendix A.

**Figure 4. The boundary field at a glance.** (a) Image boundaries are modeled as straight line segments; curved boundaries are chains of short segments. (b) The boundary field lifts the sparse segments into a dense map: every pixel near a segment stores its distance $d$ to the segment and three angles ($\theta, \phi^1, \phi^2$) that locate the segment from that pixel; parallel segments share the same orientation color in $\theta$. (c) The encoding is deliberately redundant: any single pixel $p$ in a segment's support region carries enough information to reconstruct the entire segment. (d) Conversely, segments are decoded by letting all support pixels vote and aggregating the votes (orange strokes: individual noisy votes; dark lines: aggregated segments), which remains reliable even when individual values are noisy.

**Figure 5.** Online generation of boundary targets (toy example). Even when the teacher's boundary field is noisy, as it is early in training (b), the *holistic* parameterization lets the many weak per-pixel votes in a segment's support region aggregate into coherent candidate segments anchored at corner points (c), together with spurious candidates such as face diagonals and background chords. The a-contrario test then discards unsupported candidates, and the survivors are re-rendered into the clean target field that supervises the student (d). Boundary targets thus *emerge* from raw images, with only a frozen single-block corner-point detector fixed (Sec. 3.4).

**Table 1.** Proof of concept on ImageNet-1K (ViT-L/16). All variants share the training configuration of the DINO+iBOT baseline.

Scaling LingBot-Vision

Scaling the boundary‑centric pretraining to a billion‑parameter vision model while keeping overhead tiny.

Scaling boundary‑centric masked modeling to a foundation‑scale vision backbone raises two practical pains: the data‑curation pipeline must stay tractable, and the extra boundary head must not dominate compute.

LingBot‑Vision is a ViT‑g/16 backbone (≈ 1.1 B parameters) augmented with a tiny boundary head that predicts dense K‑bin fields only on a handful of boundary tokens per image.

How does LingBot‑Vision differ from a vanilla ViT‑g pretrained with only self‑distillation?

Besides the standard semantic head, LingBot‑Vision carries a dedicated boundary head that learns explicit edge‑field predictions on a sparse set of tokens. This forces the model to encode geometric boundaries, which vanilla self‑distillation never sees.

Step 1: The two selected tokens are each split into $2\times2$ sub‑tokens because $s=2$, yielding 8 sub‑tokens.

Step 2: For each sub‑token the MLP outputs a 4‑bin distribution; the soft‑label temperature sharpens the distribution.

Step 3: The fused kernel computes cross‑entropy against the online target in a single pass, producing a scalar loss per sub‑token.

Step 4: Gradients from these 8 losses back‑propagate only through the boundary head, leaving the remaining 8 untouched.

Step 5: The total FLOPs contributed by the boundary head are $8 \times \text{MLP\_cost}$, far smaller than the $16 \times \text{self-attention\_cost}$ of the backbone.

The head’s cost scales linearly with the number of boundary tokens, not with the full image resolution, which is why it stays cheap even on high‑resolution inputs.

Our data‑curation pipeline mirrors DINOv2: we start from a 2 B‑image raw pool, use public datasets as seeds, retrieve visually similar images, deduplicate, and finally shuffle uniformly.

Distributed training uses FSDP sharding with bf16 mixed precision and memory‑efficient attention; these choices are generic and not specific to our method.

Online target generation fuses corner‑line pairing and the a‑contrario test into batched CUDA kernels, eliminating host‑CPU round‑trips and keeping the per‑step overhead negligible.

Training proceeds in three stages: 300 k iterations of self‑distillation, 100 k iterations of Gram‑anchoring, and a final 100 k high‑resolution adaptation at 512 px.

After the flagship model is trained, we distill it into 300 M, 86 M, and 21 M student backbones using the same 300 k‑iteration schedule (no Gram‑anchoring) and a 100 k high‑resolution fine‑tune.

Evaluation on Dense Tasks

Boundary‑centric features give LingBot‑Vision top depth and segmentation accuracy on dense tasks.

LingBot‑Vision achieves the best NYUv2 depth $RMSE$ of 0.296, beating the 7 B‑parameter DINOv3 (0.309) and all smaller models.

Table 2 shows these values across all competitors.

**Figure 6. PCA of frozen patch features.** The top three PCA components of the patch features are mapped to RGB, computed per image for each model (one model per column). LingBot-Vision (rightmost) resolves objects as coherent regions with crisp boundaries: individual cars and lane structure in the traffic scene, hen silhouettes against the wire fence, the winding contour of the snake, and fine structures such as flower stalks and branches. In comparison, DINOv2 exhibits per-token speckle, SigLIP 2 degrades into blocky noise, and V-JEPA 2.1 lets background texture bleed into the foreground regions.

These dense‑task results set the stage for the video‑understanding experiments that follow.

Video Understanding and Scaling

Video segmentation and image classification results showcase LingBot‑Vision’s balanced performance.

LingBot‑Vision matches top video‑segmentation models, beating DINOv2 by 6.1 J&F on DAVIS and 7.9 J&F on YouTube‑VOS.

Table 3 reports 70.0 / 73.5 for LingBot‑Vision versus DINOv2’s 63.9 / 65.6 (derived).

On ImageNet, LingBot‑Vision attains 86.32 % linear top‑1 and 83.39 % k‑NN, within 1 % of DINOv2 while preserving dense‑task superiority.

Table 5 shows 86.32 / 83.39 versus DINOv2’s 87.00 / 83.68.

Distilled LingBot‑Vision students retain the teacher’s dense advantage: the ViT‑L student reduces NYU‑v2 RMSE by 25 % (0.310 vs 0.411) while staying within 0.05 % of the teacher on classification.

Table 4 reports 0.310 RMSE for LingBot‑Vision L versus 0.411 for DINOv2.

**Table 3.** Video object segmentation with frozen features. We report the $J\&F$-Mean on DAVIS-2017 and YouTube-VOS under training-free label propagation, with the shorter image side set to 480 px for models with patch size 16 and 420 px for those with patch size 14.

**Figure 7. Boundary-token tracking on three videos.** Three boundary-token queries are seeded in the first frame of each video (crosses in the left panels) and tracked by cosine similarity of the frozen patch features over each frame’s boundary tokens; the three heatmap panels show the similarity responses of the three queries. Top pair: a manipulation episode seen from a robot’s head camera, where the queries hold the gripper fingers and the pitcher throughout the episode. Middle pair: a self-captured cat video, where the queries on the head and flank follow a full turn of the body while the query on the window frame stays put. Bottom pair: a camera pan through a home, where the queries track the cat and the cat-tree platform through a large viewpoint change and re-acquire the plant as it re-enters the view. No fine-tuning or temporal supervision is used.

**Table 5.** Comparison of LingBot-Vision distilled models with representative pretrained vision backbones. Global tasks report ImageNet-1k linear probing and KNN top-1 accuracy, while dense tasks report depth RMSE↓, and semantic segmentation mIoU. We match the patch token resolution of models with different patch size.

**Table 5.** Comparison of distilled models across different scales (L, B, S) on global and dense tasks.

LingBot-Depth 2.0

LingBot‑Depth 2.0 delivers a 50 % RMSE cut on the hardest block‑masked benchmark.

LingBot‑Depth 2.0 halves $RMSE$ on block‑masked DIODE‑Indoor compared to LingBot‑Depth 1.0.

Table 8 reports $0.132$ $RMSE$ for the 1.0 version and $0.062$ $RMSE$ for the 2.0 version.

**Figure 9. LingBot-Depth 2.0 on mirror and glass scenes.** Each group shows the input RGB, the raw sensor depth and the refined depth for consecutive frames of a captured sequence, together with front and top views of the refined point cloud. The raw depth is missing exactly on the hardest surfaces: window panes, a glass balustrade and reflective floors return no measurements. The completed regions form flat, contiguous planes in the point clouds and remain stable over time.

**Figure 10.** Qualitative comparison of depth completion methods. Each scene shows the completed depth map (top) and the corresponding point cloud (bottom) for OMNI-DC, CDMs, LingBot-Depth 1.0 and LingBot-Depth 2.0, next to the RGB frame with ground-truth depth and the raw sensor input. The baselines leave missing regions open, bend planar surfaces, or scatter floating points near depth discontinuities; LingBot-Depth 2.0 keeps planes straight and object boundaries sharp.

Ablations and Conclusion

Key ablations show how corner anchors and a‑contrario validation each affect line‑segment decoding.

We isolate each component of the boundary‑centric pipeline to measure its contribution to reliable line‑segment decoding.

**Figure 11. Corner anchors and a-contrario validation are complementary safeguards.** The same boundary field, decoded from the same LingBot-Vision teacher under identical settings, with corner points taken either from the frozen single-block detector (a, b; 920 candidate segments) or derived from the field itself via $jloc = 1 - d$ (c, d; 2,382 candidates). With reliable corner anchors, the vote-aggregation decoding is already clean before any validation (a), and the a-contrario test removes only minor spurious residues (b; 673 segments kept). With weak, self-derived anchors, the unvalidated decoding degenerates into dense spurious structure (c), yet the a-contrario test recovers almost identical line segments (d; 675 kept, 72% rejected). Either safeguard alone suffices; only when both are removed does decoding fail. Together with Finding 1, this explains why boundary targets can be bootstrapped robustly from scratch.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers