REBASE: Reference-Background Subspace Elimination for Training-Free In-Context Segmentation

Mantha Sai Gopal, Jaison Saji Chacko, Harsh Nandwana, Sandesh Hegde, Debarshi Banerjee, Uma Mahesh

REBASE improves training-free segmentation by projecting features onto the orthogonal complement of the reference image's background subspace.

How can we improve one-shot in-context segmentation without training by removing background-related noise from feature representations?

In-context segmentation models often struggle because shared background elements—like grass in a sheep image or skin texture in a lesion image—create spurious similarities that bias the model toward non-target regions. REBASE solves this by identifying the low-rank background subspace from the reference image and projecting both reference and query features onto its orthogonal complement, effectively filtering out shared contextual noise. This training-free approach establishes a new state of the art on four benchmarks, including ISIC and PACO-Part, without requiring any parameter updates or fine-tuning.

Paper Primer

The core mechanism is an episode-level orthogonal projection: the system isolates the dominant background directions in the reference image using singular value decomposition and removes these directions from the feature representations of both the reference and query images. This is like a noise-canceling filter for visual features: it identifies the "background frequency" of the current scene and subtracts it, leaving only the unique, foreground-relevant signal.

REBASE achieves state-of-the-art performance among training-free methods on four out of five evaluated benchmarks.

mIoU improvements on ISIC (63.8%), Chest X-Ray (86.3%), FSS-1000 (88.2%), and PACO-Part (39.3%). Outperforms previous training-free methods like INSID3 by up to 9.4 percentage points on medical datasets.

Why is this method specifically effective for medical and part-level segmentation?

These domains often feature highly structured, repetitive backgrounds—such as homogeneous skin or anatomical structures—that dominate the similarity map in standard models; REBASE explicitly suppresses these shared contextual modes.

How does this differ from existing positional debiasing techniques?

Positional debiasing removes global spatial artifacts estimated from synthetic noise, whereas REBASE is reference-conditioned, dynamically identifying and removing scene-specific semantic context per episode.

Researchers can now improve one-shot segmentation performance by applying per-episode subspace projection to frozen foundation model features, bypassing the need for costly fine-tuning or dataset-wide calibration.

Introduction

Training‑free one‑shot segmentation removes background bias without any retraining.

Deployed vision systems rarely see a closed set of categories; new products, rare medical structures, or domain‑specific parts appear after deployment. Updating a model with class‑incremental learning demands a full fine‑tuning pass, extra memory for replay data, and careful hyper‑parameter tuning for each increment.

Few‑shot segmentation sidesteps these costs: a single annotated reference image defines the target class, and the model must segment the target in query images without any parameter updates. This “training‑free” regime aligns with the practical needs of continual deployment.

Recent pipelines combine self‑supervised vision transformers (e.g., DINOv2) for dense semantic correspondence with promptable decoders such as SAM. The typical recipe extracts features, builds a cosine‑similarity map between reference foreground and query patches, and feeds the highest‑scoring locations as point prompts to SAM.

Because reference and query images share background context (e.g., sky, floor, anatomical structures), the similarity map is systematically elevated in non‑target regions. This background bias misleads both the point prompts and the dense mask prior, especially for small or articulated objects.

Given a single reference image with a binary mask, the system extracts features, matches them to a query image, and uses the resulting correspondence as prompts for a segmentation decoder—all without updating any model weights.

Our REBASE framework tackles the background bias by first estimating a low‑rank background subspace from the reference image and then orthogonally projecting both reference and query features away from it ( $F' = F (I - BB^{\top})$ ). This removes shared scene‑context directions before computing similarity.

From the cleaned similarity map we derive prompts using similarity‑weighted farthest‑point sampling, which spreads points across the object rather than clustering on a single peak, and we inject a normalized dense prior into SAM’s mask‑input branch.

**Figure 1.** Qualitative examples of our training-free one-shot segmentation method, demonstrating cleaner object boundaries than GF-SAM and INSID3 across two segmentation tasks (top and bottom rows). Each row shows, from left to right, the reference image with its annotation, the query image with its ground truth, and the predictions produced by GF-SAM, INSID3, and **Ours**.

Training‑free segmentation eliminates the retraining overhead inherent to class‑incremental learning.

Related Work

We situate our approach among foundation models, few‑shot segmentation methods, and feature‑debiasing techniques.

Foundation models have reshaped segmentation by providing a single, promptable backbone that works across diverse objects and domains. Successors such as SAM 2/3 and distilled variants extend this capability to video and edge devices.

SAM is a general‑purpose segmentation foundation model that turns simple cues—points, boxes, or coarse masks—into high‑quality object masks for any image.

One‑shot and few‑shot segmentation methods aim to adapt quickly to new categories with minimal supervision. Early works rely on episodic training, prototype matching, or attention mechanisms, while newer approaches integrate SAM as a universal decoder.

Feature debiasing has emerged as a training‑free way to improve dense correspondence from frozen self‑supervised backbones. Prior work such as INSID3 removes static positional priors, whereas our method eliminates a reference‑conditioned background subspace.

Method Overview

We describe how REBASE removes background from DINOv2 features and builds a similarity map for prompt generation.

One‑shot in‑context segmentation requires a clean similarity signal between a reference foreground and a query image. Our pipeline first projects out the dominant background subspace, then builds a cross‑image similarity map that drives prompt generation for the frozen SAM decoder.

DINOv2 provides dense patch‑level descriptors that capture both object appearance and surrounding context without any task‑specific training.

Why can we rely on frozen DINOv2 features for segmentation without fine‑tuning?

Frozen DINOv2 has been trained on a massive image collection to produce semantically rich, spatially consistent descriptors. Those descriptors already separate foreground from background well enough that a simple orthogonal projection can sharpen the signal without any gradient updates.

REBASE computes the dominant background direction from the reference image and removes that direction from both reference and query features, leaving only target‑relevant information.

Stack them into $F_{BG} = \begin{bmatrix}1 & 0\\0.9 & 0.1\\1.1 & -0.1\end{bmatrix}$.

Compute the dominant direction via SVD (first singular vector) $b = [1,0]^{\!\top}$.

Form the orthogonal projector $P = I - bb^{\!\top} = \begin{bmatrix}0 & 0\\0 & 1\end{bmatrix}$.

Project a query patch $f_Q = [0.8,0.6]^{\!\top}$: $\tilde f_Q = Pf_Q = [0,0.6]^{\!\top}$, removing the background component.

After projection, the query embedding retains only the dimension orthogonal to background, making similarity with foreground prototypes more discriminative.

How does REBASE differ from simply masking out background patches?

Masking discards the entire spatial region, losing any useful context that overlaps foreground. REBASE instead removes only the shared background direction in feature space, preserving the spatial layout while eliminating the confounding signal.

The similarity map measures how well each query patch aligns with the set of foreground‑weighted reference prototypes, yielding a dense score field that highlights likely target locations.

Why use cosine similarity instead of raw dot products?

Cosine similarity is invariant to the magnitude of the embeddings, focusing purely on directional alignment. This prevents patches with larger activation norms from dominating the similarity map merely because of scale.

SW‑FPS picks a set of spatially diverse points that are also highly similar to the reference foreground, balancing coverage and relevance.

First prompt $p_1$ is the global maximum: $p_1$ at position (0,0) with $S=0.9$.

Compute $\tilde d(p)$ for all remaining cells as the normalized Euclidean distance to $p_1$.

Normalize $S$ to $\tilde s(p)$ (min‑max across the grid).

Score each candidate $p$ as $0.5\,\tilde d(p) + 0.5\,\tilde s(p)$ and pick the highest; suppose $p_2$ at (2,2) with $S=0.7$ wins.

Update $P=\{p_1,p_2\}$, recompute distances, and select $p_3$; the algorithm picks (1,1) with $S=0.5$.

SW‑FPS ensures that the three prompts are not clustered together while still focusing on the most similarity‑rich regions, which helps the mask decoder capture elongated or multi‑part objects.

How does SW‑FPS avoid collapsing all prompts onto the same high‑similarity blob?

By adding the distance term $\tilde d(p)$, the selection criterion penalizes points that are too close to already‑chosen prompts, forcing later selections to spread out even if their raw similarity is slightly lower.

**Figure 2.** Overview of REBASE. Given a reference image with its binary mask and a query image, both are encoded by a frozen DINOv2 backbone to extract dense patch features. The reference background subspace is then eliminated, producing a more discriminative cross-image similarity map between the support foreground and query patches. This similarity map is then converted into two complementary conditioning signals for the frozen SAM mask decoder: (i) K spatially diverse point prompts generated via similarity-weighted farthest-point sampling (SW-FPS), and (ii) a dense prior supplied to SAM's mask-input branch. The entire pipeline is training-free, requiring no parameter updates.

Dense Similarity Prior

We replace clustered Top‑K prompts with a training‑free similarity‑weighted farthest‑point sampler.

When prompts are chosen by plain Top‑K on the similarity map, they collapse onto a single high‑score region, leaving large portions of the target un‑covered and producing fragmented masks.

How does SW‑FPS differ from the naïve Top‑K selection that the paper also mentions?

Top‑K simply takes the $K$ highest similarity scores, which often lie in a tight cluster. SW‑FPS adds a distance‑based penalty, so after the first high‑score point the next choice must also be far enough away; the result is a set of prompts that covers the whole target region while still staying in high‑similarity zones.

Compute $\mu_{+}=0.7$, $\sigma_{+}=0.15$ over positive entries; threshold $\mu_{+}+\sigma_{+}/2=0.775$ yields $\Omega$ containing the centre cell and its four immediate neighbours.

Initial prompt $\mathcal{P}$ is the centre cell (highest $S$).

For the second prompt, compute normalized distance $\tilde d$ from each remaining cell to the centre and normalized similarity $\tilde s$; the weighted score $0.6\tilde d+0.4\tilde s$ is maximal at the north‑west neighbour, which is added to $\mathcal{P}$.

For the third prompt, recompute distances to both selected cells; the south‑east neighbour now maximizes the combined score and is added.

SW‑FPS spreads the three prompts across the peak and its surrounding ridge, whereas Top‑K would have taken the centre and its two immediate neighbours, leaving the opposite side of the ridge empty.

Compute mean $\mu_{+}$ and standard deviation $\sigma_{+}$ of positive similarity values $S_{>0}$.

Define candidate set $\Omega=\{p\mid S(p)\ge \mu_{+}+\sigma_{+}/2\}$.

Obtain the diagonal length $D$ of $\Omega$’s bounding box.

Initialize prompt set $\mathcal{P}$ with the arg‑max point of $S$ inside $\Omega$.

For $k=2$ to $K$:

Compute normalized distance $\tilde d(p)=\min_{q\in\mathcal{P}}\|p-q\|_2/D$ for each $p\in\Omega\setminus\mathcal{P}$.

Compute normalized similarity $\tilde s(p)=(S(p)-S_{\min})/(S_{\max}-S_{\min})$.

Select $p^{\star}=\arg\max_{p}\bigl(\alpha\,\tilde d(p)+(1-\alpha)\tilde s(p)\bigr)$ and add it to $\mathcal{P}$.

Return $\mathcal{P}$.

**Figure 3. Prompt placement: Top-K vs. SW-FPS.** Top-K (blue) collapses all $K=8$ prompts onto the argmax of the similarity map, whereas SW-FPS (orange) disperses the prompts across the target region, guiding SAM toward a more complete target mask. The ground-truth boundary is shown in green.

**Algorithm 1** Similarity-Weighted FPS prompt sampler. **Require:** $S, K, \alpha$ 1: $\mu_{+}, \sigma_{+} \leftarrow \text{mean}(S_{>0}), \text{std}(S_{>0})$ 2: $\Omega \leftarrow \{p : S(p) \geq \mu_{+} + \frac{\sigma_{+}}{2}\}$ 3: $D \leftarrow \text{diagonal of the bounding box of } \Omega$ 4: $\mathcal{P} \leftarrow \{\arg \max_{p \in \Omega} S(p)\}$ 5: **for** $k = 2, \dots, K$ **do** 6: $\tilde{d}(p) \leftarrow \min_{q \in \mathcal{P}} \|p - q\|_2 / D$ 7: $\tilde{s}(p) \leftarrow (S(p) - S_{\min}) / (S_{\max} - S_{\min})$ 8: $p^{\star} \leftarrow \arg \max_{p \in \Omega \setminus \mathcal{P}} \alpha \tilde{d}(p) + (1 - \alpha) \tilde{s}(p)$ 9: $\mathcal{P} \leftarrow \mathcal{P} \cup \{p^{\star}\}$ 10: **end for** 11: **return** $\mathcal{P}$

Background Subspace Elimination

REBASE removes reference‑background directions from patch features to sharpen similarity for segmentation.

Because self‑attention mixes each patch with its surroundings, a reference‑query pair often shares background cues (camera pose, illumination, low‑frequency textures). Those cues embed into the foreground prototype and also linger in the query background, inflating cosine similarity for irrelevant regions and destabilizing the dense similarity prior.

How does REBASE differ from a standard PCA denoising of feature vectors?

Standard PCA computes a global basis from a large corpus and removes the top components for all images, assuming those directions are “noise”. REBASE, by contrast, builds B from the reference image’s own background patches in each episode, so the eliminated subspace is scene‑specific and only targets background cues that actually appear in the current pair.

Compute the thin SVD of $X_R$. The right‑singular vectors corresponding to the two non‑zero singular values are the standard basis vectors v₁ = [1,0,0,0] and v₂ = [0,1,0,0].

Select s = 2, so the background basis B = [ v₁ v₂ ] = $\begin{bmatrix}1&0\\0&1\\0&0\\0&0\end{bmatrix}$.

Form the projector $P_B$ = I₄ − B $B^{⊤}$ = $\begin{bmatrix}0&0&0&0\\0&0&0&0\\0&0&1&0\\0&0&0&1\end{bmatrix}$, which zeros out the first two dimensions.

Apply $P_B$ to a query patch $f_q$ = [3, 1, 5, 2]. The projected feature is $\tilde{f}_q = f_q P_B = [0, 0, 5, 2]$, discarding the background‑aligned components (first two entries) while preserving the orthogonal ones.

Apply $P_B$ to a target‑related patch $f_t$ = [0.2, 0.1, 4.5, 3.0]; the projection yields $\tilde{f}_t = [0, 0, 4.5, 3.0]$, showing that target information (last two dimensions) survives unchanged.

REBASE zeroes out exactly the dimensions that the reference background occupies; any query patch that aligns with those dimensions is strongly attenuated, whereas target patches—living outside the background span—remain intact.

With background‑correlated components removed, the subsequent dense similarity prior and SAM mask‑input branch operate on cleaner, target‑focused features, leading to more reliable localization.

Experimental Setup

Key performance gains of the training‑free method across five segmentation benchmarks.

Among training‑free approaches, our method attains the highest mIoU on four of the five benchmarks, with especially large gains on the medical datasets.

On ISIC the method reaches 63.8 % mIoU, improving over the previous best training‑free method (INSID3) by +9.4 pp; on Chest X‑Ray it reaches 86.3 % mIoU, +7.5 pp over INSID3 and only 1.2 pp shy of the fully fine‑tuned SegGPT. On FSS‑1000 it scores 88.2 % mIoU (+0.2 pp over GF‑SAM, +4.5 pp over INSID3). On PACO‑Part it attains 39.3 % mIoU (+0.6 pp over INSID3, +3.0 pp over GF‑SAM). On PASCAL‑Part it records 46.6 % mIoU, ranking second behind INSID3 (50.5 %).

We evaluate five one‑shot segmentation benchmarks spanning natural images, fine‑grained parts, and medical domains. All experiments use a frozen DINOv2 ViT‑L/14 encoder and a SAM ViT‑H decoder; inputs are resized to 518 × 518 for DINOv2 and 1024 × 1024 for SAM, with five positive point prompts per image. Training‑free baselines (Per‑SAM, Matcher, GF‑SAM, INSID3) share the same backbone, enabling a fair comparison.

Main Results

State‑of‑the‑art one‑shot segmentation results across five benchmarks.

The central premise is that REBASE removes background‑correlated components from frozen ViT features, sharpening target localization. This section shows how that translates into concrete performance gains.

Our method attains the highest mIoU on four of five benchmarks, improving over the next‑best by up to +4.8 percentage points.

Table 1 reports mIoU scores; the largest gap (PACO‑Part) is +4.8 % versus the runner‑up.

GF‑SAM applies a simple linear projection to remove the background subspace identified by REBASE, then runs the original SAM decoder on the cleaned features.

INSID3 first extracts a dense similarity prior between reference and query, then uses that prior to guide a segmentation head without any fine‑tuning.

Beyond numbers, we examine visual quality on representative examples from three domains.

**Figure 4.** Qualitative results across the three benchmark categories. Each row shows, from left to right, the reference image, the query image (both with target overlay), and the predictions of GF-SAM, INSID3, and Ours. Our method produces more accurate segmentation masks with cleaner object boundaries on these representative examples.

Ablation Study

Component‑wise analysis reveals how each REBASE element contributes to segmentation quality.

We evaluate REBASE by progressively adding its three ingredients to a vanilla argmax‑point baseline and measuring mean IoU on PACO‑Part and ISIC.

Adding SW‑FPS improves PACO‑Part by +3.69 pp and ISIC by +9.07 pp.

Table 2 shows the jump from 29.54 % to 33.23 % (PACO‑Part) and from 38.40 % to 47.47 % (ISIC) when SW‑FPS is introduced.

Incorporating the dense prior yields an additional +1.64 pp on PACO‑Part and +12.37 pp on ISIC.

Table 2 records 33.23 % → 34.87 % (PACO‑Part) and 47.47 % → 59.84 % (ISIC) after the dense prior is added.

Applying REBASE gives a further boost of +4.41 pp (PACO‑Part) and +3.93 pp (ISIC).

Table 2 reports 34.87 % → 39.28 % (PACO‑Part) and 59.84 % → 63.77 % (ISIC) with REBASE.

Symmetric projection is marginally better, gaining up to +0.24 pp on PASCAL‑Part.

Table 3 shows 46.64 % vs 46.40 % for the two variants.

Our full pipeline reaches 88.2 % mIoU on Chest X‑Ray, the highest reported among the evaluated datasets.

Table 1 lists 88.2 % for Chest X‑Ray, surpassing the next best 87.0 %.

**Figure 5.** Sensitivity to the ratio $r$. mIoU as a function of $r$ (log scale) on FSS-1000 and PASCAL-Part. Performance remains stable in the low-rank regime and progressively degrades as $r$ increases, indicating that only a small number of dominant background singular directions are sufficient for effective debiasing. Our default choice, $r = 0.005$ ($\star$), lies at or near the optimum on both benchmarks.

Additional Experiments

REBASE boosts segmentation accuracy across most benchmarks, with a 19.70‑pp jump on X‑Ray.

REBASE applied to a DINOv3‑L backbone yields a +19.70 pp mIoU gain on the X‑Ray benchmark and improves six of seven datasets.

Table S1 shows +19.70 pp on X‑Ray, +8.20 pp on LVIS‑92i, +7.10 pp on COCO‑20i, and positive gains on all other benchmarks except a –0.27 pp change on FSS‑1000.

**Figure S1.** Comparison of DINOv2-L and DINOv3-L backbones. DINOv3-L improves performance on COCO-20^i, LVIS-92^i, X-Ray and FSS-1000, while DINOv2-L performs slightly better on ISIC, PASCAL-Part, and PACO-Part. These results indicate that the relative strengths of the two backbones are dataset-dependent.

**Figure S2.** Per-benchmark REBASE improvement ($\Delta$ mIoU, pp) under a DINOv3-L backbone. REBASE improves performance on six of the seven benchmarks, with the largest gains on X-Ray, LVIS-92$^i$, and COCO-20$^i$. FSS-1000 exhibits a negligible change due to its already high baseline performance.

Table S1 confirms that REBASE improves the DINOv3‑L baseline on six of seven benchmarks, delivering the biggest jumps on X‑Ray (+19.70 pp), LVIS‑92i (+8.20 pp), and COCO‑20i (+7.10 pp). Table S2 extends this analysis to both DINOv2‑L and DINOv3‑L backbones on COCO‑20i and LVIS‑92i, showing consistent gains of +4.82 pp to +8.20 pp, with the DINOv3‑L backbone again achieving the larger improvements.

Alternative SAM Decoders

REBASE boosts segmentation across SAM 2 and SAM 3 decoders.

REBASE improves segmentation performance across all three benchmarks when using SAM 2, with an average gain of 5.1 pp over the baseline.

Table S3 shows gains of +6.25 pp (COCO‑20i), +5.82 pp (PASCAL‑Part), and +3.22 pp (ISIC2018) for SAM 2.

With SAM 3 the gains persist but are smaller (+1.09 pp to +0.21 pp), reflecting a less compatible prompt interface for our training‑free pipeline.

**Table S3. Evaluation with alternative SAM decoders.** REBASE is evaluated by replacing the original SAM ViT-H decoder with SAM 2 and SAM 3 while keeping the remainder of the segmentation pipeline unchanged. REBASE consistently improves the baseline under both SAM 2 and SAM 3, with larger gains observed for SAM 2. The uniformly lower baseline performance of SAM 3 suggests that its prompt interface is less compatible with the sparse point prompting used in our training-free pipeline.

Positional Debiasing

We examine how REBASE interacts with INSID3’s positional debiasing across benchmarks.

REBASE and INSID3’s positional debiasing are both orthogonal projections applied to DINO features, but they target distinct sources of bias. We therefore probe their interaction by replacing or stacking the two modules within the INSID3 inference pipeline.

**Table S4.** Interaction between REBASE and INSID3's positional debiasing. All configurations use INSID3's original DINOv3-L feature extractor, clustering, and mask synthesis pipeline; only the debiasing module is varied. *Baseline* denotes the original INSID3 method, *REBASE (sub.)* replaces positional debiasing with REBASE, and *Stacked* applies positional debiasing followed by REBASE. Results are reported as mIoU (%). PASCAL-Part and COCO-20$^i$ are averaged over the four standard folds, while Lung X-Ray and ISIC are evaluated in the single-fold class-agnostic setting.

When REBASE substitutes positional debiasing, performance on natural‑image benchmarks collapses, whereas the impact on medical datasets is modest. Stacking REBASE after positional debiasing leaves most scores essentially unchanged, except for a clear gain on ISIC.

Substituting positional debiasing with REBASE drops mIoU by 13.19 pp on PASCAL‑Part.

Table S4 shows Baseline = 49.90 % vs. Substitution = 36.71 %.

Substituting positional debiasing with REBASE drops mIoU by 5.94 pp on COCO‑20i.

Table S4 reports Baseline = 57.35 % vs. Substitution = 51.41 %.

Substitution yields a modest +0.59 pp gain on Lung X‑Ray.

Baseline = 78.79 % vs. Substitution = 79.38 % (Table S4).

Substitution yields a modest +0.42 pp gain on ISIC.

Baseline = 56.21 % vs. Substitution = 56.63 % (Table S4).

Stacking REBASE after positional debiasing leaves PASCAL‑Part, COCO‑20i, and Lung X‑Ray essentially unchanged (differences ≤ 0.5 pp). The only notable improvement appears on ISIC, where stacking raises mIoU by over one percentage point.

Stacking REBASE improves ISIC by +1.66 pp over the baseline.

Baseline = 56.21 % vs. Stacked = 57.87 % (Table S4).

Stacking REBASE improves ISIC by +1.24 pp over the substitution setting.

Substitution = 56.63 % vs. Stacked = 57.87 % (Table S4).

Overall, REBASE and positional debiasing are largely orthogonal: replacing one with the other harms natural‑image performance, while combining them yields at most marginal changes, except for a clear benefit on the ISIC medical benchmark.

Supplementary Overview

Adaptive rank and empirical low‑rank analysis validate REBASE’s background‑subspace design.

Across episodes the number of reference‑background patches $n_{BG}$ can differ dramatically, so a static rank may either exceed the available evidence or capture only a fragment of the true background subspace.

We therefore define the rank adaptively as $s = \lceil r \cdot n_{BG} \rceil$, where $r\in[0,1]$ is a single global hyperparameter; this lets the rank‑$s$ approximation expand or contract in step with the amount of background data present.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers