Domain Arithmetic: One-Shot VLA Adaptation under Environmental Shifts

Taewook Kang, Taeheon Kim, Donghyun Shin, Jonghyun Choi

Domain ARiThmetic (DART) adapts VLA models to new environments using one-shot weight arithmetic.

How can we adapt Vision-Language-Action (VLA) models to new environments using only a single demonstration, without the high cost of full fine-tuning?

Vision-Language-Action (VLA) models trained on one environment often fail when deployed in new settings, such as different camera angles or robot hardware. Collecting enough data to re-train these models for every new environment is prohibitively expensive. DART adapts these models by isolating a "domain vector" through weight-space subtraction: it subtracts a source-domain update vector from a target-domain update vector, then filters out noise using subspace alignment. This approach enables one-shot adaptation that outperforms existing methods, achieving significant success rate gains across diverse visual and embodiment shifts.

Paper Primer

The core move is to treat environmental adaptation as an analogy operation: "target domain = base model + (target update - source update)." By isolating the domain-specific direction from the task-specific noise, DART injects only the necessary environmental knowledge into the base model.

To ensure the extracted domain vector is clean, DART uses subspace filtering and scaling. It identifies basis vectors shared between source and target updates—which represent task semantics—and filters them out, then scales the remaining domain vector by the subspace alignment score to suppress noise.

DART significantly improves success rates in one-shot adaptation scenarios compared to existing baselines.

In LIBERO benchmark experiments with the $\pi_{0.5}$ model, DART achieved an average success rate of 79.1% across novel viewpoints, compared to 74.3% for the next-best baseline (FLA). A 24.6 percentage point gain over the zero-shot base model in novel visual domains.

The method is architecture-agnostic and generalizes to cross-embodiment shifts.

DART consistently outperformed baselines when applied to both flow-matching ($\pi_{0.5}$) and autoregressive ($\pi_{0\text{-FAST}}$) architectures, and successfully bridged the gap between Panda and UR5e robot platforms. Maintained high performance across diverse real-world tasks, including fine-grained manipulation like "Press Stapler."

Why is this method better than simply fine-tuning the model on the one-shot data?

One-shot fine-tuning is dominated by task-specific updates that fail to generalize to other tasks in the new environment. DART isolates the domain-specific component, allowing the model to adapt to the environment while preserving its original multi-task capabilities.

What is the primary assumption required to use DART?

DART requires access to a single source-domain demonstration for the adaptation task. This is a practical assumption in robotics, as source-domain data is typically available from the original training set used to build the base policy.

Introduction: The Challenge of VLA Adaptation

We expose why VLA models struggle under shifts and introduce DART for one‑shot adaptation.

The Vision‑Language‑Action (VLA) models that excel in a source environment often collapse when the camera pose changes or the robot hardware differs, because their fine‑tuning updates entangle task‑specific and domain‑specific information. Collecting task‑wise demonstrations for each new setting is prohibitively expensive, so we target one‑shot adaptation: a single demonstration should supply the transferable domain knowledge. Domain ARiThmetic (DART) isolates the domain‑specific update vector via subspace alignment and adds it to the base policy, enabling efficient adaptation without extra data or architectural changes.

**Fig. 1: One-shot VLA adaptation under environmental shifts.** Environmental shifts can cause a source-trained VLA policy to fail in the novel target domain. (a) Full-data fine-tuning adapts successfully but requires task-wise demonstrations, incurring expensive data collection costs. (b) One-shot fine-tuning is data-light but often fails to adapt across tasks. (c) Our one-shot adaptation extracts domain-specific directions from fine-tuned weights and adapts the policy via weight arithmetic.

The gap between source‑trained VLA performance and target‑domain requirements drives the need for one‑shot adaptation.

Background and Related Work

Survey of VLA adaptation methods and foundational definitions for fine‑tuning and update vectors.

Domain adaptation for Vision‑Language‑Action (VLA) models is essential because pretrained backbones such as RT‑2, OpenVLA, and the $\pi$ series degrade when deployed in novel environments. Common practice augments training data to improve robustness, but collecting diverse demonstrations is expensive. Alternative strategies—semantic‑rich visual features or architectural tweaks—reduce data needs yet sacrifice compatibility across backbones and deployment setups.

Test‑time adaptation methods adjust policies at inference without extra VLA training, but they only address modest shifts. To overcome these limits we propose an architecture‑agnostic adaptation technique that works with minimal target‑domain data under diverse shifts.

Task Arithmetic (TA) treats model updates as weight‑space arithmetic: merging (addition) composes capabilities, while analogy (subtraction‑style) estimates parameter changes that transfer target properties. Merging has been refined through interference mitigation and subspace alignment, yet analogy remains confined to direct subtraction and has only been used in language‑model cross‑lingual or human‑alignment transfer.

In VLA research, merging is the dominant tool for improving generalization or composing skills, but it cannot selectively transfer domain knowledge while preserving task performance. Our empirical analysis reveals that one‑shot fine‑tuned VLA models exhibit disentangled, additive task‑ and domain‑specific directions, making analogy a natural mechanism for isolating the domain vector.

We denote a VLA policy by $\pi_{\theta}(a_t|o_t, T)$, mapping observation $o_t$ and task instruction $T$ to an action distribution. The source and target domains are $E_{\text{src}}$ and $E_{\text{tgt}}$, respectively, and the base policy $\theta_0$ is trained on a suite of tasks in $E_{\text{src}}$.

For each target environment we receive a single demonstration $D_{m,\text{tgt}}$ for task $T_m$. Our goal is to obtain adapted parameters $\theta^{*}$ that perform well across all tasks in $E_{\text{tgt}}$ despite this one‑shot supervision. Adaptation proceeds via behavior‑cloning (BC) fine‑tuning: starting from $\theta_0$, we minimize a BC loss on $D_{m,\text{tgt}}$ to obtain $\theta_{m,\text{tgt}}$, then evaluate $\theta_{m,\text{tgt}}$ on both $T_m$ and held‑out tasks $T_{k\neq m}$.

An update vector captures the weight change induced by fine‑tuning: $\Delta = \theta_{\text{adapted}} - \theta_{\text{base}}$. This vector forms the basis for the analogy operation used later in the paper.

Why One-Shot Fine-Tuning Fails

One-shot fine‑tuning entangles task and domain updates, revealing separable directions in weight space.

One-shot adaptation uses a single demonstration $D_{m,\text{tgt}}$ to fine‑tune the base policy $\theta_0$ for task $T_m$ in a new environment $E_{\text{tgt}}$. Empirically, this fine‑tuning improves performance on the adapted task but collapses on held‑out tasks, indicating that the update mixes task‑specific and domain‑specific directions.

Fine‑tuning with a single demonstration injects a weight‑space shift that simultaneously tries to encode the new task and the new environment, so the resulting update vector entangles two distinct kinds of knowledge.

How does one‑shot fine‑tuning differ from standard multi‑shot fine‑tuning?

Standard fine‑tuning sees many examples per task, allowing the optimizer to isolate task‑specific gradients from domain‑specific ones. One‑shot fine‑tuning has only a single example, so the gradient conflates both signals into a single update vector.

$\gamma$ measures how much of one update vector lies inside the subspace spanned by another, like checking how many arrows of one set point inside the cone defined by another set.

Layer 1: SVD of $\Delta_1^{(1)}$ yields $U^{(1)}_1 = [1,0]^\top$. Project $\Delta_2^{(1)}$ onto this subspace: $\|U^{(1)}_1{}^\top \Delta_2^{(1)}\|_F^2 = 1$. Norms: $\|U^{(1)}_1\|_F^2 = 1$, $\|\Delta_2^{(1)}\|_F^2 = 1$. Hence $\gamma^{(1)} = 1/1 = 1$.

Layer 2: $U^{(2)}_1 = [0,1]^\top$. Projection of $\Delta_2^{(2)}$ gives $\|U^{(2)}_1{}^\top \Delta_2^{(2)}\|_F^2 = 4$. Norms: $\|U^{(2)}_1\|_F^2 = 1$, $\|\Delta_2^{(2)}\|_F^2 = 4$. Hence $\gamma^{(2)} = 4/4 = 1$.

Average over layers: $\gamma(\Delta_1,\Delta_2) = (1+1)/2 = 1$, indicating perfect alignment.

Even with different magnitudes, identical directionality yields $\gamma$ = 1, showing that $\gamma$ captures orientation, not scale.

Why not just use cosine similarity between two update vectors?

Cosine similarity treats each vector as a single direction, ignoring the multi‑dimensional subspace each update spans. $\gamma$ accounts for the entire subspace spanned by the singular vectors, providing a richer measure of shared structure.

The update for a specific task‑domain pair can be rebuilt by adding a task‑only direction, a domain‑only direction, and then removing the common background shift, much like mixing two colors and subtracting the gray base.

Task prototype for A: $\bar{\Delta}_A = (\Delta_{A,X}+\Delta_{A,Y})/2 = [3,0]$.

Task prototype for B: $\bar{\Delta}_B = [0,2]$.

Domain prototype for X: $\bar{\Delta}_X = (\Delta_{A,X}+\Delta_{B,X})/2 = [2,1.5]$.

Domain prototype for Y: $\bar{\Delta}_Y = [1,0.5]$.

Global prototype: $\bar{\Delta} =$ average of all four updates = $[1.5,1]$.

Compose estimate for (A,Y): $\bar{\Delta}_A + \bar{\Delta}_Y - \bar{\Delta} = [3,0] + [1,0.5] - [1.5,1] = [2.5,-0.5]$.

Compute $\gamma$ between the composed vector $[2.5,-0.5]$ and the true $\Delta_{A,Y}=[2,0]$; alignment is high (≈0.98), confirming the additive model.

The subtraction of the global prototype corrects for the baseline drift that would otherwise inflate both task and domain contributions.

Is the subtraction of the global prototype really necessary?

Yes. Without subtracting $\bar{\Delta}$, the task and domain prototypes each retain the common background shift, so adding them would double‑count that component, leading to an over‑estimated update direction.

**Figure 2.** Properties of one-shot fine-tuning. (a) The model is fine-tuned on adaptation tasks in target (Medium) camera viewpoint. Performance remains high in adaptation tasks but generalizes poorly to other held-out tasks. (b) Subspace alignment $\gamma(\cdot, \cdot)$ among update-vectors $\Delta_{m,tgt} = \theta_{m,tgt} - \theta_0$ on $m \in \{1, 2, 3\}$ and $tgt \in \{\text{Source, Medium}\}$. Vectors align for the same task and domain, showing task- and domain-shared directions.

The DART Method

Isolate the domain shift in a fine‑tuned policy and inject it back into the base model.

One‑shot fine‑tuning collapses the update vector onto task‑relevant directions, so the policy cannot adapt to a new environment.

Instead of applying the whole fine‑tuned update, DART isolates the part that encodes the environmental shift and adds only that to the base policy.

Compute the difference: $\delta$ = [5, 3] − [2, −1] = [3, 4].

The magnitude of $\delta$ is √(3² + 4²) = 5, indicating a sizable shift.

Because the source and target updates share the task direction, $\delta$ points toward the pure domain change.

Simple subtraction already isolates a direction that aligns with the environmental shift, but it may still contain small task‑leakage that later filtering will remove.

Why not just use the raw target update $\Delta$ₜₐᵣ₉ₑₜ instead of subtracting the source update?

$\Delta$ₜₐᵣ₉ₑₜ contains both the task direction (which the base policy already knows) and the domain shift. Adding the whole vector would double‑count the task knowledge and waste capacity, whereas subtraction leaves only the novel domain component.

We keep only those basis directions of the source update that lie in the target’s subspace, then down‑weight the whole domain vector if the two subspaces barely overlap.

Compute C = Uₛᵣ𝚌ᵀ Uₜₐᵣ₉ₑₜ = I · I = I.

Energy for each source basis: e₁ = ‖C e₁‖² = 1, e₂ = ‖C e₂‖² = 1.

Assume $\gamma$ = 0.8 (high overlap). Both energies exceed $\gamma$, so $J_l$ = {1, 2} (no filtering).

Domain vector before scaling: $\Delta$ₜₐᵣ₉ₑₜ − $\Delta$ₛᵣ𝚌 = [[1, 0],[0, ‑0.5]].

Scale by $\gamma$: \tilde{$\delta$} = 0.8 · [[1, 0],[0, ‑0.5]] = [[0.8, 0],[0, ‑0.4]].

When the subspaces are well aligned, filtering does nothing and scaling simply reduces the magnitude, acting as a confidence knob.

How does this filtering differ from a plain low‑rank approximation of the update?

Low‑rank approximation discards components based on singular‑value magnitude alone, which may remove task‑relevant directions. Our filter discards bases that are *misaligned* with the target subspace, preserving any low‑energy but task‑aligned directions.

Fine‑tune the base policy $\theta$₀ on source data Dₘ,ₛᵣ𝚌 to obtain $\theta_{m,src}$ and compute $\Delta_{m,src}$ = $\theta_{m,src}$ − $\theta$₀.

Fine‑tune $\theta$₀ on target data Dₘ,ₜₐᵣ₉ₑₜ to obtain $\theta_{m,tgt}$ and compute $\Delta_{m,tgt}$ = $\theta_{m,tgt}$ − $\theta$₀.

Subtract $\Delta_{m,src}$ from $\Delta_{m,tgt}$ to get the raw domain vector $\delta$.

Perform SVD on both updates, compute alignment matrix $C$, and filter source bases using the energy threshold derived from $\gamma$.

Scale the filtered domain vector by $\gamma$ to obtain \tilde{$\delta$}_{tgt}.

Add $\alpha$ \tilde{$\delta$}_{tgt} to $\theta$₀ to produce the adapted policy $\theta^*$.

**Fig. 3:** Additive task-domain directions in update-vectors. (a) Prototypes are computed by 16 update-vectors from 4 tasks and 4 domains. Strong alignment by additive composition suggests orthogonal, linearly combinable task and domain components. (b) View is viewpoint shift, Noise is camera noise, Light is light change. Alignment among similar domain shifts shows that domain components are structured and correlated with semantic of domain shifts.

**Fig. 4: Overview of the proposed VLA adaptation approach.** (a) We compute update-vectors $\Delta_{m,src}$ and $\Delta_{m,tgt}$ by fine-tuning a base policy $\theta_0$ on a single task $\mathcal{T}_m$ using source and target data. (b) A domain vector $\tilde{\delta}_{tgt}$ is extracted by subtracting task directions, with subspace filtering to suppress misaligned components. Adding $\tilde{\delta}_{tgt}$ back to $\theta_0$ yields a multi-task policy $\theta^*$ adapted to the target domain.

Experimental Setup

Key performance numbers illustrate DART’s advantage across benchmarks.

DART outperforms all baselines on LIBERO by a clear margin.

Average success rate 79.1 % versus 74.3 % for the strongest baseline (FLA) (Table 2).

We evaluate DART on two families of VLA models—$\pi_{0.5}$ (flow‑matching) and $\pi_{0\text{-}FAST}$ (autoregressive)—across simulation, cross‑embodiment, and real‑world domains. Each experiment uses a single demonstration per target scene (one‑shot) and repeats three random adaptation seeds.

LIBERO is a suite of 40 robot‑manipulation tasks that can be perturbed with camera viewpoint changes and visual noise to mimic real‑world shifts.

MimicGen measures how well a policy trained on one robot arm transfers to a different arm using only a single demonstration.

**Fig. 5: Overview of experimental setups.** We experiment on four setups: simulation setups with novel viewpoints (top-left) and combined visual perturbations (bottom-left) on LIBERO [37], a cross-embodiment transfer setup on MimicGen [42] (middle), and a real-world setup on two third-person camera viewpoints (right).

Experimental Results

DART consistently outperforms baselines across visual and embodiment shifts.

DART attains an average success rate of 81.7 % on cross‑embodiment tasks, surpassing the next‑best method by 26.7 percentage points.

Table 4 shows DART’s 81.7 % average versus FLA’s 55.0 % and other baselines.

**Figure 6.** Performance under hyperparameter choices on LIBERO across novel viewpoints. We average success rates (%) across Small, Medium, Large camera views.

DART also dominates under combined visual perturbations (Table 2) and transfers to the autoregressive VLA model $\pi_{0\text{-FAST}}$ (Table 3), confirming that the additive update‑vector structure is architecture‑agnostic.

In a real‑world UR10e setup (Table 5), a single Stack‑Cube demonstration suffices for DART to reach 91 % success on Stack‑Cube and 100 % on Lemon, far exceeding all baselines that struggle under the same one‑shot budget.

Merging domain vectors from three novel viewpoints (Table 7) yields lower averages (74.5–75.7 %) than using a single DART vector (79.1 %), indicating that a consolidated vector preserves performance while simplifying storage.

DART’s superior performance across novel viewpoints and across robot embodiments demonstrates that isolated domain vectors enable reliable one‑shot VLA adaptation.

Extended Results

Key quantitative findings and detailed tables for MimicGen and real‑world experiments.

DART (Ours) attains the highest average success rate across novel viewpoint shifts on LIBERO.

Table 18 reports a 92.0 % average for DART, exceeding the next‑best method by 11.2 %.

Table 14 defines the progress‑rate rubric used on MimicGen, where the 50 % milestone corresponds to grasping the target object and 100 % to completing the placement.

In MimicGen we pretrain $\pi_{0.5}$ on Square and Threading (3,800 demonstrations) under D₀ randomization, then train the source model $\theta_{\text{src}}$ on Stack and Stack Three (1,900 demonstrations) and perform one‑shot fine‑tuning with a single demonstration per embodiment; action‑normalization statistics from pretraining are fixed, and training runs on two NVIDIA A100 GPUs.

Evaluation runs ten no‑op steps, uses $\alpha=0.4$, and limits episodes to 200 steps for Stack and 400 steps for Stack Three.

Table 15 shows the base $\pi_{0.5}$ policy achieves near‑perfect success (≈100 %) in the source viewpoint but drops to 43 % average in the target viewpoint, illustrating the severity of domain shift.

Real‑world experiments employ a UR10e arm with a Robotiq 2F‑85 gripper and three RealSense D455 cameras (one wrist‑mounted, two fixed for source and target viewpoints); we evaluate three pick‑and‑place tasks (Eggplant, Lemon, Carrot) and two fine‑grained manipulation tasks (Stack Cube, Press Stapler).

The source policy is trained on 120 teleoperated demonstrations (24 per task) and fine‑tuned with a single target‑domain demonstration for Stack Cube; during evaluation each policy runs on 12 predefined object positions, predicts 20 actions but executes the first 15, all on a single NVIDIA A6000 GPU.

Tables 16 and 17 provide upper‑bound performance when fully fine‑tuning, reaching up to 93.7 % ($\pi$₀.5) and 87.9 % ($\pi$₀‑FAST), indicating the performance ceiling for these settings.

Table 18 compares DART to model‑merging baselines (TIES, Iso‑C, RESM), where DART consistently outperforms them across all viewpoint magnitudes.

Table 19 shows DART also surpasses the test‑time adaptation method SCALE on $\pi_{0\text{-}FAST}$, achieving the highest average success across all shifts.

Table 20 demonstrates that after adapting with DART, performance on the original source domain remains strong (79.4 % average), confirming that adaptation does not degrade previously learned capabilities.

**Fig. 7:** Visualization of feature shifts induced by task and domain prototypes.

**Fig. 8: Images of each LIBERO scene under different viewpoints.** Columns represent viewpoint shift levels, and rows correspond to scenes.

Additional Performance Metrics

Key supplementary findings on DART’s robustness, merging, and layer choices.

Beyond the primary experiments, we evaluate DART’s ability to retain source‑domain performance, to compose multiple domain vectors, and to identify which VLA components benefit most from adaptation.

DART attains the highest average success (79.1 %) across novel‑viewpoint shifts, outperforming all merging baselines.

Table 21 shows DART’s average of 79.1 % versus the next‑best merging method (TSV) at 75.7 %.

**Table.** Novel Viewpoints (Success Rate, %) comparison of DART against various merging methods.

The table presents performance metrics for target-domain adaptation tasks $T_m$ based on different source-domain adaptation tasks $T_{m'}$. The rows represent different selection strategies for the source task ($m'=m$, Cosine-Sim Top 1/2, and Random 1/2), while the columns represent specific target tasks ($m=1, 2, 3$) and their average performance.

**Fig. 13: Impact of choice of layers to adapt.** Vis is vision encoder, LLM is language model, and Action is action expert in the VLA model, $\pi_{0.5}$. We report the average success rate (%) on LIBERO in three novel viewpoint shifts (Small, Medium, Large) (Left). We also measure the average absolute value of the domain vectors across the chosen layers (Right).

**Fig. 9:** Example rollouts on MimicGen. **Top:** Example rollout of the Stack task with UR5e. **Bottom:** Example rollout of the Stack Three task with UR5e.

**Fig. 10: Real-world setup and example rollouts.** Left: Viewpoint configuration for real-world experiments, using one wrist-mounted camera and a third-person camera (source or target viewpoint). Right: Example rollout of the Lemon task from the source viewpoint (top) and the target viewpoint (bottom).

Supplementary Methods

Supplementary details on DART’s algorithm, assumptions, acceleration, and experimental configurations.

This supplementary section expands on the core DART method, its assumptions, computational shortcuts, and the experimental pipelines used throughout the paper. It also documents baseline implementations and additional analyses that support the main results.

**A.1 Algorithm.** DART computes one‑shot update vectors for a source‑domain demonstration and a target‑domain demonstration, extracts a domain‑specific subspace via per‑layer SVD, and adds the resulting domain vector (scaled by $\alpha$) to the base multi‑task policy. The procedure is summarized in Algorithm 1.

**A.2 Justification for Using Source‑domain Demonstrations.** The method assumes access to the original pre‑training dataset or at least a single demonstration from it; such data are often publicly available in large‑scale robotic repositories. When exact task matching is impossible, the paper shows that selecting a similar source task (by feature cosine similarity) still yields a useful domain vector, with only modest performance loss.

**A.3 Accelerating DART with Randomized SVD.** Full SVD on each layer’s update matrix is costly ( O(m n min(m,n)) ); replacing it with a truncated randomized SVD of rank r ≈ 256 reduces the per‑layer cost to O(m n r) while preserving success rates. Table 8 demonstrates that the randomized variant cuts runtime from 15 min 35 s to 6 min 33 s with only a 0.3 % drop in success on average.

**A.4 Motivation for Task‑Domain Decomposition in Update‑Vectors.** VLA inputs contain overlapping token groups that encode task‑relevant language and visual cues versus domain‑specific background and embodiment information. One‑shot fine‑tuning therefore tends to move weights along partially disentangled directions, enabling the isolation of a domain vector that can be transferred across tasks.

**B.1 RETAIN** and **B.2 FLA** baselines are reproduced with the authors’ publicly released code, matching the original hyper‑parameters (learning rate 1e‑4, batch size 64, Adam optimizer). Minor implementation details—such as weight initialization seeds and data augmentation pipelines—are aligned with the cited repositories.

**C.1 VLA Model & Training.** The base policy uses a transformer encoder with 12 layers, hidden size 512, and 8 attention heads; training employs AdamW (lr 3e‑4) for 200 k steps on the source dataset. **C.2 LIBERO** experiments follow the standard split of 30 tasks and evaluate on unseen visual domains. **C.3 MimicGen** transfers policies across simulated robot embodiments using the provided generative pipeline. **C.4 Real‑World** tests run on a 7‑DoF manipulator with RGB‑D input, using the same hyper‑parameters as simulation.

**D.1–D.5 Additional Analyses.** Suite‑wise LIBERO results (D.1) confirm that DART consistently improves across all task groups. Full‑data fine‑tuning (D.2) provides an upper bound that DART approaches with a single demonstration. Comparisons with model‑merging (D.3) and test‑time adaptation (D.4) show DART’s superior efficiency and comparable accuracy. Further ablations (D.5) explore the impact of rank r, alignment thresholds, and the number of source demonstrations.

**Table 8.** Effect of randomized SVD. We report success rate (%) and runtime on novel viewpoints. For randomized SVD, we use a target rank of r=256. Runtime is averaged over three runs and measured on a machine with 1TB RAM and an Intel Xeon Platinum 8562Y+ CPU.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers