SyncWorld: Visual Calibration Enables World Models as Zero‑Shot Simulators

Yuncong Yang, Zhengtao Han, Furkan Ozyurt, Zeyuan Yang, Han Yang, Junyi Cao, Haoyu Zhen, Yilun Du, Chuang Gan

SyncWorld uses in-context visual calibration to enable zero-shot robotic simulation across unseen camera views and embodiments.

How can we make action-conditioned world models generalize to unseen robot embodiments and camera views without retraining?

Numerical robot actions are not a universal language in pixel space; the same control signal produces different visual motion depending on camera placement, robot base, or embodiment. This setup-dependence causes world models to break when deployed in new environments. SyncWorld resolves this by conditioning video prediction on a short "visual calibration" episode—a sequence of paired frames and actions that explicitly demonstrates the current setup's action-to-visual mapping. The model learns to interpret actions through this visual evidence, allowing it to function as a zero-shot simulator without further training. Experiments show that SyncWorld consistently outperforms existing world models on unseen environments and enables test-time policy improvement by ranking candidate actions via imagined rollouts.

Paper Primer

SyncWorld treats the Action–Visual Mapping as a latent relation that is resolved in-context. The core move is to prepend a canonical sequence of 12 short video segments—each demonstrating a specific degree of freedom—to the model's input, effectively "calibrating" the model to the current setup's geometry.

To ensure the model doesn't just memorize fixed conventions, the authors use action-coordinate augmentations during training, randomly flipping and permuting axes while keeping the video constant. They also employ calibration distillation: the model is trained to match its own calibration-conditioned predictions using only interaction history, enabling flexible inference when an explicit calibration snippet is unavailable.

SyncWorld achieves superior zero-shot simulation quality and multi-view spatial consistency compared to state-of-the-art action-conditioned world models.

Across ManiSkill, LIBERO, and real-world benchmarks, SyncWorld consistently outperforms baselines like IRASim and WorldGym on PSNR, SSIM, LPIPS, and Met3r consistency metrics. The model approaches the ground-truth upper bound for multi-view consistency, demonstrating robust 3D-aware action grounding.

SyncWorld enables zero-shot policy improvement without additional training.

By using the world model to rank candidate action chunks via imagined rollouts (GPC-Rank), the system improves success rates on unseen tasks, approaching the performance of an oracle using a ground-truth simulator. The model serves as a reliable substitute for a simulator, allowing policies to select better actions in novel environments.

Why is a "visual calibration" episode necessary instead of just using a larger, more diverse training dataset?

Even with massive data, the Action–Visual Mapping remains setup-dependent; a model trained on diverse data often struggles to fit multiple incompatible mappings for the same action values. Calibration provides the specific visual evidence needed to resolve this ambiguity for a particular camera or robot configuration at inference time.

Does the model require a dedicated calibration episode every time it is deployed?

No. While explicit calibration provides the strongest controllability, the model is trained via distillation to approximate the mapping from interaction history alone, allowing it to operate in a history-only mode when a calibration snippet is unavailable.

SyncWorld demonstrates that world models can function as general-purpose, zero-shot simulators for robotics by learning to interpret actions through in-context visual evidence rather than fixed coordinate conventions.

The Challenge of Generalizable World Models

World models struggle to generalize because low‑level actions lack a universal visual meaning.

World models are increasingly used as imagination environments for policies, but reliable rollouts demand fine‑grained control over low‑level robot actions. In practice, the same numeric action can look very different in pixel space because camera pose, robot placement, and embodiment all change the visual effect. This setup dependence creates two core problems: conflicting supervision when mixing data from many setups, and a brittle test‑time mapping that breaks without additional training.

To use world models for planning, we must give them a reliable way to translate a low‑level command into its exact visual consequence, no matter how the robot or camera is configured.

**Figure 1:** **SyncWorld** allows for zero-shot simulation across previously unseen camera angles and embodiments through in-context visual calibration. By conditioning on a brief interaction episode that demonstrates all six motion DoFs (left), the model resolves setup-specific Action-Visual Mappings and can anticipate accurate future rollouts (right) in novel environments without additional training.

The fundamental gap is that action‑conditioned world models lack a mechanism to bridge the diversity of real‑world visual outcomes.

The SyncWorld Framework

Method introduces SyncWorld, a calibration‑driven world model that grounds actions in setup‑specific visual evidence.

World models struggle when the same low‑level command produces different visual motion across robots and camera setups. SyncWorld solves this by inserting a brief, setup‑specific visual calibration episode that tells the model how actions map to pixels in the current environment.

SyncWorld treats a calibration video as a “reference shot” that tells the model what a unit move looks like for the current robot and camera, then conditions all future predictions on that reference.

How does SyncWorld differ from domain randomization or data‑augmentation approaches?

Domain randomization varies camera poses or robot parameters during training but never gives the model a concrete visual example of the current setup. SyncWorld instead supplies an actual short video $C_s$ that directly shows the mapping, forcing the model to read the semantics from that episode.

Think of the calibration episode as a quick “pose‑to‑pixel” demo: the robot moves each joint a little while a fixed camera records the resulting image shift, producing a set of reference clips that encode the current Action–Visual Mapping.

Segment $C_{s,x,+}$ captures frames 1→2 where the arm shifts right by 5 px.

Segment $C_{s,x,-}$ captures frames 3→4 where the arm shifts left by 5 px.

Segment $C_{s,z,+}$ captures frames 5→6 where the arm lifts up by 4 px.

Segment $C_{s,z,-}$ captures frames 7→8 where the arm lowers down by 4 px.

These four 2‑frame clips are concatenated in the order $+x, -x, +z, -z$ to form $C_s$.

Even with only a handful of pixels of motion, the model can infer the direction‑to‑pixel relationship for the current camera pose.

Why not simply calibrate by measuring joint angles and camera extrinsics instead of using video?

Measuring extrinsics requires a calibrated rig or marker system, which adds hardware complexity. Using a short video episode avoids any explicit geometry estimation— the model learns the mapping directly from raw pixels.

The Diffusion Transformer (DiT) processes spatio‑temporal video latents with self‑attention layers, enabling high‑fidelity frame prediction conditioned on arbitrary context tokens.

**Figure 2: SyncWorld Architecture.** SyncWorld is an action-conditioned world model that employs a Diffusion Transformer (DiT) backbone to predict future video frames. The architecture incorporates a visual calibration episode and interaction history to clarify setup-specific Action-Visual Mappings. These inputs are encoded as pose embeddings and video latents, which enable the DiT to generate temporally and spatially consistent future frames.

**Figure 3.** **Calibration Episode.** To resolve the setup-specific Action-Visual Mapping, we collect a brief interaction sequence where the robot performs directional motions across each DoF. We do not include gripper openness as the visual consequence is trivial. Each segment has a single control dimension, offering visual evidence of how numerical actions translate into pixel-space motion in the current camera view.

Record a raw episode $E_s$ with a fixed camera while the robot executes a sequence of motions.

For each degree of freedom $d$, randomly choose a direction $+$ or $-$ and move the joint once.

Return the robot to a nominal pose after each motion to isolate the effect of that DoF.

From $E_s$, extract the strongest contiguous signed segment $C_{s,d,\sigma}$ for each $d$ and sign $\sigma$.

Concatenate the 12 segments in a canonical order to form the calibration context $C_s$.

Randomly flip the signs of the translational axes $\{x, y, z\}$ in both $C_s$, $H_t$, and $A_t$.

Permute the order of the three translation axes.

Apply a global scale factor to the translational dimensions.

Leave the video observations unchanged, forcing the model to rely on $C_s$ for the correct mapping.

Construct a teacher input that includes the full calibration context $C_s$.

Construct a student input that omits $C_s$, leaving only the interaction history $H_t$.

Run both inputs through the same DiT backbone to obtain predictions.

Minimize the divergence between student and teacher predictions, encouraging the student to infer the Action–Visual Mapping from $H_t$ alone.

At test time we improve a given policy by sampling $K$ candidate action chunks, rolling out each candidate with the calibrated world model, scoring the imagined videos with a visual language model, and executing the highest‑scoring chunk.

Empirical Evaluation and Policy Improvement

We evaluate SyncWorld’s zero‑shot simulation quality, cross‑view consistency, and policy‑improvement gains.

SyncWorld trains to predict 512×512 videos in just 2–3 days on a 4×8 H100 GPU cluster.

Training uses a batch size of 64 and converges within the reported time frame.

Table 1 reports PSNR, SSIM, LPIPS, and FID for five methods across three domains; SyncWorld with calibration attains the best values on every metric, confirming superior video quality in unseen environments.

Table 2 evaluates cross‑view consistency with the Met3r score (lower is better). SyncWorld consistently yields lower scores than IRASim, WorldGym, and Ctrl‑World, and its results are close to the ground‑truth upper bound, demonstrating reliable 3‑D‑aware predictions.

Table 3 compares task success rates for direct policy, SyncWorld + GPC‑Rank, and an oracle simulator. In diagnostic tasks where the oracle provides headroom, SyncWorld + GPC‑Rank improves success over the baseline and approaches the oracle ceiling, evidencing practical zero‑shot policy improvement.

Table 4 ablates visual calibration and distillation. Removing calibration degrades generalization, while removing distillation harms performance when test‑time calibration is unavailable, highlighting both components’ roles.

Given a language instruction, the VLA model proposes several action candidates; SyncWorld predicts their future visual outcomes from multiple viewpoints, and a vision‑language model scores these rollouts to select the best candidate without any additional training.

How does SyncWorld differ from a standard video prediction model that simply conditions on past frames?

Standard predictors rely only on past visual context and cannot adapt to new action‑visual correspondences without explicit calibration. SyncWorld, by contrast, learns a brief in‑context calibration episode that explicitly aligns actions with their visual effects, enabling zero‑shot generalization to unseen viewpoints and embodiments.

**Figure 4.** **Zero-shot Policy Improvement via SyncWorld.** Given a natural language instruction and current observations, a VLA policy generates multiple different candidate action chunks. SyncWorld predicts future visual outcomes for each candidate from multiple viewpoints to ensure spatial consistency. A VLM serves as an outcome evaluator, assigning scores to the imagined rollouts according to the original instruction. The action candidate with the highest reward is selected for execution, enabling test-time policy improvement in novel environments without additional training.

**Figure 5.** **Qualitative Results.** We evaluate the performance of SyncWorld on a real-world setup with an unseen robot embodiment and novel camera viewpoints. Compared to the Ctrl-World baseline, which fails to accurately ground numerical actions in this new pixel space, SyncWorld produces future rollouts that are highly consistent with the Ground-truth observations.

SyncWorld delivers substantial zero‑shot performance gains across video quality, cross‑view consistency, and policy improvement, confirming its practicality as a plug‑and‑play simulator.

Data Generation and Experimental Setup

Details of data generation, inference setup, and evaluation metrics.

Our data pipeline first randomizes camera poses, records a brief calibration episode, then replays expert trajectories and optionally generates counterfactual rollouts. The resulting videos, robot states, and metadata are stored hierarchically by task, episode, and camera setup.

**Figure 6.** Extreme camera view

**Figure 8.** Interacting with multiple objects

Met3R evaluates cross‑view consistency by projecting DINO features into 3‑D using DUSt3R pointmaps, then measuring cosine similarity on overlapping pixels. The final cross‑view score averages Met3R between generated and ground‑truth views.

Policy improvement uses a zero‑shot MPC loop: the base policy $\pi$₀ proposes $n_{actions}=16$‑step chunks, $N=8$ candidates are evaluated by a GPT‑5 reward model on three subsampled frames, and the highest‑scoring chunk is executed. Hyperparameters are listed in Table 7.

Ablations and Robustness Checks

Ablations reveal which components truly drive SyncWorld’s performance.

We evaluate how each component of SyncWorld contributes to performance by ablating them and measuring the impact on key metrics.

Using the default augmentation probability 0.6 yields a PSNR gain of +1.7 over the no‑augmentation variant, confirming that moderate augmentation is optimal.

Table 8 shows PSNR 28.3 for SyncWorld versus 26.6 for the 0.0‑augmentation variant.

On Ketchup and Put Cream Cheese, the oracle simulator underperforms the direct policy by 0.06 and 0.04 respectively, indicating the ranking step is the bottleneck.

Table 9 reports Ketchup 0.80 (baseline) vs 0.74 (oracle) and Put Cream Cheese 0.34 (baseline) vs 0.30 (oracle).

Conclusion and Summary

Appendix Overview: summarizing conclusions and cited works.

Recall that world models lack calibrated action–visual mappings, and SyncWorld addresses this with a brief in‑context calibration episode. The authors demonstrate that SyncWorld scales across heterogeneous robot data, generalizes zero‑shot to unseen camera views and embodiments, and yields high‑quality video predictions and spatial consistency. Moreover, the calibrated model enables test‑time rollout‑based action selection, offering a promising zero‑shot simulator for generalizable robotic learning.

The bibliography lists over 70 works covering latent action world models, video diffusion, and embodied policy learning, reflecting the broad context of this research.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers