DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training
Junyan Li, Ruizhi Li, Yu Liu, Xiangshuo Liu, Mingchao Sun, Hongyu Pan, Mu Xu, Lue Fan, Zhaoxiang Zhang
DREAMTRUE improves robot world model action-faithfulness and physical plausibility via geometric calibration and reward-guided counterfactual post-training.
How can we train a robot world model that remains physically consistent and action-faithful when predicting counterfactual future videos?
Robot world models often struggle to follow specific action trajectories accurately due to poor camera calibration, and they frequently hallucinate successful outcomes even when the robot misses a grasp or slips. The authors introduce DREAMTRUE, which uses offline geometric calibration to align robot renderings with video frames and employs a human-annotated reward model to guide post-training toward physically plausible interactions under counterfactual actions. On the AgiBot benchmark, this approach achieves state-of-the-art action following and reduces the human-assessed interaction defect rate from 48.12% to 6.25%.
Paper Primer
The core mechanism hinges on two stages: first, refining camera parameters using image correspondences from existing recordings to ensure action conditions are spatially accurate; second, using a reward model trained on human defect annotations to penalize physically impossible outcomes (like objects floating after a missed grasp) during reinforcement learning post-training. This is like a simulator that first fixes its "eyes" to see the robot's arm exactly where it is, then learns to "feel" physics by being told which of its own generated mistakes are physically nonsensical.
DREAMTRUE achieves superior action-following accuracy compared to existing world models.
Normalized dynamic time warping (nDTW) scores on the AgiBot benchmark. 0.8772 nDTW, outperforming all evaluated baselines.
Reward-guided post-training significantly improves physical interaction plausibility.
Human-assessed interaction defect rates on counterfactual test conditions.
Why is offline geometric calibration necessary if robot datasets already provide camera parameters?
Existing recordings often lack precise calibration, leading to spatial misalignment between the rendered robot and its appearance in the video; this inconsistency degrades the model's ability to follow actions accurately.
How does the model learn to avoid interaction errors without paired ground-truth future videos for counterfactual actions?
The model uses a reward model trained on human-annotated defect datasets to identify observable physical errors—such as object-gripper detachment or floating objects—and uses these scores as feedback to guide the generator toward more plausible outcomes.
Introduction and Motivation
We expose why robot world models lack action‑faithfulness and introduce DREAMTRUE to close the gap.
Robot world models aim to predict how a scene evolves under a given action sequence, but they routinely struggle with action‑faithfulness and physical plausibility. Two core obstacles cause this gap: (1) imprecise camera calibration misaligns rendered robot trajectories with recorded videos, breaking action following; and (2) datasets are dominated by successful demonstrations, leaving models blind to failed grasps, slips, and other erroneous interactions.
**Figure 1.** Overview of DREAMTRUE. Camera calibration aligns robot renderings with recorded videos. Stage I learns action-conditioned prediction across embodiments, while Stage II uses video rewards for counterfactual post-training to improve interaction plausibility.
The gap between current world models and physically consistent, action‑faithful video prediction drives the need for calibrated visual conditions and reward‑guided counterfactual training.
The DREAMTRUE Architecture
How calibrated renderings and a two‑stage pipeline enable action‑faithful video prediction.
Robot world models often misalign rendered robot poses with real video frames and generate implausible interactions, which hampers action‑faithful prediction.
We first tighten the spatial relationship between robot renderings and recorded video by adjusting camera intrinsics, extrinsics, and any arm‑mount offsets so that projected robot geometry lands on the same pixels as the real robot.
Render the robot at joint state $s$; the tip projects to pixel $(350,260)$.
Observed pixel from the real video is $(355,265)$, giving a reprojection error of $\sqrt{5^2+5^2}=7.07$ px.
Joint optimization reduces the error to $1.2$ px by adjusting $f$ to $805$ px and shifting $\xi$ by $-2$ mm.
Ray‑coplanarity between the two cameras adds a constraint that aligns their baseline, further decreasing the total loss by $15\%$.
Even a few pixels of misalignment can cause large pose errors after several prediction steps; the joint calibration eliminates this cascade.
How does this calibration differ from a standard camera‑calibration routine?
Standard calibration only fits intrinsics/extrinsics using a static checkerboard, ignoring the robot’s geometry and any arm‑mounted cameras. Our joint calibration simultaneously aligns the rendered robot surface and enforces multi‑view ray coplanarity, directly tying the parameters to the downstream video‑prediction task.
First we teach the model to predict future videos from accurate renderings (Stage I), then we refine it with counterfactual actions and a learned video‑reward signal (Stage II), ensuring both fidelity and physical plausibility.
Render robot trajectories $a_{1:T}$ with calibrated parameters $(\Theta^{*},\xi^{*})$ to obtain RGB images $R^{v}_{t}$ for each view $v$ and time $t$.
Encode $R^{v}_{t}$ with a pretrained video VAE; encode masks $M^{v}_{t}$ and Plücker maps $P^{v}_{t}$ with a 3‑D geometry encoder.
Form the action representation $C=\{(\tilde{R}^{v}_{t},P^{v}_{t})\}_{t,v}$ and feed it into the VACE conditioning branch.
Tile video latents from all views along the width dimension and inject VACE residuals into the DiT layers.
Train with flow‑matching loss $\mathcal{L}_{\text{SFT}}$ on paired future videos from dataset $\mathcal{D}$.
Stage II: Reward‑guided post‑training (DiffusionNFT)
Render $R^{1}_{1}$ (view 1, t=1) as a 64×64 RGB image showing the arm at 0 mm.
Encode $R^{1}_{1}$ to a latent $z_{1}^{1}$; similarly obtain $z_{1}^{2}$ for view 2.
Tile $z_{1}^{1}$ and $z_{1}^{2}$ → combined latent $Z_{1}$.
Inject VACE residuals derived from the gripper opening (mapped to intensity 0) into the DiT block.
Apply flow‑matching loss $\mathcal{L}_{\text{SFT}}$ against the ground‑truth future latent $z_{\text{gt}}$.
The tiled multi‑view layout lets the model share information across cameras, while the VACE residuals encode the exact gripper state for each view.
Why not train the whole system end‑to‑end without separating Stage I and Stage II?
End‑to‑end training would mix accurate supervised signals with noisy counterfactual rewards, causing the model to underfit the high‑quality paired videos. The two‑stage design first locks in a strong conditional prior (Stage I) and then uses the reward only to adjust the parts of the model that struggle with physical plausibility (Stage II).
**Figure 2.** Correspondences used for calibration: (a) robot renderings and observations, (b) static backgrounds across time, and (c) synchronized camera views.
**Figure 3.** Unified multi-view world model. Robot renderings, gripper openings, and camera rays condition the video DiT through a VACE branch to predict multi-view future videos.
**Figure 10.** Training-data curation pipeline from raw demonstration sources through geometric calibration and motion filtering to the two training stages; numbers denote cumulative hours.
By first aligning renderings to reality and then refining predictions with counterfactual rewards, the method achieves both accurate video forecasting and physically plausible robot‑object interactions.
Counterfactual Prediction Performance
DREAMTRUE sets new benchmarks for action following and physical plausibility.
DREAMTRUE achieves the highest action‑following nDTW (0.8831) and best visual fidelity across both recorded and counterfactual actions.
Table 1 shows the full model reaching nDTW 0.8772 (recorded) and 0.8831 (counterfactual), while also attaining the top PSNR, SSIM, and lowest LPIPS among all baselines.
**Figure 5.** Qualitative comparison of counterfactual action predictions. Compared with GE-Sim 2.0 and our model without RL, the full model avoids unsupported scanner motion and object disappearance or duplication in these examples. Yellow annotations highlight relevant regions. The reward model scores L1–L3 assess embodiment, object, and interaction quality, respectively.
**Figure 6.** Predictions from the same initial scene under multiple action conditions.
DREAMTRUE significantly reduces defect rates in counterfactual scenarios compared to baselines.
Policy Outcome Alignment
Additional experiments quantify the impact of calibration and reward modeling on policy fidelity.
Across 24 task‑policy‑setting combinations, the model attains a Spearman rank correlation of 0.937, indicating near‑perfect ordering of policy success rates.
Evaluated on three bimanual RoboTwin 2.0 tasks under easy and hard settings, comparing generated rollouts to human‑annotated video outcomes versus ground‑truth simulation labels.
Figure 8 visualizes this improvement: our linear fit ($y = 1.032x - 0.009$) aligns closely with the ideal diagonal, while the baseline ($y = 0.919x + 0.087$) consistently overshoots.
**Figure 8.** Policy outcome alignment against ground-truth simulation across 24 evaluation conditions (inset: success-rate MAE).
Refined geometric calibration at inference yields a PSNR gain of 1.30 dB on DROID, demonstrating the benefit of precise spatial alignment.
When calibration is refined only during inference, PSNR rises from $0.8159$ to $0.8711$ while sync error drops from $11.78$ px to $8.06$ px.
Table 4 also reports mean IoU improvements of +0.228 on AgiBot (130 k episodes) and +0.212 on DROID (63 k episodes), with over 97 % of episodes benefiting.
The embodied video reward model attains an average accuracy of 86.33 % and macro‑F1 of 74.21 % on the held‑out validation benchmark.
Evaluated on 4,893 manipulation clips across three defect dimensions (L1–L3).
Compared to PointWorld, our RGB‑only approach outperforms on 75.5 % of 38 k DROID episodes, despite PointWorld’s reliance on stereo depth.
Calibration and Training Details
We evaluate how each implementation component affects action‑faithful video prediction.
The paper’s core claim is that aligning rendered robot actions with video data improves both action‑faithfulness and physical plausibility. To verify which parts of the pipeline drive this gain, we ablate each component and measure the resulting drop in performance.
Calibration refines camera intrinsics, extrinsics, distortion, and arm offsets so that rendered robot masks line up with real‑world observations.
The preprocessing pipeline stores, for every frame and view, a set of channels that become the conditioning input to the video generator.
Gripper opening widths are encoded as pixel intensities so the generator can perceive grasp state directly from the RGB condition.
To teach the model physically plausible interactions, we perturb the demonstrated end‑effector pose and regenerate the entire trajectory.
**Figure 9.** Geometric calibration results. (a,b) SAM3 masks (green) overlaid with Original (red) and Ours (blue) contours. (c–e) IoU comparisons against Original and PointWorld, using AgiBot’s head view and DROID’s two external view mean. Points above the diagonal favor Ours.
Embodied Video Reward Model
We introduce a reward model that scores video realism across embodiment, object, and interaction defects.
The reward model addresses the core pain of evaluating video realism: existing metrics miss subtle embodiment, object, and interaction flaws. By turning these flaws into explicit defect probabilities, the model provides a direct signal for post‑training refinement.
It scores a generated video by predicting three binary defect labels—embodiment, object, and interaction—and rewarding low defect probabilities.
Compute raw rewards: $R_1=-0.20$, $R_2=-0.10$, $R_3=-0.05$.
Apply GDPO normalization (here each channel is scaled by its own max, which is 1, so values stay unchanged).
Sum the normalized rewards: $R_{\text{total}} = -0.20 - 0.10 - 0.05 = -0.35$.
Because the reward is the negative of defect probability, any increase in predicted defect directly lowers the total reward, driving the generator toward videos with fewer observable flaws.
How does this reward differ from a standard classification loss that maximizes the likelihood of the correct label?
Standard classification encourages the model to assign high probability to the observed label, but it does not penalize high defect probabilities when the label is “none”. By using the negative defect probability as the reward, the model is explicitly discouraged from assigning any probability mass to the defective class, even if the ground‑truth label is “none”. This creates a stronger gradient toward defect‑free generations.
**Figure 4.** Representative defects across three dimensions: robot embodiment (L1), object consistency (L2), and interaction plausibility (L3).
The reward‑model corpus mixes ground‑truth videos with predictions from both recorded and counterfactual actions. Training uses 31 600 videos, validation 4 893, split by source‑episode identifier to avoid leakage.
Performance is measured with a probability‑based mean absolute error: $\text{MAE}_k = \frac{1}{N}\sum_{i=1}^{N}\bigl|p_k^{(i)} - y_k^{(i)}\bigr|$, where $N=4{,}893$ validation videos and $y_k^{(i)}$ is the binary defect annotation.
For downstream evaluation we also report the normalized Dynamic Time Warping (nDTW) action‑following metric, which quantifies how closely the generated gripper trajectory matches the recorded one, evaluated separately for each defect dimension.
Evaluation Benchmarks and Baselines
We detail the evaluation protocols, human annotation process, and benchmark results.
This section describes how we evaluate all methods, how human annotators assess counterfactual videos, and the quantitative results that support our claims.
A standardized test that measures how well a model predicts video outcomes when actions differ from those actually taken.
A simulation environment that provides photorealistic robot renderings and precise geometric information for training and evaluation.
A minimal toy world where a single point moves according to prescribed actions, used to sanity‑check the model’s basic dynamics.
Baseline protocol: every method receives the same initial observation, task instruction, and counterfactual action semantics via its native conditioning interface. Human annotators (three per video) label Embodiment, Object, and Interaction defects on 960 counterfactual videos, and a defect is recorded if at least two annotators agree.
**Table 8.** Inter-annotator agreement on the 960 counterfactual videos used in Table 1. Observed agreement is the mean fraction of agreeing annotator pairs per video.
**Table 9.** Majority-vote counterfactual defect rates with Wilson 95% confidence intervals. Each rate uses the same 160 conditions as Table 1.
A video‑prediction model that learns from large‑scale robot video datasets without explicit geometric supervision.
The simulation platform that provides photorealistic robot renderings and precise geometric calibration for training world models.
A vision‑language model that generates future video frames conditioned on textual instructions.
An action‑conditioned video model that incorporates explicit physics constraints during training.
The proposed model without the reward‑based post‑training stage.
The full proposed pipeline, including VACE alignment and Counterfactual Post‑Training.
Policy evaluation (C.2) uses three dual‑arm manipulation tasks from RoboTwin 2.0 under Easy and Hard visual settings. We freeze 60 evaluation scenarios (10 seeds × 3 tasks × 2 settings) and benchmark four open‑source VLA policies—EventVLA, $\pi$0.5, X‑VLA, and starVLA—each run for 240 episodes total.
Generalization and OOD Performance
OOD rollouts show our model cuts defect rates and boosts outcome classification.
Our model dramatically lowers defect rates and yields more accurate outcome predictions than the baseline.
Table 9 shows zero embodiment and interaction defects for the full model; Table 10/11 report MAE 7.08 pp vs 12.92 pp and accuracy 0.904 vs 0.838.
Action‑conditioned rollouts are evaluated by blind human annotators who vote on task‑success criteria such as block placement and handover integrity. The full model’s counterfactual defect rates are zero for embodiment and interaction, outperforming all baselines.
**Figure 7.** Prediction across embodiments and environments. A shared checkpoint generates rollouts on 4 datasets and generalises without additional fine-tuning to (e) unseen real-world scenes and (f) an unseen robot embodiment (WidowX250).
**Table 11.** Detailed outcome confusion matrix.
**Table 11.** Episode-level outcome classification breakdown across 240 policy rollouts (96 ground-truth successes, 144 ground-truth failures).
Qualitative Rollout Examples
Cross‑embodiment rollouts showcase 24 videos across four benchmarks.
We provide 24 cross‑embodiment rollout videos spanning four training benchmarks.
Figures 13–16 contain six rollouts per benchmark, totaling 24 qualitative clips.
**Figure 13.** Cross-embodiment rollouts: AgiBotWorld-Beta. Six predicted rollouts, tagged (a)–(f), each pairing synchronized head (top), left-gripper (middle), and right-gripper (bottom) views, covering beverage pouring (a, b), object carrying (c), and checkout scenes (d)–(f). Columns progress forward in time, with elapsed video time in seconds below.
**Figure 14.** Cross-embodiment rollouts: DROID. Six real-world Franka rollouts, tagged (a)–(f), with synchronized left-exterior (top), right-exterior (middle), and gripper (bottom) views, spanning diverse scenes and collection sites. Columns progress forward in time, with elapsed video time in seconds below.
**Figure 15.** **Cross-embodiment rollouts: RoboMIND 2.0.** Six predicted rollouts, tagged (a)–(f), each pairing synchronized head (top), left-gripper (middle), and right-gripper (bottom) views, covering dual-arm toy handover (a), inter-arm tape passing into a tray (b), open-box toy placement (c), drawer manipulation (d), color-constrained block placement (e), and dual-arm capacitor assembly (f). Columns progress forward in time, with elapsed video time in seconds below.
**Figure 16.** **Cross-embodiment rollouts: RoboTwin 2.0.** Six simulated rollouts, tagged (a)–(f), each pairing synchronized head (top), left-gripper (middle), and right-gripper (bottom) views, covering hammering (a), color ranking (b), roller grasping (c), block handover (d), can-and-pot transport (e), and A-to-B placement (f). Columns progress forward in time, with elapsed video time in seconds below.