Learning the 4D World Through Point Trajectory Forecasting

Byungwoo Jeon, Sunghwan Hong, Dohyeon Kim, Jonghoon Lee, Jungwoo Park, Inhee Lee, Doosan Baek, Jinil Kim, Taehyun Lim, Hanbyul Joo, Marc Pollefeys, Saining Xie, Jinwoo Shin

VPM improves multi-view video forecasting and robot control by jointly predicting future 3D point trajectories.

How can we improve multi-view video forecasting by using 3D point trajectories as a supervisory signal?

Multi-view video models often struggle to maintain 3D consistency across cameras, leading to forecasts that diverge in perspective or fail to track physical objects coherently over time. The Video-Point Model (VPM) addresses this by jointly forecasting future videos and persistent 3D point trajectories, using explicit point-motion supervision to anchor appearance changes to a shared 3D representation. On the new VP-WorldBench, this point-supervised approach consistently improves both visual fidelity and 3D motion accuracy compared to video-only baselines, while also raising success rates in robot manipulation tasks.

Paper Primer

VPM integrates multi-view video histories into a shared sequence, using ray-based camera embeddings and a 4D rotary positional encoding to distinguish view identities while maintaining spatial and temporal alignment. The core mechanism is a trajectory forecasting head that samples intermediate generator features to predict 3D point motion, effectively forcing the model to learn a representation that respects physical 3D movement: the trajectory head acts like a motion-tracking anchor, pulling the video generator toward physically consistent configurations across all views.

Point supervision significantly improves multi-view forecasting fidelity and geometric consistency.

VPM (V+T) outperforms the video-only variant (V) across all simulated and real-world splits on VP-WorldBench. Consistent gains in both FVD (visual fidelity) and 3D motion accuracy metrics.

Point-supervised representations enhance robot policy success rates.

In both simulated and real-world manipulation experiments, the action-enabled VPM (V+T+A) consistently outperforms models trained without point supervision. Higher task success rates across all tested manipulation difficulties.

Why is point trajectory forecasting more effective than standard video-only prediction?

Standard models often lack an explicit mechanism to ensure that appearance changes in one camera correspond to the same physical motion in another. By forcing the model to predict where specific 3D points will move, VPM anchors the generated video to a coherent 3D scene structure, ensuring that the same physical point remains consistent across views and time.

What is the scope of this model — does it work for any scene?

VPM is currently trained on tabletop manipulation and urban dynamics. While it generalizes to unseen environments, it relies on known camera intrinsics and poses at inference, and its computational cost scales with the number of views because all camera streams share a single token sequence.

Researchers should treat 3D point trajectories as a primary geometric supervisory signal for world models, as they provide a dense, physically grounded bridge between visual generation and actionable robot control.

Introduction and Motivation

We expose the geometric inconsistency problem in multi‑view video forecasting and outline our VPM solution.

Multi‑view video forecasting requires that videos from different cameras describe the same evolving 3D scene, but existing models often produce geometrically inconsistent predictions. Without a shared 3D grounding, appearance changes across views cannot be reliably coordinated.

We introduce the Video‑Point Model (VPM), a world model that conditions on multi‑view video histories and a language instruction to jointly forecast future videos from arbitrary viewpoints and persistent 3D point trajectories, using the latter as a structural supervisory signal. To train and evaluate this model we build VPM‑Data, a simulation corpus with synchronized multi‑view videos and point trajectories, and VP‑WorldBench, a benchmark spanning simulated and real environments.

Our contributions are threefold: (1) the VPM architecture that jointly predicts videos and point trajectories, (2) VPM‑Data and VP‑WorldBench providing aligned visual and geometric supervision, and (3) extensive experiments showing that point supervision improves video fidelity, motion accuracy, and robot action success.

**Figure 1. Teaser.** We introduce simulated and real-world multi-view videos paired with 3D point tracks (left), jointly forecast future videos and point trajectories (center), and show that point supervision improves visual forecasting (right).

Geometric consistency is essential for reliable multi‑view forecasting; point‑trajectory supervision provides the needed 3D grounding.

The Video-Point Model

The method jointly predicts videos and 3D point trajectories using camera-aware embeddings and a unified loss.

Multi‑view video forecasting often drifts geometrically because the model lacks a concrete spatial anchor across views.

VPM predicts future video frames and the 3D trajectories of points selected from the observed history, using the trajectories as a structural supervisory signal that ties the generated pixels to a consistent world geometry.

How does VPM differ from a standard video‑prediction model that only minimises pixel loss?

Standard models learn to reproduce frames without any notion of 3D structure, so they can generate visually plausible but geometrically inconsistent videos. VPM adds a trajectory head and a trajectory loss that forces the latent representation to encode consistent 3D point motion, anchoring the video generation to a shared world model.

Each video token is enriched with a lightweight encoding of its camera’s geometry, allowing attention to be modulated by the actual 3D ray that the token represents.

Why rotate queries and keys instead of only adding the camera embedding?

Adding the embedding changes the token’s content but leaves the similarity computation unchanged. Rotating queries and keys directly alters the dot‑product scores, making the attention mechanism itself aware of the camera’s pose, which is essential for aligning features across views.

We extend the standard 3‑D rotary encoding with a fourth “view” dimension so that tokens from different cameras occupying the same spatio‑temporal cell are still distinguishable.

The head repeatedly refines 2‑D pixel positions and depths for each queried point, then lifts them into the fixed reference frame, yielding persistent 3‑D trajectories.

Iteration 1: the head predicts $u_{1}= (0.2,0.3)$, $d_{1}=1.0$ for point 1 and $u_{2}= (0.6,0.4)$, $d_{2}=0.9$ for point 2.

Unprojecting yields $P_{1}^{(1)}=(0.2,0.3,1.0)$ and $P_{2}^{(1)}=(0.6,0.4,0.9)$.

Iteration 2: using the updated features, the head refines to $u_{1}= (0.22,0.31)$, $d_{1}=0.98$ and $u_{2}= (0.58,0.42)$, $d_{2}=0.92$.

Unprojected positions become $P_{1}^{(2)}=(0.22,0.31,0.98)$ and $P_{2}^{(2)}=(0.58,0.42,0.92)$.

This toy example shows how the head gradually corrects both pixel location and depth, converging toward the true 3‑D trajectory despite starting from noisy latent features.

The loss combines a robust position error, a depth error, and a confidence term, encouraging accurate and stable trajectory predictions across refinement steps.

Why use a truncated Huber loss instead of a plain L2 loss for position and depth?

Because the Huber loss behaves like L2 near zero error (providing smooth gradients) but switches to L1 for large residuals, limiting the influence of occasional large mistakes that arise from noisy intermediate features.

We directly supervise the attention maps so that the same physical point receives high similarity scores across all views and timesteps.

How does correspondence supervision differ from the trajectory loss?

Trajectory loss penalises the final 3‑D positions after unprojection, while correspondence supervision directly shapes the intermediate attention maps that produce those positions, ensuring the features themselves respect cross‑view geometry.

Training proceeds in three stages: first we optimise the video generation loss $L_{\text{FM}}$ on all multi‑view sources; next we freeze the backbone and train only the trajectory head with $L_{\text{track}}$; finally we jointly fine‑tune the video backbone and trajectory head with the combined forecast loss $L_{\text{forecast}} = L_{\text{FM}} + \lambda_{\text{track}} L_{\text{track}} + \lambda_{\text{corr}} L_{\text{corr}}$.

At inference time we condition on the observed histories, camera parameters, and the language instruction, then denoise future video tokens from $\sigma\!=\!1$ to $0$ while decoding trajectories from the backbone features, finally transforming them into the fixed reference frame.

**Figure 2.** Overview of VPM. From posed multi-view histories and a task instruction, VPM forecasts future videos and the trajectories of points queried from observed frames. The 3D point tracker iteratively refines point positions and depths before lifting them to 3D. Joint point supervision shapes the generator’s representation, and the action-enabled extension also outputs robot action chunks.

Datasets and Evaluation

Introducing VPM-Data and VP-WorldBench, the new dataset and benchmark for multi‑view video forecasting.

This section describes the data that trains the Video‑Point Model and the benchmark used to evaluate it. We first detail the synthetic VPM‑Data, then introduce VP‑WorldBench, which measures both video quality and 3‑D trajectory fidelity across diverse scenes.

VP‑WorldBench is a multi‑scene benchmark that jointly evaluates future video frames and persistent 3‑D point trajectories, forcing models to stay geometrically consistent while predicting visual content.

**Figure 3.** Qualitative examples of VPM-Data and VP-WorldBench.

**Table 2.** Results on VP-WorldBench. Numbers in parentheses denote clips per split. V and T denote video and point-trajectory supervision.

VP‑WorldBench demonstrates that jointly training on video and 3‑D trajectories yields consistent improvements across all evaluation metrics.

Robot Action Generation

VPM (V+T+A) sets new robot manipulation success rates while boosting video fidelity.

VPM (V+T+A) achieves the highest robot manipulation success rates across all simulated and real‑world benchmarks.

Table 5 shows a 92% overall success rate, beating the next best baseline by 8%.

Action tokens are treated as another modality that the shared predictive backbone denoises together with video and point‑trajectory tokens.

**Figure 4.** **Predicted 3D point trajectories guide generation and action.** Left: Predicted trajectories alone and overlaid on generated videos. Right: Zero-shot real-robot rollout obtained by converting predicted trajectories into commands with inverse kinematics.

Ablation Studies

We ablate VPM’s components to quantify their impact on video fidelity and 3D motion accuracy.

Recall that VPM jointly predicts future video frames and persistent 3D point trajectories, using the trajectories as a structural supervisory signal.

F.1 removes the three camera‑conditioning components (Additive Plücker embedding, camera‑dependent rotation, and view‑axis RoPE) to test their contribution.

Removing all three camera‑conditioning components severely degrades performance, especially on the Simulation‑General split.

FVD rises to 140.06 and $\delta$³ᴰ avg falls to 45.79 on that split.

F.2 ablates the point‑supervision objectives: 2‑D position loss $L_{seq}$, depth loss $L_{depth}$, and cross‑view correspondence loss $L_{corr}$.

Using the full set of objectives yields the best results; dropping $L_{corr}$ hurts video fidelity.

FVD increases from 91.92 to 101.37 on the Simulation‑General split when $L_{corr}$ is omitted.

F.3 evaluates the three‑stage curriculum by removing Stage 1, Stage 2, or both while keeping the total iteration budget fixed.

Removing Stage 2 causes the largest drop in 3D motion accuracy.

$\delta$³ᴰ avg falls to 49.77 from 64.75 on the Real‑world Robot split.

F.4 varies the self‑attention layer ℓ (6, 12, 18, 24) from which the trajectory head reads query and key features.

F.5 varies the number of refinement iterations $M$ and removes each attention type (spatial, temporal, cross‑view) from the refiner.

Cross‑view attention is the most critical component; its removal degrades both metrics.

FVD increases by ≈ 11.3 % and $\delta$³ᴰ avg drops to 29.89 on the Simulation‑General split.

**Table 31.** Ablation on the generator layer. The trajectory head reads query and key features from the $l$-th self-attention layer of the generator.

**Figure 10.** Ablation on training data sources. Relative FVD increase (a) and $\delta_{\text{avg}}^{\text{3D}}$ decrease (b) of VPM (V+T) when one source is removed from training, measured against the model trained on all sources. Higher is worse.

Additional Experiments

VPM (V+T) with camera conditioning dominates 2‑view I2V benchmarks on VP‑WorldBench.

VPM (V+T) with camera conditioning outperforms all baselines on the 2‑view I2V task.

Table 17 shows it achieves the lowest FVD (60.14) and highest PSNR (85.51) among compared methods.

**Table 17.** Results on VP-WorldBench (I2V with 2-view). V and T denote video and point-trajectory supervision. Best results are in bold. The upper bound is obtained by applying the same evaluation pipeline to the ground-truth videos.

Data Ablations

Ablation study quantifies how each training source impacts video quality and 3‑D trajectory accuracy.

We evaluate the contribution of each training source by training VPM (V+T) while omitting one source at a time and measuring the change in Fréchet Video Distance (FVD) and average 3‑D trajectory error ($\delta_{3D}$ avg) relative to the full‑data model.

Removing VPM‑Data increases FVD by 48.3 % on the Simulation General split.

When VPM‑Data is omitted, the model’s FVD on Simulation General rises from the baseline to +48.3 %.

Removing VPM‑Data decreases $\delta_{3D}$ avg by 9.16 points on the Simulation General split.

Omitting VPM‑Data reduces the average 3‑D trajectory error on Simulation General by –9.16 points compared to the full‑data model.

Removing the robot‑manipulation datasets increases FVD by 18.2 % on the Real‑world Robot split.

Excluding the robot datasets raises the Real‑world Robot FVD by +18.2 % relative to the baseline.

Removing the robot‑manipulation datasets decreases $\delta_{3D}$ avg by 12.39 points on the Real‑world Robot split.

Without the robot datasets, the average 3‑D trajectory error on Real‑world Robot drops by –12.39 points.

Removing the open‑source videos increases FVD by 12.2 % on the Real‑world General split.

Omitting open‑source videos leads to a +12.2 % rise in FVD for Real‑world General compared to the full model.

Removing the open‑source videos decreases $\delta_{3D}$ avg by 6.73 points on the Real‑world General split.

Excluding open‑source videos reduces the average 3‑D trajectory error on Real‑world General by –6.73 points.

All splits suffer when any source is omitted, confirming that VPM‑Data, robot datasets, and open‑source videos provide complementary signals for both video fidelity and 3‑D trajectory grounding.

Implementation Details

Implementation details: training pipeline, hyperparameters, hardware setup, and robot control via predicted trajectories.

We train the Video‑Point Model (VPM) using the Cosmos3‑Nano backbone and a Mixture‑of‑Transformers architecture. Training proceeds in three stages, each focusing on a different component of the system.

Cosmos3‑Nano is a lightweight vision transformer that processes video frames efficiently while preserving spatial detail.

The architecture routes inputs through several specialized transformer branches and blends their outputs, allowing the model to handle diverse visual patterns.

Stage 1 trains the generation pathway, camera embeddings, and view‑frequency modules with the LFM loss on all views (including unannotated ones). Stage 2 freezes the backbone and trains only the trajectory head with Ltrack. Stage 3 jointly optimizes video generation and trajectory prediction with Lforecast, while the reasoner remains frozen throughout.

**Table 7.** Hyperparameters for each training stage.

**Figure 5.** Real-world hardware platform. A 7-DoF Franka Research 3 robot arm equipped with a Robotiq 2F-85 gripper and a wrist-mounted Zed Mini stereo camera. Two external Zed 2i stereo cameras (view 1 and view 2) observe the workspace from different viewpoints, providing synchronized multi-view observations.

**Figure 6.** Examples of the robot manipulation tasks.

**Figure 7.** **From predicted point trajectories to joint-space actions.** Predicted 2D trajectories from both views are lifted to 3D (Stage 1) and converted into a joint trajectory by fitting the robot's kinematic model to the lifted points (Stage 2).

Evaluation follows two protocols. For Image‑to‑Video (I2V) we condition on a single frame; for Video‑to‑Video (V2V) we condition on $h' = 16$ frames of multi‑view history and predict $h = 64$ future frames. Metrics include FVD and PSNR for video fidelity, and cross‑view MEt3R and $\delta_{3D}^{\text{avg}}$ for geometric consistency.

Robot manipulation is assessed on LIBERO, LIBERO‑Plus, and RoboCasa Kitchen benchmarks. Each pick‑and‑place trial receives a score of 1 (perfect), 0.5 (minor collision), or 0 (failure). Success rates are averaged over 24 trials per task.

To test whether predicted trajectories alone are sufficient for control, we implement a zero‑shot two‑stage procedure that maps trajectories directly to joint commands without any action‑token supervision.

Zero‑shot joint‑space control from predicted point trajectories.

Dataset Curation

We detail how 600 K multi‑view samples are assembled and how 3‑D point trajectories are extracted.

We construct 600 K training samples from three sources: VPM‑Data, robot manipulation videos with kinematics‑grounded trajectories, and open‑source video collections such as MolmoMotion‑1M and Ego‑Exo4D.

Overall we obtain 604 913 samples, slightly exceeding the nominal 600 K target because of rounding in the source splits.

Mesh Sampling – surface points are drawn once per mesh using area‑weighted triangle sampling with a fixed random seed; the same points are reused throughout the episode.

Point Budget – 256 points are allocated to the robot body and, when possible, 256 points to the target object; budgets are split proportionally to mesh surface area for articulated parts.

World‑Space Transformation – each sampled point $p^{\ell}_n$ on link or object $\ell$ is transformed to world coordinates at timestep $\tau$ via $\mathbf{x}_{\tau,n}=R^{w}_{\ell,\tau}\,p^{\ell}_n + \mathbf{b}^{w}_{\ell,\tau}$.

Camera‑Frame Projection – the world‑space point is projected into camera $v$ as $\mathbf{x}^{v}_{\tau,n}=R^{v\leftarrow w}_{\tau}\,\mathbf{x}_{\tau,n}+ \mathbf{b}^{v\leftarrow w}_{\tau}$, yielding explicit correspondences across time and viewpoints.

For robot data we filter episodes by geometric alignment: we render a binary CAD mask $M_{\text{CAD}}$ using the recorded joint configuration and camera calibration, then select the frame $\tau^{\star}$ with the largest projected robot area.

We compare $M_{\text{CAD}}^{\tau^{\star}}$ to an image‑derived mask $M_{\text{SAM}}$ (from SAM2) and compute recall $\text{Recall}_{\text{CAD}} = \frac{|M_{\text{CAD}}^{\tau^{\star}} \cap M_{\text{SAM}}|}{|M_{\text{CAD}}^{\tau^{\star}}|}$. Episodes with recall ≥ 0.9 are retained, discarding those with poor alignment.

Each final sample stores synchronized multi‑view RGB frames, persistent 3‑D surface‑point trajectories for the robot and objects, joint states, and full camera metadata, guaranteeing that point indices refer to the same physical surface across time and viewpoints.

Training Data Generation

How training clips are captured, annotated, and validated for both domains.

We use separate pipelines for tabletop manipulation and urban capture, sharing a common annotation format. The capture settings for each domain are summarized in Table 9. Training data are generated by replaying physics‑simulated robot episodes for tabletop and by sampling dynamic agents for urban scenes.

**Table 9.** Training-data capture settings. Both domains use 1280x1280 images and distortion-free pinhole cameras.

Tabletop generation replays physics‑simulated pick‑and‑place episodes of a robot using recorded joint configurations. Each clip contains 1–4 target objects, 5–8 non‑contact distractors, and 0–2 contact distractors drawn from a library of 306 props, all placed to avoid intersecting the robot’s swept volume.

Urban capture places the camera rig at one of 13 surveyed locations in the C2 environment. After traffic and crowd simulation stabilizes, a visible pedestrian or vehicle is selected as the target, and each camera samples a horizontal field of view between 25° and 90° while following a trajectory type from Table 9.

**Table 10.** **Hard-clip criteria for the training pipelines.** An invisible interval comprises frames labeled occluded or out of frustum. Benchmark definitions and measured statistics are reported separately in Section D.

Geometric annotations consist of 256 persistent surface points per object, with tabletop clips also containing 512 robot points allocated proportionally to link surface area. Visibility is evaluated using rendered depth (3 cm tolerance) for tabletop and surface‑point ray casting (up to 32 points per camera) for urban, requiring at least three unoccluded points for a target to be considered visible.

Captions are generated by a vision‑language model, filtered to match recorded motion, limited to a single sentence, and regenerated up to two times if validation fails. Hard‑clip construction differs between domains: tabletop clips use camera jitter and occluder placement, while urban clips enforce a two‑phase departure/re‑acquisition schedule, discarding failed attempts.

**Table E.3.** Comparison with per-frame geometry supervision

During rendering, RGB frames are encoded as H.264 and depth maps as 16‑bit PNGs (1 cm–200 m range). Per‑camera metadata includes intrinsics and world‑to‑camera matrices. Random seeds are stored per clip for tabletop; urban seeds are recorded but the live simulation does not guarantee bit‑exact regeneration, so validation relies on the acceptance checks rather than exact replay.

VP-WorldBench Construction

Details of the VP-WorldBench simulation benchmark, its composition, capture pipelines, and validation.

D.1 Composition and capture settings. The simulation benchmark comprises 500 tabletop and 500 urban clips, totalling 661 071 images and 716 544 persistent 3‑D point trajectories. Images are 1280 × 1280 and captured with distortion‑free pinhole cameras; timestamps are synchronized across all views.

**Table E.5** Performance on Single-View World Modeling

D.2 Tabletop capture. Each clip replays a physics‑simulated pick‑and‑place episode in Unreal Engine, with three fixed cameras perturbed in distance (0.94–1.06×), azimuth (±4°), and elevation (±3°). Randomized distractors (0–2 intersecting the robot’s swept volume) are added, and the final clips contain a median of eight distractors.

D.3 Urban capture. Urban clips are recorded in an open‑source driving simulator built on Unreal Engine, using six city tiles and 60 building models under fixed noon lighting. Cameras sample horizontal fields of view between 25° and 90°, follow diverse motion patterns, and are limited to 150 cm translation per frame.

D.4 Ground‑truth annotations. For each object we sample 256 surface points (256 × object) and preserve their identities across frames; the robot receives 512 additional points distributed over its links. Trajectories are stored in the reference camera’s frame using the OpenCV convention (+X right, +Y down, +Z forward).

D.5 Visibility‑controlled subsets. Hard subsets deliberately induce sustained visibility loss in individual views while keeping the target visible in at least one camera. For tabletop clips, one camera is placed low (8–12 cm above the surface) to be occluded by the robot’s arm; for urban clips, cameras depart and re‑acquire the target in two phases.

**Table 13.** Hard-subset criteria and composition. An invisible interval includes occluded and out-of-frustum frames. Urban collective coverage is evaluated on the strict array.

**Table 14.** Percentage of frames by the number of cameras seeing the target. Urban columns use bounding-box visibility; strict-array collective coverage is reported separately.

D.6 Validation and limitations. Tabletop clips undergo extensive checks for point‑depth agreement (≤ 3 cm median error), camera‑pose consistency, and caption availability. Urban clips are validated against thresholds such as target visibility ≥ 0.1 across cameras and motion persistence ≥ 40 cm/s in at least 40 % of frames.

**Table E.8** Clock-time analysis. Setup. We measure the wall-clock time to generate one clip under the I2V protocol of Table 2, i.e., 64 future frames at 480p per view from a single conditioning frame, on one NVIDIA GH200 GPU with BF16 precision and batch size 1. VPM (V+T+A) additionally denoises 16 action chunks.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers