Self-Supervised Learning of Structured Dynamics from Videos

Lukas Knobel, Andrew Zisserman, Yuki M. Asano

SDM decomposes video dynamics into primary and residual motion tokens using frozen image backbones.

How can we decompose video motion into camera and object dynamics using only frozen image features and weak scene-level supervision?

Video analysis is hindered because frame-to-frame changes entangle camera motion and object movement, making it difficult to isolate meaningful dynamics without dense geometric supervision. The Structured Dynamics Model (SDM) addresses this by predicting future features from frozen image backbones, using a two-stage recurrent architecture that explicitly separates primary global motion from residual object-centric dynamics. Across the ProbeMotion benchmark, SDM outperforms standard frozen-feature baselines and competes with models trained on extensive 3D geometric supervision.

Paper Primer

SDM functions as a structured predictor: it first compensates for dominant global motion using a primary token, then uses a residual token to account for remaining object-level discrepancies. This iterative refinement mirrors coarse-to-fine estimation, allowing the model to adapt its tokens to the specific motion regime of each video.

SDM achieves competitive performance against strongly supervised 3D models despite using significantly weaker supervision.

SDM surpasses supervised baselines on 3 out of 7 ProbeMotion benchmarks, including object-motion tasks on Kubric and static YouTubeVOS. SDM improves over the strongest supervised baseline by 0.13 MSE on Kubric object motion and 0.04 MSE on static YouTubeVOS.

Structured motion tokens outperform naïve global-feature baselines.

SDM consistently beats CLS and average-pooled feature baselines across ProbeMotion tasks, including an 8.2 percentage point improvement on SSv2-110k action recognition. SDM outperforms the self-supervised DeltaTok baseline on 5 of 7 tasks, with a 9.6 percentage point gain on SSv2-110k.

Why is explicit decomposition into primary and residual tokens necessary?

Standard models often summarize scene change with a single latent token, which mixes dominant global motion with residual object dynamics. Explicitly separating these allows the model to organize temporal change into interpretable physical factors.

What is the scope of this approach?

SDM is designed for feature-space future prediction using frozen image backbones. It is most effective on temporally coherent benchmarks where motion regimes are consistent, though it remains robust across both synthetic and real-world video datasets.

The Challenge of Entangled Motion

We expose and address the entanglement of camera and object motion in video.

Video frames change because two distinct dynamics are at play: the camera moves through the scene and objects move within it. These two sources become tightly coupled, so a model that sees only the raw pixel difference cannot tell which motion belongs to the camera and which belongs to objects.

These are the fixed visual embeddings that a pretrained image transformer extracts from each frame; they capture appearance and geometry but do not change across time.

We introduce the Structured Dynamics Model (SDM), a recurrent architecture that predicts future frozen features by first accounting for the dominant motion (Primary Motion) and then modeling the leftover changes (Residual Motion). This two‑stage compensation explicitly disentangles camera‑induced change from object‑centric dynamics.

SDM is trained with a hybrid objective: self‑supervised future‑feature prediction on unlabeled real video, plus weak supervision on synthetic Kubric clips that indicate whether the camera or the scene is static. The weak signal guides the model to allocate the bulk of change to Primary Motion when the camera moves, and to Residual Motion otherwise.

We evaluate SDM on ProbeMotion, a suite of benchmarks covering pure camera motion, pure object motion, and their combination in both synthetic and real videos. Across all settings SDM beats naïve baselines that use a single global CLS token or average‑pooled features, and it matches or exceeds the strongly supervised VGGT representations on three of seven probes.

The core problem is that camera motion and object motion are entangled in video, preventing clean motion representations.

The Structured Dynamics Model

SDM splits video change into dominant primary motion and a residual component to disentangle dynamics.

Predicting future video features with a full video encoder is costly, and entangled camera‑object motion makes a single prediction token insufficient.

We first extract frozen visual features $f_t = E(x_t)$ from a pretrained vision transformer $E$, keeping the backbone fixed while the motion model operates on these maps.

SDM first removes the dominant change (primary motion) from the source feature, then fine‑tunes the remainder (residual motion) – like first fixing a gross misalignment before polishing the leftover error.

Primary extractor computes $p_t$ as the average difference: $p_t = \frac{1}{4}\sum (f_t - f_{t-1}) = +1$.

Primary predictor adds $p_t$ to every entry of $f_{t-1}$, yielding $\hat{f}_t = f_{t-1} + p_t = \begin{bmatrix}2&3\\4&5\end{bmatrix}$.

Residual extractor sees zero mismatch ($\hat{f}_t = f_t$) and outputs $r_t = 0$.

Residual predictor adds $r_t$ (no change), so $\ddot{f}_t = \hat{f}_t$ matches the target exactly.

This toy shows that when a single dominant motion explains the whole transition, the residual stage becomes idle, confirming that the two‑stage design does not over‑constrain the model.

How does SDM differ from a standard recurrent network that directly predicts $f_t$ from $f_{t-1}$?

Standard recurrence collapses all change into one hidden state, mixing camera and object dynamics. SDM explicitly separates the dominant (primary) motion from the leftover (residual) motion, giving each a dedicated token and predictor, which yields interpretable motion representations.

Primary motion captures the biggest, usually global, transformation (e.g., camera shift), while residual motion accounts for the finer, local changes that remain.

Why not use a single motion token instead of separating primary and residual?

A single token must encode both global and local dynamics simultaneously, which forces the model to compress heterogeneous information and makes the token hard to interpret. The two‑stage split lets each token specialize, improving both prediction accuracy and downstream probing.

Extract primary token $p_t = \phi_p(p_{t-1};\,f_{t-1},\,f_t)$.

Compensate source feature: $\hat{f}_t = \psi_p(f_{t-1};\,p_t)$.

Extract residual token $r_t = \phi_r(r_{t-1};\,\hat{f}_t,\,f_t)$.

Refine prediction: $\ddot{f}_t = \psi_r(\hat{f}_t;\,r_t)$.

Use $(p_t, r_t)$ as the structured motion representation for downstream tasks.

Implementation uses a frozen DINOv2‑B/14 backbone, concatenates multi‑layer patch features, and projects them to $D\!=\!768$ before feeding into SDM; both extractors and predictors are shallow transformer decoders with 4 blocks and 4 learnable registers, employing 3D RoPE for spatial‑temporal positions.

**Figure 1.** The Structured Dynamics Model (SDM) remodels frozen image features into structured motion tokens. Left: A frozen image encoder extracts frame features, and a recurrent SDM is trained by self-supervised future-feature prediction. SDM structures temporal change into a primary motion token $p$ for the dominant source of change and a residual motion token $r$ for remaining dynamics. Right: Linear probes on ProbeMotion show that these structured motion tokens outperform direct frozen-backbone descriptors such as CLS and average-pooled features, and even surpass VGGT probing performance on 3/7 tasks (see Table 2), while showing stronger generalization to motion-adjacent semantic action classification.

**Figure 2.** Overview of the structured dynamics model shown for T=3. For each transition, SDM predicts the target frozen feature map $f_t$ from the previous one $f_{t-1}$ through two sequential compensation stages. The primary stage extracts a recurrent motion token $p_t$ from the feature pair $(f_{t-1}, f_t)$ and uses it to produce a primary-compensated feature map $f_t^>$, explaining the dominant change. The residual stage extracts $r_t$ from the mismatch between $f_t^>$ and $f_t$, and refines $f_t^>$ into the final prediction $f_t^>>$. The motion tokens are autoregressive but non-causal across time, allowing temporal information to accumulate.

Experimental Setup and Data

We evaluate SDM on synthetic and real video data using the ProbeMotion benchmark.

We train SDM on a balanced mix of synthetic Kubric videos and real‑world SSv2 and DL3DV clips, then evaluate the learned motion tokens with the ProbeMotion suite.

Kubric tags each clip only with “camera static” or “scene static”, letting the model infer motion without per‑frame annotations.

How does weak scene‑level supervision differ from full per‑frame motion labels?

Full labels would specify the exact motion vector for every pixel at every time step. Weak supervision only tells the model whether the camera or the scene is moving for the whole clip, forcing it to discover the fine‑grained decomposition itself.

**Figure 6.** **Kubric examples.** Example sequences from our generated Kubric training data. The three subsets provide weak scene-level supervision for disentangling primary and residual dynamics: moving-camera/static-scene sequences contain only camera-induced motion, static-camera/dynamic-scene sequences contain only object motion, and moving-camera/dynamic-scene sequences combine both sources of motion. Examples are shown at 1 fps.

**Table 1.** **ProbeMotion evaluation datasets.** Summary of the linear-probe datasets. Numbers given for source videos refer to the samples taken from the corresponding public or generated dataset after clip-length filtering. Dashes indicate that the source dataset does not provide a split suitable for ProbeMotion. For these datasets, we create our own probing splits. Parentheses indicate partial dynamics for SSv2’s camera (see Section B.1). Further details, including split creation for these datasets, are provided in Section A.3.

Training mixes Kubric, SSv2, and DL3DV in a 0.5 / 0.25 / 0.25 ratio; within each batch 1/4 of samples receive static‑scene supervision, 1/8 static‑camera supervision, and the remaining 5/8 are trained with the standard prediction loss only.

All models are trained for 200 k AdamW steps (batch 128, lr 6.25 × 10⁻⁵, 1 k warmup, weight decay 0.05, $\lambda_{r}$eg = 0.5) on 4 H100 GPUs, taking roughly 11 h per run.

Evaluation probes the frozen SDM motion tokens with linear classifiers (for categorical targets) or linear regressors (for continuous motion), reporting the best held‑out metric over three random seeds.

The model is trained on a mix of synthetic (Kubric) and real‑world (SSv2, DL3DV) data.

Baseline Comparisons

SDM beats all baselines, delivering up to 9.6 pp higher accuracy and 0.16 MSE lower error.

We evaluate SDM against a spectrum of baselines: simple frozen‑backbone descriptors, the self‑supervised DeltaTok, and heavily supervised 3‑D models. All baselines are constructed from either CLS tokens or average‑pooled patch features, using concatenation or feature differences as described in the ablations.

SDM achieves a 9.6 percentage‑point higher accuracy on the SSv2‑110k action‑prediction probe than DeltaTok.

DeltaTok reaches 71.2 % accuracy while SDM reaches 80.8 % on the same probe.

**Table 12.** Additional structured-motion results. Full results with standard deviations showing that SDM learns specialized primary and residual motion tokens. The primary token captures the dominant motion source, while the residual token captures remaining dynamics. The only exception is static DAVIS2017, where the residual token outperforms the primary token. However, the high variability of this result makes it less reliable. We report MSE on standardized regression targets and accuracy in % for classification tasks. Among individual tokens, the best result is shown in bold. Representations and targets that match are highlighted in green. The last row reports an upper-bound probe using the concatenated tokens [p, r].

Quantitative Analysis

SDM’s accuracy climbs with longer temporal context, reaching 23.1% on SSv2‑110k.

SDM’s accuracy on SSv2‑110k improves monotonically with temporal context, reaching 23.1 % at T=7, a +6.4 % gain over the 2‑frame baseline.

Table 3 reports 16.7 % at T=2 and 23.1 % at T=7; the baseline AVG‑pooled method attains only 15.5 % at T=7.

Increasing temporal context also benefits regression tasks: on Kubric, camera‑motion MSE falls from 0.26 to 0.16 and object‑motion MSE from 0.22 to 0.19 when moving from T=2 to T=4. The multi‑frame AVG‑pooled baseline shows similar trends but never catches up to SDM’s single‑pair performance.

**Table 3.**

**Table 5.** SDM leads to structured motion representations where $p$ and $r$ specialize in primary and residual motion when the features are used to linearly regress or classify motion or action. We report MSE on standardized regression targets and accuracy in % for classification tasks. Among individual tokens, the best result is shown in bold. Representations and targets that match are highlighted in green. The last row reports an upper-bound probe using the concatenated tokens $[p, r]$.

Temporal Context and Stability

Temporal context improves motion prediction accuracy across datasets.

SDM achieves the lowest DL3DV MSE of 0.58 when using the DINOv3 backbone, outperforming the other three backbones.

Table 4 reports DL3DV MSE values of 0.58 (DINOv3), 0.62 (MAE), 0.65 (SigLIP), and 0.71 (DINOv2r).

Across backbones, the temporal‑context aggregation in SDM consistently improves both pose‑estimation error and downstream action recognition accuracy, confirming that the benefit is not tied to a specific feature extractor.

**Table 4.** **SDM works across backbones.** We report DL3DV MSE on standardized frame-to-frame pose deltas and CameraBench / SSv2-110k accuracy in %. MAE [16], SigLIP [44], and DINOv3 [29] use the base variant with patch size 16. DINOv2 with registers [8] (DINOv2r) uses patch size 14.

Evaluating Motion Specialization

We evaluate how SDM’s primary and residual tokens encode motion via linear probes.

Linear probing tests whether a representation linearly separates a target property by training a simple linear classifier on frozen features.

Table 5 shows that the primary token p excels at capturing camera‑induced motion, while the residual token r specializes in object‑level dynamics, as measured by linear probes.

**Figure 7.** Structured feature prediction. We visualize feature-map error to the target after primary and residual compensation of a CameraBench sample. Errors are highlighted in red. Compared to the error without any compensation, the primary-motion-compensated feature map reduces the error significantly for camera-motion-induced differences, particularly in the background. The residual compensation further decreases the residual error and compensates for the error introduced by scene dynamics.

**Figure 3.** Stage-wise structured feature prediction. We visualize feature-map error to the target after primary and residual compensation on a sample from DAVIS2017. Errors are highlighted in red. The primary stage corrects the dominant global change, including the camera-induced movement of the edges of the track. Notably, primary motion compensation increases the error around the independently moving car. The residual stage reduces this localized error.

**Figure 4.** **Motion extrapolation.** We repeatedly apply the last motion token from the observed frames to predict future feature maps on a sample from the DAVIS2017 test split. Visualization is done by PCA projection to RGB after normalization. Black boxes mark the last observed ground-truth object position, which is repeated for the extrapolated feature maps. The extrapolated object features follow the object's motion beyond its last observed position over short horizons and drift at longer horizons.

**Figure 5.** Latent motion swapping. We apply motion tokens from a source clip to the feature map of a target clip with a different object, both from the Kubric test set, and visualize the resulting features by PCA projection to RGB after normalization. Black boxes mark the initial object position. The target feature map follows the transferred motion over short horizons and drifts at longer horizons.

Related Work

We position SDM among self‑supervised encoders, video models, and 3D reconstruction approaches.

Self‑supervised image encoders such as DINOv2, AIM, and MAE produce strong frozen visual features. These features embed useful 3D cues and support recent reconstruction pipelines, yet they do not expose explicit temporal or motion information.

Self‑supervised video encoders learn spatio‑temporal representations directly from unlabeled footage, often via masked video modeling or latent predictive objectives. They aim for general video features, whereas SDM keeps the image backbone frozen and adds explicit primary and residual motion tokens.

Several works adapt image‑pretrained models to video by enforcing temporal feature consistency or using correspondence learning such as SiamMAE. Other studies train image encoders from scratch on long, uncurated videos. In contrast, SDM leaves the image backbone frozen and learns structured motion tokens on top.

Purpose‑built 3D reconstruction models like DUSt3R, VGGT, and CUT3R estimate camera pose, depth, and point clouds from strong 3D supervision, with recent extensions targeting dynamic scenes and 4D reconstruction. SDM does not reconstruct full 3D geometry; instead it uses weaker supervision to remodel frozen image features into primary and residual motion tokens.

Latent‑action and world‑model approaches predict future frames or pretrained feature representations, avoiding dense annotations and pixel‑level generation. Prior work typically encodes temporal change in a single latent action or spatial future tokens. SDM is closest to feature‑space prediction but explicitly separates motion into primary and residual components.

Additional Camera Motion Analysis

Additional quantitative analyses of motion, baselines, temporal context, and token specialization.

SDM outperforms all baselines on both regression and classification benchmarks.

Across the evaluated datasets, SDM achieves a regression mean‑squared error of 0.16 and a classification accuracy of 85.3 %.

The table compares camera motion metrics (rotation and translation) across the SSv2 dataset and three subsets of the CameraBench dataset: slow-speed, regular-speed, and fast-speed.

**Table 1.** Comparison of baseline and SDM methods across various datasets (Kubric camera, Kubric object, DL3DV, Static DAVIS2017, Static YouTubeVOS, and CameraBench) at different time steps (T).

**Table 10.** Additional temporal-context results. SDM and the baseline generally benefit from additional frames on the temporally coherent Kubric and CameraBench benchmarks. On benchmarks with less temporal coherence, our method remains stable with increasing temporal context, while the frame-pair baseline can degrade. We report MSE on standardized regression targets and accuracy in % for classification tasks.

**Table 12.** Structured motion results with standard deviations.

Component Ablations

We test each SDM component to see how its removal harms motion specialization.

The Structured Dynamics Model (SDM) predicts future frozen image features by first removing camera motion (Primary Motion) and then refining object motion (Residual Motion).

**Table (a)** Weak supervision and sequential extraction improve token specialization. Matched token–target pairs are highlighted in green.

With full supervision, the primary token $p$ predicts camera motion more accurately than the residual token $r$ (MSE $0.16$ vs. $0.28$).

Table 13 reports $p$ = $0.16$ ± $0.0$, $r$ = $0.28$ ± $0.0$ for camera motion on Kubric.

Conversely, the residual token $r$ excels at object motion (MSE $0.19$ vs. $0.21$ for $p$).

Table 13 shows $r$ = $0.19$ ± $0.0$, $p$ = $0.21$ ± $0.0$ for object motion on Kubric.

Removing static‑scene supervision erodes this specialization: the primary token’s camera error rises to $0.18$ and the residual token improves to $0.21$, making them nearly indistinguishable.

Table 13 (no static‑scene sup.) reports $p$ = $0.18$, $r$ = $0.21$ for camera motion.

**Table 5.** SDM leads to structured motion representations where $p$ and $r$ specialize in primary and residual motion when the features are used to linearly regress or classify motion or action. We report MSE on standardized regression targets and accuracy in % for classification tasks. Among individual tokens, the best result is shown in bold. Representations and targets that match are highlighted in green. The last row reports an upper-bound probe using the concatenated tokens $[p, r]$.

Table 9 compares several multi‑layer direct‑feature baselines that pool DINOv2 descriptors across all 12 backbone blocks, either by concatenation or feature difference, and optionally across adjacent frames.

Even with access to all backbone layers, direct descriptors fall well short of SDM on every benchmark except static DAVIS2017.

Across the regression and classification columns in Table 9, SDM’s MSE and accuracy consistently beat the best AVG‑pool or CLS baseline, while static DAVIS2017 is the sole exception where a direct descriptor matches SDM.

This gap indicates that the improvement does not stem merely from aggregating intermediate features; the two‑stage motion compensation is essential.

Table 10 adds temporal context by feeding multiple frames to both SDM and the baseline. Both benefit, but SDM gains a larger boost on the temporally coherent Kubric and CameraBench datasets.

Adding extra frames improves SDM’s camera‑motion MSE on Kubric from $0.16$ to $0.12$, a larger reduction than the baseline’s improvement ($0.20$ → $0.15$).

Table 10 reports the SDM and baseline values before and after adding frames.

Extended Quantitative Results

Further tables detail SDM’s gains across temporal context and motion token variants.

SDM’s accuracy on SSv2‑110k rises steadily with more input frames, surpassing the frame‑pair baseline even at the longest context.

Table 11 shows SDM accuracy increasing from 10.9 % (T=2) to 23.1 % (T=7), while the baseline peaks at 13.3 % (T=5) and falls behind.

Across motion‑token analyses, the primary token yields the lowest regression error on five of six benchmarks, but on static‑DAVIS2017 the residual token attains higher classification accuracy.

Primary motion token achieves the lowest regression error on most benchmarks, confirming its dominance in capturing the dominant motion source.

Table 12 reports primary MSE of 0.16 ± 0.01 on Kubric camera versus residual 0.21 ± 0.03, and similar gaps on other datasets.

Static‑camera regularization balances object motion and action prediction; setting $\lambda_{reg}=0.5$ yields 0.19 MSE on Kubric object motion and 23.1 % accuracy on SSv2‑110k.

**Table.** Static-camera regularization balances object motion and action prediction. We vary $\lambda_{reg}$, the weight of the static-camera regularization term. Smaller values yield stronger SSv2-110k performance, while larger values improve Kubric object-motion probing. We use $\lambda_{reg} = 0.5$ as a compromise, preserving strong Kubric object-motion performance (0.19) and SSv2-110k accuracy (23.1%).

Additional Ablation Details

Ablation study quantifies each SDM component’s impact on regression and classification performance.

This section reports a systematic ablation of the SDM components listed in Table 13, measuring both regression $MSE$ and classification accuracy %.

Removing weak scene‑level supervision raises $MSE$ by +0.02 and drops accuracy by ‑1.2 %.

Baseline $MSE$ = 0.16, accuracy = 85.3 %; without supervision $MSE$ = 0.18, accuracy = 84.1 % (Table 13).

Replacing the two‑stage primary/residual extraction with joint motion extraction degrades $MSE$ by +0.05 and accuracy by ‑3.3 %.

Baseline $MSE$ = 0.16, accuracy = 85.3 %; joint extraction $MSE$ = 0.21, accuracy = 82.0 % (Table 13).

Using only the final encoder feature (no intermediate features) inflates $MSE$ by +0.34 and cuts accuracy by ‑15.3 %.

Baseline $MSE$ = 0.16, accuracy = 85.3 %; final‑only $MSE$ = 0.50, accuracy = 70.0 % (Table 13).

Training without any real‑video data raises $MSE$ by +0.01 and reduces accuracy by ‑5.3 %.

Baseline $MSE$ = 0.16, accuracy = 85.3 %; no real data $MSE$ = 0.17, accuracy = 80.0 % (Table 13).

Switching the feature‑prediction loss from $MSE$ to $L_1$ increases $MSE$ by +0.025 while leaving accuracy unchanged.

Baseline $MSE$ = 0.16 (with $MSE$ loss); $L_1$ loss $MSE$ = 0.185 (≈ 0.16 + 0.025) (Table 13).

Setting the static‑camera regularization strength to $λ_{reg}=0.1$ lowers accuracy by ‑1.3 % relative to the optimal $λ_{reg}=0.5$.

Optimal $λ_{reg}=0.5$ yields 85.3 % accuracy; $λ_{reg}=0.1$ yields 84.0 % (Table 13).

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers