Self Gradient Forcing: Native Long Video Extrapolation

Junhao Zhuang, Shiyi Zhang, Yuxuan Bian, Yaowei Li, Yawen Luo, Yijun Liu, Weiyang Jin, Songchun Zhang, Xianglong He, Xuying Zhang, Haoran Li, Haoyang Huang, Zeyue Xue, Nan Duan

Self Gradient Forcing (SGF) enables long-horizon video extrapolation by supervising how self-generated history is encoded into memory.

How can we train autoregressive video models on their own generated histories without the memory explosion of backpropagating through long, frozen-cache sequences?

Autoregressive video diffusion models struggle with long-form generation because they treat self-generated history as a frozen cache, preventing future losses from correcting how that history is written into memory. SGF introduces a two-pass training strategy: it performs a no-gradient rollout to record history, then re-encodes that history in a parallel pass that allows gradients to flow back into the memory-writing computation. This restores the missing supervision signal, enabling models trained on short 5-second windows to maintain subject identity and temporal stability over minutes of extrapolation.

Paper Primer

In standard "Self Forcing" training, the model generates a history and stores it in a Key-Value (KV) cache. Because this cache is treated as a detached, frozen state, the model cannot learn how to write better, more coherent history for future frames to read. SGF is a two-pass training strategy: it performs a serial no-gradient rollout to record the history, then reconstructs the exit-step computation in parallel to allow future losses to supervise the memory-writing process.

SGF significantly improves long-horizon video consistency compared to frozen-cache baselines.

Across 60-second and 240-second extrapolation tests, SGF consistently outperforms Self Forcing on metrics for subject identity, background consistency, and temporal stability. Models trained on only 5-second windows successfully extrapolate to 240-second videos with higher human preference scores (GSB) across all tested initializations.

Why not just make the entire serial cache differentiable?

Directly backpropagating through a serial cache requires retaining the entire autograd graph for every cache write, which grows linearly with rollout length and leads to out-of-memory (OOM) errors. SGF avoids this by using a bounded, parallel reconstruction pass that only optimizes the specific exit-step computation.

Does this method require more compute during training?

The overhead is modest. SGF increases peak memory usage slightly (e.g., from ~79 GB to ~87 GB) and increases wall-clock time by roughly 10-15% per training step, but it avoids the memory blow-up associated with full-rollout backpropagation.

Introduction and Motivation

We expose and address the missing gradient supervision in self‑forcing video diffusion.

Autoregressive video diffusion models must keep scenes, objects, and dynamics coherent far beyond the training window. Self Forcing reduces exposure bias by training on self‑generated histories, but because the historical key‑value cache is frozen, future losses cannot teach the model how to write useful memory for later frames, creating a historical context‑gradient gap.

During training the model is fed the very histories it will later generate at inference, so it learns to operate on its own predictions instead of perfect ground‑truth frames.

A model trained only on ground‑truth contexts learns to rely on perfect past frames, so at test time—when it must condition on its own imperfect predictions—it degrades.

**Figure.** Comparison between "Self Forcing" (top row) and "Self Gradient Forcing" (bottom row) over a 240-second duration. The top row shows visual artifacts like "View jump," "Scene break," and "Identity drift," while the bottom row maintains consistent subject, pose, and restaurant layout. The prompt used is: "A Chinese boy with neatly combed hair wears glasses and sits in a warm, inviting fast food restaurant, eyes closed as he takes a big bite of a juicy cheeseburger. He holds the burger with both hands, savoring the moment with a slightly dreamy, ..."

The trade‑off between training stability and memory cost defines the core challenge of long‑horizon autoregressive video generation.

Related Work

We position Self Gradient Forcing among prior video diffusion and long‑horizon caching approaches.

Modern video generation builds on diffusion objectives, accelerated samplers, latent diffusion, and diffusion transformers, extending them with 3D denoising, cascaded generation, latent video modeling, motion modules, space‑time architectures, and multimodal autoregressive token modeling.

For long‑form generation the key distinction is the causal use of self‑generated history: each generated frame or chunk becomes part of the context for later steps. Prior works such as Diffusion Forcing and CausVid connect next‑token prediction to diffusion by independently noising sequence elements, while Self Forcing reduces exposure bias by training on the model’s own rollouts.

Several forcing variants—Causal Forcing, Causal Forcing++, Rolling Forcing, Self‑Forcing++, Matrixgame3, ShotStream, Video‑Mirai, and Next Forcing—extend rollout exposure through longer segments or multi‑chunk supervision. Independent lines of work improve long‑horizon context by redesigning the cache: Gen‑L‑Video, FreeNoise, FIFO‑Diffusion, StreamingT2V, and a suite of recurrent‑sequence and efficient‑attention techniques (segment recurrence, rotary embeddings, attention sinks, KV compression, etc.). Self Gradient Forcing is orthogonal to these cache‑design advances, improving how self‑generated content is written into future‑readable KV memory.

The Historical Context Gap

We expose the missing gradient from cache writes and outline the practical SGF solution.

The autoregressive video diffusion model writes each generated latent into a historical key/value (K/V) cache at a clean‑context timestep. Because the cache entries are treated as detached state during Self Forcing, later losses cannot back‑propagate through the cache‑writing operation, creating a “historical context‑gradient gap.”

The cache is a growing table of key/value pairs that stores a clean‑context representation of each previously generated latent, enabling later blocks to attend to past content without recomputing it.

Block 1 clean‑context pass computes $K_1=(0.2,\,0.5)$, $V_1=(0.7,\,0.1)$ and appends $(K_1,V_1)$ to the cache.

Block 2 reads the cache, forming attention scores $\text{score}=Q_2 K_1^\top$ where $Q_2$ is Block 2’s query.

The attention output is $ \text{softmax}(\text{score}) \, V_1$, using the stored $V_1$ without recomputing it.

The cache lets later blocks reuse a clean representation of earlier latents, but if gradients cannot update $K_1$ and $V_1$, the stored values may become stale as the model parameters evolve.

How does the historical K/V cache differ from the temporary attention memory used within a single transformer layer?

The K/V cache persists across time steps, accumulating entries from all previous blocks, whereas the per‑layer attention memory is recomputed each forward pass and discarded afterward. The cache therefore carries long‑range information, while temporary memory only captures short‑range interactions.

A naïve way to close the gradient gap would be to keep the entire cache differentiable, allowing back‑propagation through every stored entry. However, each cache write would need to retain its backward graph until the final block finishes, causing the computational graph to grow linearly with rollout length and making training infeasible for long videos.

**Figure 2.** From frozen-cache Self Forcing to Self Gradient Forcing. Self Forcing trains on self-generated histories but treats historical K/V entries as detached cache state. SGF keeps the no-gradient rollout unchanged and adds a parallel reconstruction pass, where detached context latents are re-encoded at the clean context timestep so future DMD losses supervise K/V writing without serial rollout backpropagation.

Self Gradient Forcing Mechanism

Self Gradient Forcing adds a parallel reconstruction pass to supervise cache writing without exploding memory.

Long‑horizon video diffusion drifts because the model never receives gradient feedback on how it writes history into the cache. Naïvely back‑propagating through the entire autoregressive rollout would require storing every intermediate K/V entry, which quickly exceeds GPU memory.

SGF keeps the usual no‑gradient rollout but adds a second, parallel pass that re‑encodes the recorded latents at the clean‑context timestep, allowing gradients to flow into the cache‑writing path.

How does SGF differ from the original Self Forcing gradient handling?

Self Forcing records the rollout but treats the resulting K/V entries as a detached cache, so later losses cannot adjust how those entries are written. SGF lifts the stop‑gradient from the K/V path to the recorded latents, enabling the reconstruction loss to back‑propagate into the cache‑writing projection.

Pass 1 (no‑gradient): block 1 samples $z^{t_1}_1$, runs two denoising steps to produce $\tilde{x}_1$, and stores $KV \leftarrow KV \cup C_\theta(\tilde{x}_1,0)$.

Block 2 repeats the same procedure, yielding $\tilde{x}_2$ and extending the cache.

Pass 2 (gradient enabled): the recorded latents $\{\tilde{x}_1,\tilde{x}_2\}$ are fed at $t_{\text{ctx}}=0$ to compute $\hat{X}_{\text{tar}} = G_\theta(\{z^{t_2}_1,z^{t_2}_2\};\tilde{X}_{\text{ctx}},0,\mathcal{M}_{\text{rec}})$.

The reconstruction loss $L_{\text{recon}}$ is evaluated between $\hat{X}_{\text{tar}}$ and the stored $\tilde{X}_{\text{ctx}}$, and its gradient updates the parameters that produced the cache entries.

Even with only two blocks, the parallel pass supplies a gradient signal to the cache‑writing projection without ever materialising the full serial autograd graph.

The second pass re‑creates the exact causal computation of the sampled exit state, but does so in a single parallel sweep so that gradients can flow through the cache‑writing modules.

Why is the reconstruction performed in parallel rather than re‑running the serial rollout?

Running the serial rollout with gradients would require storing every intermediate K/V pair, exploding memory. The parallel pass re‑uses the recorded latents and the same causal mask, achieving the same supervision while keeping the autograd graph fixed‑size.

Self Gradient Forcing training step (Algorithm 1)

**Algorithm 1 Self Gradient Forcing Training** **Require:** Denoising steps $\mathcal{T} = (t_1, \dots, t_K)$ and scheduler $\Psi$ **Require:** Autoregressive blocks $N$ and clean context timestep $t_{\text{ctx}} = 0$ **Require:** Reconstruction causal mask $\mathcal{M}_{\text{rec}}$ **Require:** Causal DiT $G_\theta$ with serial cache and parallel reconstruction interfaces 1: **loop** 2: Initialize serial cache $KV \leftarrow []$ and records $Z^\star, X_{\text{ctx}} \leftarrow []$ 3: Sample exit index $s \sim \text{Uniform}\{1, \dots, K\}$ and set $t^\star \leftarrow t_s$ 4: **Pass 1: no-gradient self-rollout** 5: Disable gradient computation for all Pass-1 operations 6: **for** block $i = 1, \dots, N$ **do** 7: Sample initial noisy latent $z_i^{t_1} \sim \mathcal{N}(0, I)$ 8: **for** denoising step $k = 1, \dots, s$ **do** 9: $\hat{x}_i^{(k)} \leftarrow G_\theta(z_i^{t_k}; t_k, KV)$ 10: **if** $k < s$ **then** sample $\epsilon$ and set $z_i^{t_{k+1}} \leftarrow \Psi(\hat{x}_i^{(k)}, \epsilon, t_{k+1})$ **end if** 11: **end for** 12: Set $Z_i^\star \leftarrow z_i^{t^\star}$ and $\tilde{x}_i \leftarrow \hat{x}_i^{(s)}$ 13: $KV \leftarrow KV \cup C_\theta(\tilde{x}_i, t_{\text{ctx}}; KV)$ 14: Append $\tilde{x}_i$ to $X_{\text{ctx}}$ and $Z_i^\star$ to $Z^\star$ 15: **end for** 16: **Pass 2: parallel context-gradient reconstruction** 17: Set $X_{\text{rec}} \leftarrow \text{sg}(X_{\text{ctx}})$ 18: Enable gradient computation 19: $\hat{X}_{\text{tar}} \leftarrow G_\theta(Z^\star, t^\star; X_{\text{rec}}, t_{\text{ctx}}, \mathcal{M}_{\text{rec}})$ 20: Update $\theta$ with $\mathcal{L}_{\text{DMD}}(\hat{X}_{\text{tar}})$; keep context-K/V gradients 21: **end loop**

**Figure 3.** Frame-wise 240-second comparison under TF initialization. Both rows use matched prompts, random seeds, initialization, horizon, and inference geometry.

By confining gradients to the parallel reconstruction, SGF delivers the missing supervision for cache writing while keeping the training graph constant‑size, enabling long‑horizon video diffusion without the prohibitive memory cost of full back‑propagation.

Long-Horizon Generation Results

SGF outperforms Self‑Forcing on long‑horizon video metrics across both frame‑wise and chunk‑wise settings.

Recall that autoregressive video diffusion drifts over long horizons; Self Gradient Forcing mitigates this by supervising cached latents with a parallel reconstruction loss.

Self Gradient Forcing consistently beats Self‑Forcing on long‑horizon video generation across both frame‑wise and chunk‑wise evaluations.

Tables 1 and 2 show higher scores for SGF on 12 of the 14 metric‑initialization‑horizon combinations.

**Table 1.** Comparison of different initialization methods (Bidirectional ODE, Causal ODE, Causal CD, and TF) across two time horizons (60s and 240s) using SF and SGF metrics.

**Table 2.** Chunk-wise 60-second and 240-second long-horizon metrics. 60s uses VBench-Long prompts; 240s uses MovieGen-128 prompts.

Training Feasibility Analysis

We evaluate whether adding context‑gradient training hurts memory or speed.

Recall that training with self‑generated histories (Self Forcing) eliminates drift but requires a frozen cache to keep memory bounded. This section asks whether the SGF trick—opening a gradient path only in a second reconstruction pass—adds prohibitive cost.

SGF adds only a modest memory and runtime overhead while preserving trainability.

Peak memory rises by ≈8 GB and wall‑clock time per five steps grows by ≈1.3 s compared with the frozen‑cache baseline.

Latent Recovery Fidelity

Ablation of memory‑heavy variants shows SGF’s bounded reconstruction is far more efficient.

We compare three training variants: (1) a direct differentiable cache, (2) full‑rollout back‑propagation through time, and (3) the proposed Self Gradient Forcing (SGF) bounded‑reconstruction.

Keep the historical K/V cache differentiable while still treating generated latents as stop‑gradient, which forces the autograd graph to become recurrent across the rollout.

How does this differ from SGF’s parallel reconstruction?

SGF discards the serial graph entirely; it records the latents in Pass 1 and then reconstructs them in a single bounded window, so no per‑block activations need to be kept for back‑propagation.

Back‑propagate through the entire autoregressive rollout, including the denoising trajectory that generated each latent.

Why is full‑BPTT impractical for long video sequences?

Because each additional denoising step adds a full‑size activation tensor to the graph; with hundreds of steps the memory requirement exceeds any realistic GPU.

Pass 1 runs the autoregressive rollout without gradients and records the context latents; Pass 2 replays only a fixed‑size window in parallel, back‑propagating through the context‑side forward at $t_{\text{ctx}}=0$.

SGF’s memory footprint is substantially lower than both the direct differentiable‑cache and full‑rollout BPTT variants.

The scaling analysis shows $M_{\text{SGF}} < M_{\text{direct}} < M_{\text{full-BPTT}}$, and empirical runs confirm that the direct‑cache variant runs out of memory while SGF completes training.

We now verify that Pass 2 faithfully reproduces the latent predictions of Pass 1, which is the core assumption behind SGF’s bounded reconstruction.

Protocol: for each of 24 prompts we run Pass 1 at four exit steps (1000, 750, 500, 250), record the noisy latent, then run Pass 2 with the same teacher‑forcing mask and compare the two latent vectors before VAE decoding.

Metrics: RMSE, relative L2 error (scaled by the bf16 precision $\epsilon_{\text{bf16}}=2^{-7}$), and cosine similarity between the two latent vectors.

**Figure 4.** Decoded Pass-1 and Pass-2 recovery comparison. Rows show paired decoded outputs from Pass 1 and Pass 2 at exit steps 1000, 750, 500, and 250. The decoded pairs are visually nearly indistinguishable, matching the latent-space recovery results in Tab. 7.

VAE Boundary and Sink Policy

We examine VAE boundary effects, sink size choices, and the limits of Self Gradient Forcing.

Recall that autoregressive video diffusion drifts when it must rely on its own generated history; Self Forcing mitigates this but requires prohibitive memory because gradients must flow through the entire cache.

The policy keeps a small set of early “sink” latents in memory while discarding the oldest frames in a first‑in‑first‑out (FIFO) window, thereby preserving the VAE’s start‑boundary information without exhausting the context budget.

How does the Sink‑plus‑FIFO policy differ from a plain FIFO buffer?

Plain FIFO discards the oldest latent outright, losing the VAE’s initial boundary information. Sink‑plus‑FIFO retains a handful of early latents (the “sinks”) regardless of age, so the model still sees the start‑boundary pattern while still benefiting from a sliding recent‑frame window.

The VAE encodes time with an asymmetric grouping: a 1‑frame boundary group followed by 4‑frame groups. Because of this, the first few latents of a freshly encoded clip differ from steady‑state latents.

**Figure 5.** VAE boundary and sink-latent choice. Left: sink 1 preserves only the stream-start anchor, while sink 4 preserves the short boundary-transition prefix. Right: local re-encoding mismatch to the full-prefix 21st latent drops sharply after adding previous latent groups and plateaus around the first four positions. This motivates using four sink latents for frame-wise streaming inference.

These measurements show a sharp mismatch (0.607) for a fresh 4‑frame group, which shrinks to 0.271 with the boundary anchor and to 0.060 after adding four preceding groups. The early VAE latents therefore form a short transition region rather than a single isolated token.

We then varied the number of sink latents in frame‑wise SGF (Table 8). With a fixed total context budget, larger sinks preserve more of the VAE boundary but consume FIFO slots. Sink 1 underperforms on aesthetics and flickering; sink 4 reaches the stable region identified by the diagnostic; sink 8 yields only marginal gains while further shrinking recent context.

**Table 8.** Sink ablation in frame-wise SGF with TF initialization on 60-second evaluation. Higher is better for all listed scores after VBench orientation.

The image displays a grid of video frames showing a couple walking under umbrellas, with rows representing different "Sink size" values (1, 2, 4, 8) and columns representing time intervals (0 s, 12 s, 24 s, 36 s, 48 s, 60 s). The prompt used is: "In a black and white, classic film style, a well-dressed couple in elegant evening wear are walking together under their black umbrellas as they make their way home. They are caught in a sudden, heavy downpour."

Limitations

SGF is a bounded surrogate for the missing context‑gradient signal; it supervises how cached self‑generated latents are written but does not back‑propagate through the sampled latents themselves or optimize the sequence of denoising decisions that produced them.

SGF also assumes the parallel Pass‑2 reconstruction exactly mirrors the serial context relation at the sampled exit step. Mismatches in teacher‑forcing mask, sink positions, FIFO window, RoPE handling, timestep alignment, or chunk boundaries can cause SGF to train on an incorrect attention relation.

Finally, SGF complements rather than replaces other long‑video techniques such as streaming‑rollout tuning, retrieval‑augmented memory, sparse attention, stronger causal initialization, or long‑context teachers. Combining SGF with these methods is a natural next direction.

Qualitative Comparisons

Qualitative frame‑wise and chunk‑wise videos show SGF markedly curtails long‑horizon drift versus Self Forcing.

Frame‑wise generation writes each newly produced latent frame back into the historical K/V cache, so any error in the cache propagates at the finest temporal granularity.

Across TF, causal CD, and causal ODE initializations, Self Forcing remains plausible for the first few frames but gradually drifts: subjects shrink or disappear, camera distance changes, and backgrounds collapse into unrelated textures.

Self Gradient Forcing consistently mitigates these failure modes, preserving subject‑scene relations, camera framing, and object layout over both 60‑second and 240‑second rollouts.

**Figure 8.** Frame-wise 60-second comparison under causal CD initialization. Under causal CD initialization, Self Forcing does not always collapse, but it exhibits systematic crop and scale drift. The sea-turtle and elephant examples progressively zoom into the subject, reducing scene context and sometimes clipping the object. SGF keeps the animal scale, background, and camera distance more stable, showing that SGF improves non-catastrophic long-horizon composition drift as well as outright failure.

**Figure 9.** Frame-wise 240-second comparison under causal CD initialization. The otter prompt shows that Self Forcing can preserve broad semantics but accumulates pose and boundary distortions, while SGF keeps the otter centered on the surfboard with a more stable tropical-water layout. In the anime reading prompt, Self Forcing drifts into partial crops and high-saturation color streaks by the late timestamps; SGF preserves the boy, book, and cloud background over the full 240 seconds.

**Figure 10:** Frame-wise 60-second comparison under causal ODE initialization. In the library-reading prompt, Self Forcing introduces chromatic background artifacts by 24 seconds, drops the seated reader by 36 seconds, and devolves into bookshelf/window fragments by 48–60 seconds; SGF keeps the reader, armchair, open book, and bookcase layout stable. In the clock-and-vase prompt, Self Forcing drifts from the intended frontal two-object composition into cropped or blurred close-ups and background fragments, whereas SGF preserves the clock-left/vase-right tabletop arrangement across the rollout.

**Figure 11.** **Frame-wise 240-second comparison under causal ODE initialization.** Under matched initialization and frame-wise streaming settings, Self Forcing undergoes severe long-horizon drift: the candle/creature example degenerates into color blocks and partial subject replacements, and the disco kangaroo example repeatedly loses the stage layout and subject identity. SGF preserves the prompt-specific subject, pose family, and scene layout more consistently through 240 seconds.

Chunk‑wise generation updates the memory at a coarser granularity, writing whole blocks of frames as context for later chunks.

The qualitative pattern mirrors the frame‑wise case: Self Forcing often replaces actions, changes identities, or drifts into unrelated textures, while SGF better maintains action continuity, subject identity, and scene layout over long horizons.

**Figure 12.** Chunk-wise 60-second comparison under TF initialization. In the apple-peeling prompt, Self Forcing with TF initialization changes the person identity and camera viewpoint, at times replacing the tabletop action with close-ups of a smiling face. SGF keeps the hands, apples, and countertop action more consistent. In the eyebrow-makeup prompt, Self Forcing changes the number and identity of people across time; SGF maintains the same seated client, makeup gesture, and salon environment.

**Figure 13.** Chunk-wise 240-second comparison under TF initialization. The coarser chunk-wise rollout tests whether a generated block remains useful as context several chunks later. In the bacterium prompt, SGF maintains a population of translucent spherical cells more consistently, whereas Self Forcing often turns the scene into unrelated circular textures or sparse particles. In the lizard prompt, Self Forcing frequently loses the animal or collapses into foliage; SGF better preserves a green lizard among leaves across the long horizon.

This figure displays two sets of qualitative video generation results, each corresponding to a specific text prompt. The top section shows a couple walking in the rain, while the bottom section shows a Corgi in a park. Each section compares four initialization methods (SF + BiODE init, SF + ODE init, SF + CD init, and SGF + CD init) across six time steps (0s, 12s, 24s, 36s, 48s, 60s).

The figure displays two sets of video generation results, each corresponding to a specific text prompt. The top set shows a young man sprinting down a city street, while the bottom set shows a cloud giant unleashing lightning. Each set compares four initialization methods: "SF + BiODE init", "SF + ODE init", "SF + CD init", and "SGF + CD init". The columns represent temporal progression at 0s, 48s, 96s, 144s, 192s, and 240s.

Human Preference Study

Human raters prefer SGF over Self‑Forcing in long‑horizon video generation.

We ran a blind GSB preference study with more than 1,900 paired judgments across ten matched comparisons, each pitting SGF against the Self‑Forcing baseline under identical initialization, prompt, horizon, and inference geometry.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers