SGF+: Decoupling Gradient Flows for Autoregressive Video Generation
Zihan Su, Junhao Zhuang, Yaowei Li, Siwen Lu, Haoran Li, Lingen Li, Haoyu Wu, Weiyang Jin, Songchun Zhang, Haoyang Huang, Chun Yuan, Zeyue Xue, Nan Duan
SGF+ decouples context-writing and denoising parameters to resolve gradient conflicts in autoregressive video generation.
How can we resolve the gradient conflict between context-writing and denoising in autoregressive video generation models?
Autoregressive video models generate frames by denoising current content while simultaneously encoding that content into a memory cache for future steps. These two roles typically share the same parameters, but their optimization demands are fundamentally misaligned, leading to conflicting gradient updates that degrade long-term visual consistency. SGF+ assigns independent, role-specific parameters to the context writer and the denoiser while maintaining their causal interaction. This architectural split allows both roles to be jointly optimized under the original generation objective without requiring auxiliary losses or additional training data. This simple decoupling enables native long-horizon extrapolation, allowing models trained on 5-second windows to generate coherent video for up to 24 hours without further fine-tuning.
Paper Primer
The core issue is a "gradient conflict" where the context-writing and denoising roles pull shared weights in opposing directions. Analysis reveals that the mean gradient directions for these two roles exhibit angular separation exceeding 90°, effectively causing the updates to cancel each other out during training.
SGF+ implements role-specific parameterization: it splits the model into a dedicated context writer and a denoiser, each with its own parameter set. The method is like a two-person relay team where one runner focuses exclusively on mapping the path ahead while the other focuses on the current sprint, ensuring their individual efforts don't interfere with the other's technique.
SGF+ improves visual quality and temporal consistency over shared-parameter baselines across long-horizon generation.
Quantitative evaluation on VBench-Long (60s) and MovieGen-128 (240s) prompts shows consistent gains in subject consistency, background consistency, and temporal flickering metrics. SGF+ achieves native extrapolation to 24-hour continuous video generation from only 5-second training rollouts.
Why does this problem matter for video generation?
Autoregressive models rely on historical context to maintain consistency; if the context-writing process is poorly optimized due to gradient conflict, the model accumulates errors that manifest as visual artifacts or scene drift over long sequences.
Does this approach require significantly more compute?
No; while the parameter count doubles, the inference latency remains nearly unchanged because each token only uses its role-specific parameters during a forward pass, incurring only a modest 8.2% increase in training time per step.
Role-specific parameterization is a highly effective design principle for autoregressive video models, proving that separating conflicting optimization objectives is more impactful than simply increasing training horizons or adding auxiliary losses.
Introduction and Motivation
We expose and resolve gradient conflict between denoising and context‑writing in autoregressive video generation.
Autoregressive video diffusion models generate frames sequentially: each new frame is denoised and then encoded into a key‑value (KV) cache that serves as context for future predictions. The same set of parameters is used for both denoising and for writing that context, which forces the two roles to compete for gradient updates.
When a single weight matrix tries to satisfy two opposing objectives—producing high‑quality frames now and building useful future context—the gradients from each objective point in opposite directions, pulling the parameters apart.
Self Forcing (SF) avoids exposure bias by training on self‑generated histories, but it freezes the KV cache during rollout, leaving the context‑writing gradients unsupervised. Self Gradient Forcing (SGF) restores those gradients via a differentiable replay, yet it still shares parameters between the two roles, so residual degradation remains.
**Figure 1.** Qualitative comparison of autoregressive video generation methods. Self Forcing (SF) blocks context gradients, preventing future predictions from supervising context writing and causing pronounced visual degradation over long rollouts. Self Gradient Forcing (SGF) restores context gradients and improves long-horizon consistency, but conflicts between context-writing and denoising updates keep both consistency and visual quality suboptimal. Self Gradient Forcing Plus (SGF+) decouples context writing and denoising, achieving the best visual quality and long-horizon consistency among the three methods. Trained with only 5s rollouts, SGF+ supports continuous generation for up to 24 hours without long-video fine-tuning.
Motivated by this gradient conflict, we propose Self Gradient Forcing Plus (SGF+), which assigns distinct parameter sets to context‑writing and denoising while preserving their interaction through causal attention. Both roles are jointly optimized with the original generation objective, requiring no auxiliary losses, extra video data, or longer training horizons.
Our contributions are threefold: (1) we reveal the conflicting gradient patterns that limit shared‑parameter video models; (2) we introduce SGF+, a role‑specific parameterization that resolves this conflict; and (3) we demonstrate superior visual quality and long‑horizon consistency on VBench, achieving continuous 24‑hour generation from only 5‑second rollouts.
The conflict between denoising and context‑writing gradients is the primary bottleneck for long‑horizon video generation.
Background and Prior Methods
We survey forcing strategies and related video models, highlighting SF and SGF.
Prior work on autoregressive video diffusion has explored a spectrum of forcing and context‑management techniques, from early Teacher Forcing to recent SGF‑based methods. These approaches differ in how they construct histories, supervise rollouts, and propagate gradients, shaping both training efficiency and long‑horizon fidelity.
SF generates its own short rollouts during training, so the model sees the same noisy histories it will encounter at inference time.
SGF restores the gradient path through the self‑generated history, allowing future‑generation losses to supervise how context is written.
**Figure 2. Method lineage for autoregressive video generation.** SF addresses train–test history mismatch via self-generated rollouts. SGF restores context-writing gradients, while SGF+ decouples context writing from denoising to resolve their gradient conflict.
Adapts SGF’s differentiable context reconstruction to supervise future video generation, enabling interactive editing of generated clips.
Extends SGF to multimodal audio‑video generation, supervising both visual and auditory streams with shared context gradients.
Incorporates SGF‑style context supervision into a diffusion‑based video generator, improving long‑term consistency without extra training data.
Builds on SGF to supervise context writing via future‑generation losses, targeting high‑resolution video synthesis.
Adapts SGF’s differentiable context reconstruction for interactive video editing, allowing user edits to propagate through future generations.
The SGF+ Decoupling Mechanism
We decouple context writing and denoising by assigning separate parameters, eliminating gradient interference.
When a single set of parameters serves both denoising and context‑writing, their gradients point in opposite directions, slowing convergence and degrading long‑horizon consistency.
**Figure 4.** Training pipeline of SGF and SGF+. Pass 1 performs a no-gradient autoregressive rollout and records detached clean context latents $X$ together with noisy target latents $Z^*$ at a sampled denoising exit. Pass 2 reconstructs the forward computation from the recorded latents, allowing future-generation losses to supervise context writing through differentiable context KV states. SGF accumulates context-writing and denoising gradients on shared parameters $\theta$, whereas SGF+ assigns separate parameters $\theta_c$ and $\theta_d$ to the two roles.
Pass 1: Run an autoregressive rollout with the denoiser D_{$\theta_{d}$}; at each block i record the noisy latent $z_{t*}^i$ and the clean estimate \tilde{x}_i, then write \tilde{x}_i into the KV cache using the context writer C_{$\theta_{c}$} at timestep $t_{ctx}$=0. No gradients are retained.
Pass 2: Reconstruct the entire sequence in parallel under a causal mask $M_{rec}$. Feed the detached clean estimates \mathbf{X} into C_{$\theta_{c}$} (stop‑gradient) to obtain differentiable KV states M_{$\theta_{c}$}.
Use the same mask $M_{rec}$ to run the denoiser D_{$\theta_{d}$} on the stored noisy latents \mathbf{Z}^* together with M_{$\theta_{c}$}, producing target reconstructions \hat{\mathbf{X}}_{tar}.
Compute the future‑generation loss L = $L_{DMD}$(\hat{\mathbf{X}}_{tar}) and back‑propagate: gradients flow to $\theta_{c}$ through the context writer and to $\theta_{d}$ through the denoiser, never mixing.
Step 1: Pass 1 writes \tilde{x}_1 into KV slot 1 using C_{$\theta_{c}$}, then \tilde{x}_2 into slot 2, and \tilde{x}_3 into slot 3. No gradients are stored.
Step 2: Pass 2 feeds the detached set \mathbf{X} = {\tilde{x}_1,\tilde{x}_2,\tilde{x}_3} into C_{$\theta_{c}$} (stop‑gradient) to obtain differentiable KV states M_{$\theta_{c}$} = {$m_1$,$m_2$,$m_3$}.
Step 3: D_{$\theta_{d}$} processes the noisy latents \mathbf{Z}^* = {$z_{t*}^1$,$z_{t*}^2$,$z_{t*}^3$} together with M_{$\theta_{c}$} to produce reconstructions \hat{x}_1,\hat{x}_2,\hat{x}_3.
Step 4: The loss L compares \hat{x}_i to the ground‑truth clean frames, yielding separate gradient vectors ∂L/∂$\theta_{c}$ and ∂L/∂$\theta_{d}$.
Because $\theta_{c}$ and $\theta_{d}$ are distinct, the two gradient vectors add to the total update without canceling each other, directly addressing the interference observed in SGF.
Instead of forcing one weight matrix to both write clean context and denoise noisy latents, we give each role its own dedicated parameters, so their gradient signals no longer fight.
By cleanly separating the two roles, SGF+ achieves higher visual fidelity and more stable long‑horizon generation without altering the overall architecture.
Empirical Evaluation
SGF+ delivers the strongest long‑horizon video quality while keeping overhead low.
Recall that shared parameters for denoising and context‑writing cause conflicting gradients, which SGF+ resolves by separating their roles.
It measures how aligned the context‑writing gradient $g_C$ and the denoising gradient $g_D$ are, using cosine similarity of their vectors.
How does Gradient Angular Distance differ from ordinary cosine similarity?
Ordinary cosine similarity compares two feature vectors; Gradient Angular Distance applies the same formula to two gradient vectors, so it measures conflict between learning signals rather than similarity of data representations.
SGF+ improves long‑horizon video quality across all metrics, achieving the highest Subject Consistency at 240 s framewise.
Table 1 shows SGF+ reaches 98.74 versus 96.36 for SF, a +2.38 improvement.
**Figure 3. Visualization of gradient conflict.** We analyze SGF gradients over 128 prompts and 4 denoising timesteps, separately for Attention and FFN. (a) Angular-distance t-SNE visualizes the distributions of context-writing and denoising gradients. $t_0 = 0$ denotes context writing at timestep 0, while $t_1, t_2, t_3, t_4$ denote denoising at timesteps 250, 500, 750, and 1000, respectively. (b) Gradients are aggregated to compute the mean directions. (c) At each denoising timestep, we compute the paired cosine similarity between the denoising and context-writing gradients for the same prompt. Layer-wise gradient distributions are provided in Appendix E.
**Figure 5. Qualitative comparisons.** The top two cases show chunkwise generation, and the bottom case shows framewise generation. Additional qualitative comparisons covering a wider variety of scenes are provided in Appendix F.
SGF+ significantly reduces gradient conflict and improves video generation metrics.
Efficiency and Trade-offs
We examine SGF+ efficiency trade‑offs, highlighting modest memory and training time costs versus unchanged inference speed.
Efficiency is the first lens through which we assess SGF+. Although the model’s parameter count doubles, the per‑token compute remains unchanged, allowing a fair comparison of runtime and memory.
Beyond raw numbers, we observe role‑specific gradient conflict already at teacher‑forcing (TF) initialization, before any SGF training occurs.
Compared with standard autoregressive language models, video diffusion alternates between clean‑context encoding and noisy‑latent denoising, which explains the observed gradient divergence.
**Table 3. Training and inference efficiency.** Compared with SGF, SGF+ modestly increases training time per step, with a small increase in inference memory and nearly unchanged inference latency.
Overall, role‑specific parameterization proves a practical design principle: it preserves generation speed while incurring only modest training resource costs.
Additional Video Results
SGF+ dominates VBench metrics in both chunkwise and framewise settings.
Across the VBench evaluation at 5 s, SGF+ consistently outperforms SF and SGF in both chunkwise and framewise modes, achieving the top score for every reported metric.
**Table 1.** Comparison of different methods (Self Forcing, Self Gradient Forcing, and Self Gradient Forcing Plus) across various metrics, evaluated in both Chunkwise and Framewise settings.
Gradient Analysis Details
Separating context‑writing and denoising parameters improves gradient descent under shared budgets.
Recall: autoregressive video models share parameters for denoising and context‑writing, which creates conflicting gradient updates; SGF⁺ decouples these roles into separate parameters.
When the same parameters serve both context‑writing and denoising, the admissible update direction is forced onto the line where the two role‑wise updates are identical, discarding any component that points differently for the two roles.
Why does tying the two parameter groups hurt descent even when the gradients are positively aligned?
Because the update direction is forced onto the average $ (g_C+g_D)/2$, discarding the orthogonal component $ (g_C-g_D)/2$ that would otherwise contribute additional decrease. The loss of that component is quantified by the term $\|g_C-g_D\|_2^2$ in the gap.
If the context‑writing and denoising gradients are exact opposites, tying the parameters creates a stationary point for the shared objective even though the underlying surrogate still has a non‑zero gradient.
When the two roles are allowed independent updates, the split update can achieve a strictly larger first‑order decrease than the shared update, leading to a lower surrogate value after one step.
When the two role‑wise gradients point in opposite directions, using the same learning rate for the shared parameter vector can make the shared update strictly worse than the split update.
Separating only a subset of modules (e.g., the feed‑forward network) yields a descent advantage proportional to the number of untied blocks.
**Figure 6.** **Adjacent-timestep gradient directions at TF initialization.** We measure gradient directions on 128 video samples over a 50-step diffusion schedule. The curve shows the mean angle between adjacent timesteps. The sharp 0 → 1 transition reveals distinct gradient directions for clean-context writing and denoising. Additional visualizations of context-writing and denoising gradients at TF initialization are provided in Appendix C.
**Figure 7. Visualization of context–denoising gradients at TF initialization.** We use 128 video samples and 50 randomly sampled denoising timesteps, with context fixed at $t = 0$. (a) Angular-distance t-SNE visualizes 6,400 context-writing and 6,400 denoising gradient samples per module family using coordinate-sampled gradient representations. Color indicates the denoising timestep. (b) Mean gradient directions and relative lengths are estimated from the sampled coordinates, with the context mean length normalized to one in each family. (c) Paired cosine similarities are computed from the full selected weight gradients. Points show video–timestep pairs, while lines and bands show the median and interquartile range across videos.
**Figure 9. Layer-wise gradient distributions in Attention.** We analyze SGF gradients over 128 prompts and 4 denoising timesteps, concatenating the Q/K/V/O weight gradients within each block. Angular-distance t-SNE uses gradient sketches with perplexity 20. *sil.* is the role silhouette score computed from angular distances before projection. Higher scores indicate stronger role separation.
**Figure 10.** Layer-wise gradient distributions in FFN. We concatenate the up/down weight gradients within each block and use the settings in Figure 9. Block 29 has zero context-writing gradients for all 512 prompt–timestep pairs, so its angular distances are undefined and no t-SNE is shown. sil. denotes the role silhouette score before projection, with higher values indicating stronger role separation.
Decoupling context‑writing from denoising (SGF⁺) restores the gradient components lost to parameter sharing, yielding provable first‑order and finite‑step descent advantages.
Supplementary Overview
Key supplementary details on SGF+ and its training benefits.
We identified an optimization conflict between context‑writing and denoising in the shared‑parameter Self Gradient Forcing (SGF) setup. SGF+ resolves this conflict by separating the roles into distinct parameter groups, which yields noticeably better visual quality and long‑horizon consistency compared to both SF and SGF. The improvement holds for both framewise and chunkwise generation pipelines.
SGF+ also supports generation up to 24 hours from merely 5 s of training rollouts, demonstrating its scalability. The role distinction is already visible at TF initialization, which motivates a role‑aware parameterization throughout autoregressive video training. Detailed results and analyses are provided in the supplementary sections A–H.
Generation Policy Details
Key implementation details and qualitative results for SGF⁺ video generation.
Framewise generation uses a window of 21 latents (4 sink, 16 FIFO, 1 current), while chunkwise generation uses a window of 12 latents (3 sink, 6 FIFO, 3 current).
Both generator and critic are optimized with AdamW ($\beta_1=0$, $\beta_2=0.999$); the generator learns at $2\times10^{-6}$ and the critic at $4\times10^{-7}$, with one generator update per five critic updates and four denoising steps per generation.
At transformer initialization the attention and FFN gradients point in opposite directions (mean angles $105.0^\circ$ and $114.5^\circ$), and most cosine similarities are negative, indicating early role conflict.
The image displays two rows of sequential frames representing temporal progression (0h to 24h) for two different prompts. The top row shows an otter surfing, and the bottom row shows a woman reading a book.
The figure displays two sets of image sequences, each accompanied by a descriptive prompt. The top row shows eight variations of a lakeside cabin scene, with a prompt describing a serene cabin surrounded by pine trees and reflected in a lake. The bottom row shows eight variations of a coastal beach scene, with a prompt describing an idyllic beach in an oil painting style with golden sunlight.
Layer‑wise silhouette scores for attention blocks rise from 0.035 (Block 0) to 0.143 (Block 28), indicating increasing role separation deeper in the network.
FFN blocks show a similar trend, with scores climbing from 0.045 (Block 0) to 0.175 (Block 26), except Block 29 where context‑writing gradients are absent.
**Figure 11.** Additional qualitative comparisons in chunkwise generation. SF, SGF, and SGF+ are compared in three scenes. From top to bottom: a fluffy monster gazing at a candle, a neon-lit Tokyo street, and a person reading.
**Figure 12.** Additional qualitative comparisons in chunkwise generation. SF, SGF, and SGF+ are compared in three scenes. From top to bottom: an astronaut, a boy riding a bicycle, and a woman in a flower garden.
**Figure 13.** Additional qualitative comparisons in chunkwise generation. SF, SGF, and SGF+ are compared in three scenes. From top to bottom: a red balloon drifting through an urban street, a woman aboard a train, and a sunflower by a windowsill.
**Figure.** Qualitative comparison of video generation results using different methods (SF, SGF, and SGF+) across three different prompts. Each row represents a method, and each column represents a time step from 0s to 240s.
**Figure 15.** Additional qualitative comparisons in framewise generation. SF, SGF, and SGF+ are compared in three scenes. From top to bottom: a porcelain toilet in an old-fashioned bathroom, a white cat driving a red car, and a volcanic eruption in a coffee cup.
**Figure.** Qualitative comparison of video generation results using different methods (SF, SGF, and SGF+) across three different prompts. Each row represents a method, and each column represents a time step in seconds.
Local optimization analysis confirms that separating context‑writing and denoising gradients reduces interference and improves generation stability.