Looking Ahead: Modeling Future Evolution in Autoregressive Video Generation
Ruopeng Gao, Yukun Huang, Jiarui Hu, Shuai Wang, Wenbo Hu, Ying Shan, Limin Wang
Future Chunk Prediction (FCP) improves autoregressive video generation by forcing models to predict distant frames, bypassing local temporal shortcuts.
How can we force autoregressive video models to learn long-term scene evolution rather than just local frame-to-frame continuity?
Autoregressive video models often rely on the high visual similarity between neighboring frames to "cheat" the generation process, failing to learn the underlying physical dynamics of scene evolution. The authors introduce Future Chunk Prediction (FCP), an auxiliary training objective that forces the model to predict temporally distant video chunks without seeing the intervening frames. This forces the internal representations to capture long-range temporal progression rather than just local visual continuity. This approach consistently improves physical plausibility across model scales, outperforming larger bidirectional models on standard benchmarks while maintaining the same inference-time architecture.
Paper Primer
Standard teacher-forced training allows models to recover the current chunk by simply interpolating from the immediately preceding ground-truth frame. Because video is highly redundant, this local continuity acts as a shortcut that masks the model's inability to actually simulate how objects move or interact over time.
FCP is an auxiliary objective: it adds a lightweight, parallel predictor that takes the generator's intermediate hidden states and attempts to reconstruct a future chunk at a temporal offset $\Delta > 1$. By excluding the intervening frames from the context, the model is forced to encode the "physics" of the scene into its hidden states to succeed at the prediction task.
FCP significantly improves physical plausibility in autoregressive video generation.
Controlled comparisons on the Physics-IQ Verified benchmark show consistent IQ score gains across 1.3B and 5B parameter models. For the 5B model, FCP improved I2V IQ scores from 30.4 to 33.8 and V2V scores from 40.6 to 44.4.
FCP-trained models outperform larger, standard autoregressive and bidirectional baselines.
The 5B FCP-trained model achieves higher IQ scores than the 14B Wan2.2 and 8.3B HunyuanVideo-1.5 models on the Physics-IQ benchmark. The 5B model achieves an IQ of 41.1, compared to 32.2 for the 14B Wan2.2 baseline.
Why does this approach use an auxiliary predictor instead of just training the main generator to predict further ahead?
Preliminary experiments showed that sharing generator parameters for both immediate denoising and distant future prediction degraded temporal coherence, as the two objectives place competing demands on the same weights. Using a separate, lightweight auxiliary branch allows the model to learn these dynamics without destabilizing the primary generation process.
Does this method increase the computational cost of generating video?
No. The auxiliary predictors are used only during training to regularize the generator's internal representations. They are discarded at inference, meaning the autoregressive rollout process remains identical to the standard teacher-forced baseline.
Future-chunk supervision provides a data-native way to improve world modeling without requiring external pretrained encoders or complex inference-time context management. Researchers can now treat "predicting the future" as a primary regularizer for learning better spatiotemporal representations.
Motivation and Problem Framing
We expose why teacher forcing misses long-term evolution and propose Future Chunk Prediction.
Autoregressive video models trained with teacher forcing rely on local temporal redundancy, which masks the need for long‑term physical reasoning. Future Chunk Prediction (FCP) adds an auxiliary objective to predict distant future chunks, forcing the model to learn scene evolution.
During training the model is always given the true previous video frames, so it learns to rely on exact past pixels rather than on how the scene should evolve.
How does teacher forcing differ from scheduled sampling?
Teacher forcing always feeds the ground‑truth previous chunk, while scheduled sampling gradually replaces ground‑truth inputs with the model’s own predictions. The latter forces the model to confront its own errors during training, encouraging it to learn more robust temporal dynamics.
The failure of local temporal redundancy to capture scene evolution limits the usefulness of teacher‑forced training for autoregressive video generation.
Flow Matching Foundations
Introduces flow matching and its role in chunk‑wise autoregressive video generation.
Autoregressive video models only see short‑range context, so they never learn how scenes evolve over long horizons. The trick is to inject a learning signal that forces the model to reason about distant future frames.
Learn a time‑dependent vector field that continuously deforms simple Gaussian noise into a realistic video sample.
Compute $x_{0.5}= (1-0.5)\,x + 0.5\,\epsilon = 0.5\,x + 0.5\,\epsilon$ for each frame: $x_{0.5}^{(1)} = (0.5\cdot1,0.5\cdot0)+(0.25,0.25) = (0.75,0.25)$, $x_{0.5}^{(2)} = (0,0.5)+(0.25,0.25) = (0.25,0.75)$.
Target velocity is $\epsilon - x$: for frame 1, $\epsilon - x^{(1)} = (0.5-1,0.5-0)=(-0.5,0.5)$; for frame 2, $\epsilon - x^{(2)} = (0.5-0,0.5-1)=(0.5,-0.5)$.
The model $v_\theta$ is trained to output exactly these velocity vectors at $t=0.5$, i.e. $v_\theta(x_{0.5}^{(1)},0.5)=(-0.5,0.5)$ and $v_\theta(x_{0.5}^{(2)},0.5)=(0.5,-0.5)$.
During inference, starting from pure noise $\epsilon$, the sampler integrates the learned field backward: at $t=1$ we have $\epsilon$, at $t=0.5$ we move toward $x_{0.5}$, and at $t=0$ we arrive at the clean frames $(1,0)$ and $(0,1)$.
Because the velocity is linear in $x$ and $\epsilon$, the model only needs to learn a simple subtraction, which makes the training signal exact and eliminates the stochasticity of diffusion‑based denoising.
How does flow matching differ from standard diffusion models?
Diffusion models learn a stochastic denoising trajectory that gradually removes noise, requiring a schedule of many small steps. Flow matching, by contrast, learns an explicit deterministic velocity field that directly maps any noisy state to the clean video, so the same transformation can be applied in far fewer integration steps.
Chunk‑wise autoregressive video generation rolls out a fixed‑length generator repeatedly, producing a sequence of chunks $x_{1:K}$ conditioned on earlier chunks and optional text $c$. This factorization $p(x_{1:K}\mid c)=\prod_{k=1}^{K}p(x_k\mid x_{<k},c)$ makes long videos tractable while preserving temporal coherence.
Teacher‑forced training supplies the model with clean historical chunks $x_{<k}$ and asks it to predict the velocity that would transform a noisy version $x_{k,t}$ of the target chunk $x_k$ back to the clean video. The resulting loss $L_{TF}(\theta)$ is the same flow‑matching objective applied chunk‑wise.
Future Chunk Prediction
Method introduces Future Chunk Prediction to force long‑term video reasoning.
Teacher forcing leans on the immediately preceding frames, so the model can cheat by copying local context instead of learning how scenes actually evolve.
Predicting a temporally distant video chunk that the recent context cannot directly explain forces the model to capture genuine scene evolution.
Available context: $x_{\le 2} = \{x_1, x_2\}$.
Intervening chunks $x_3$ are omitted from the conditioning set.
The auxiliary predictor receives the hidden state $h_2$ produced after processing $x_2$.
It outputs a distribution $p_{\text{FCP}}(x_4\mid h_2)$ and is trained to match the true $x_4$.
Back‑propagation updates both the predictor and the generator’s parameters that produced $h_2$.
This toy example shows how the model must infer the missing evolution between $x_2$ and $x_4$ without any direct visual cue from $x_3$, encouraging genuine long‑term reasoning.
How does Future Chunk Prediction differ from standard teacher forcing?
Teacher forcing predicts the next immediate chunk $x_{k+1}$ using the full recent context, so the model can rely on local redundancy. FCP skips the intervening chunks and predicts $x_{k+\Delta}$ from $h_k$ alone, forcing the network to learn how the scene changes over a longer horizon.
The FCP architecture builds on a standard video diffusion model with stacked DiT blocks, adding lightweight parallel predictors for each future offset.
Run the autoregressive generator on chunks $f_{t-3}\dots f_t$ to obtain hidden states $h_{t-3}\dots h_t$.
For each desired offset $\Delta\in\{1,2,3\}$, feed $h_t$ into a dedicated predictor DiT block.
Each predictor receives a noisy latent $f_{t+\Delta}$ (dashed green box) and outputs a cleaned latent $f_{t+\Delta}$ (solid green box).
Fuse multi‑layer features from the generator, concatenate them, and pass through an MLP.
Inject the fused features into every predictor DiT block via token‑wise AdaLN modulation.
Omit text cross‑attention in all predictor blocks, ensuring they rely solely on $h_t$.
**Figure 1.** Overview of Future Chunk Prediction (FCP). Intermediate representations from the autoregressive backbone condition auxiliary transformer blocks for future-chunk prediction.
**Figure.** (a) Chained prediction: information from noisy intermediate targets may leak to subsequent predictors along the red arrows. (b) Parallel prediction: offset-specific predictors can run concurrently across GPUs and be flexibly enabled or disabled during training.
Training jointly optimizes the generator’s teacher‑forcing loss $L_{\text{TF}}$ and the auxiliary FCP losses $L_{\text{clean}}^{\Delta}$ and $L_{\text{noisy}}^{\Delta}$ across all offsets, as expressed in Equation (7).
At inference time all FCP modules are removed; the generator runs exactly as the baseline teacher‑forcing model, incurring no extra computational cost.
Physics-IQ Performance
We report quantitative gains of our FCP‑augmented models on Physics‑IQ benchmarks.
Physics‑IQ Verified aggregates several quality metrics (IQ, SP, ST, WS, MSE) to assess how well a video model respects physical plausibility and visual fidelity; higher scores indicate better performance across all aspects.
Table 1 lists Physics‑IQ scores for a variety of video generation models. Our FCP‑augmented autoregressive models achieve the strongest SP and competitive IQ scores among autoregressive baselines. Table 2 shows controlled ablations of the FCP design, highlighting the impact of offset choices.
Our 1.3 B autoregressive model with FCP attains the highest SP score (35.7), beating the next‑best autoregressive baseline by 7.2 points.
Table 1, row “Wan2.1‑1.3B + FCP (Ours)”, SP = 35.7 versus Helios‑Base SP = 28.5.
Controlled Ablations and Analysis
Key ablations show how Future Chunk Prediction improves video modeling.
Autoregressive video models trained with teacher forcing rely on short‑term redundancy; adding Future Chunk Prediction forces the model to learn longer‑range scene evolution.
**Table 4.** Ablation of the main FCP design components. The shaded row and column mark the default setting and overall IQ score, respectively.
Future Chunk Prediction raises the IQ score from 27.6 to 30.9 for the 1.3 B model.
Table 4, “Overall IQ” row for the 1.3 B model.
For the 5 B model, I2V improves from 30.4 to 33.8 with FCP.
Table 4, I2V column for the 5 B model.
For the 5 B model, V2V improves from 40.6 to 44.4 with FCP.
Table 4, V2V column for the 5 B model.
Our configuration requires 42.6 % less training compute to reach an IQ of 29.
Section 5.5 discussion of estimated FLOPs.
Our configuration requires 50.7 % less training compute to reach an IQ of 30.
Removing explicit text conditioning from the auxiliary predictor drops IQ by 1.1.
Table 4, rows (d) vs (e) comparing text‑free vs text‑conditioned predictors.
Qualitative examples illustrate the effect: with FCP the domino sequence preserves the correct toppling direction, while the teacher‑forcing baseline collapses the central domino; in the ice‑melting video FCP captures the gradual water‑level rise, whereas the baseline misses this subtle change.
**Figure 5.** Spatial Feature Similarity Maps. Each map shows within-frame similarity to a selected query token. The top and bottom rows correspond to teacher forcing and FCP, respectively.
**Figure 6.** Spatiotemporal Feature Similarity Maps. The top row shows video frames, while the bottom row visualizes similarity to a selected query token across space and time.
Extended Ablations and Visualizations
Ablation results and visual analyses of component choices and future predictions.
This appendix expands the ablation studies and visualizes the effect of Future Chunk Prediction. It shows how different component choices impact quantitative metrics and how the learned representations differ qualitatively.
Future Chunk Prediction raises the IQ metric from 27.6 (teacher forcing) to 30.9 (AdaLN fusion), a +3.3 improvement.
Table 7 shows the AdaLN fusion strategy achieving IQ 30.9 versus 27.6 for the baseline.
**Figure 7.** Additional Spatial Feature Similarity Maps. Each map shows within-frame similarity to a selected query token; two further examples continue in the next panel. Within each example, the top and bottom rows correspond to teacher forcing and FCP, respectively.
**Figure 8.** Additional Spatiotemporal Feature Similarity Maps. In each example, the top row shows video frames, while the bottom row visualizes similarity to a selected query token across space and time.
**Figure 9.** Detailed Spatiotemporal Feature Similarity Maps. In each example, the top row shows video frames, while the middle and bottom rows show similarity to a selected query token under teacher forcing and FCP, respectively.
**Figure 10.** Auxiliary Future Predictions. In each example, the top row shows the ground-truth video, while the middle and bottom rows show outputs from auxiliary branches at the future offsets indicated on the left.
Limitations and Related Work
Discusses limitations, compares with Next Forcing, and outlines future directions.
Autoregressive video models trained with teacher forcing rely on local temporal redundancy; Future Chunk Prediction (FCP) adds an auxiliary loss that forces the model to predict distant future chunks, encouraging long‑term physical reasoning.
Next Forcing augments autoregressive training with an auxiliary predictor that conditions on the entire visible history, encouraging the model to anticipate future frames.
Both works aim at future prediction for autoregressive video generation, but they target different downstream tasks: Next Forcing focuses on robotic action learning, whereas we target high‑fidelity video generation as a step toward generative world modeling.
Next Forcing conditions its auxiliary predictors on the full visible history h ≤ k, while our design uses only the source‑chunk representation $h_k$. This forces the source chunk to encode predictive information itself.
Next Forcing adopts a chained prediction architecture, passing intermediate future chunks to later predictors; we instead predict multiple future offsets in parallel from $h_k$.
In the chained variant, a gain appears at 1.3 B parameters with three offsets, but the I2V gain is marginal and there is no V2V gain at 5 B parameters, suggesting information leakage becomes more harmful as model capacity grows.
Our approach also supervises both clean‑context and noisy‑target representations and injects source features into the auxiliary predictors via token‑wise AdaLN, reflecting our focus on visual evolution rather than action planning.
Although the auxiliary predictors can anticipate future dynamics, directly replacing the main generator with them degrades visual fidelity; they may be useful early in high‑noise denoising or in combination with the main generator to reduce sampling cost.
Current experiments are limited to models initialized from pretrained bidirectional diffusion checkpoints; training from scratch at larger scales or on longer videos remains unexplored.
We suspect that bidirectional and autoregressive models organize representations differently, which may limit the benefits of FCP when fine‑tuning from bidirectional checkpoints; future work will investigate large‑scale from‑scratch pretraining with FCP as a native objective.
Implementation Details
Implementation, training, and evaluation details for reproducibility.
Future Chunk Prediction (FCP) adds an auxiliary objective that predicts temporally distant video chunks during training; the auxiliary predictors are stripped away at inference, leaving the autoregressive generation pipeline unchanged.
Our code is released at https://github.com/TencentARC/Looking-Ahead; model checkpoints are omitted, and we provide links and selection criteria for the public portion of the training data while acknowledging that the full internal corpus cannot be shared.
Generative AI tools assisted with implementing FCP, training scripts, and translation, but were not used to generate datasets, formulate hypotheses, or perform qualitative analysis; the authors retain full responsibility for the final content.
**Table 5.** Model configurations. All layer indices are one-based. Hidden and FFN dimensions apply to both the backbone and its auxiliary DiT blocks.
We initialize generators from the official Wan checkpoints; source features are taken from four DiT blocks, concatenated, and fused by a two‑layer MLP ($4d arrow 2d arrow d$) with GELU activation. Each future offset has its own auxiliary branch (fusion MLP, patch embedding, DiT blocks, output head), sharing parameters between clean and noisy source pathways within the offset.
The table evaluates the impact of different source types and conditioning mechanisms on model performance metrics (IQ, SP, ST, WS, MSE). The columns represent "clean source", "noisy source", "text condition", "chained connection", and the metrics. Rows indicate different combinations of these features, with checkmarks denoting inclusion.
Training Wan2.1‑1.3B runs for 20 000 steps on eight GPUs with 85‑frame clips (768 × 432, 16 FPS) and a global batch size of eight. We use AdamW ($\beta_1=0$, $\beta_2=0.999$), weight decay $0.01$, gradient clipping $0.1$, and a learning rate linearly warmed to $1\times10^{-5}$ over the first 1 000 steps, then held constant.
The main teacher‑forcing loss receives weight $1$, while each future offset receives $w_{\Delta}=0.5$ (Eq. 7). Text prompts are replaced by an empty string with probability $0.1$, and an exponential moving average (EMA) of generator parameters (decay $0.9995$) starts at step 2 000.
Wan2.2‑5B follows the same hyperparameters but trains for 25 000 steps on 32 GPUs using 121‑frame clips (1024 × 576, 24 FPS) and a global batch size of 32.
Sampling uses UniPC with 50 denoising steps per chunk, classifier‑free guidance scale $3.0$, and a timestep shift of $5.0$. At inference we drop all FCP modules, keep EMA weights, and retain the first three latent frames as attention sinks when the sequence exceeds the training length.
For Physics‑IQ Verified we generate five‑second continuations at 16:9 resolution, conditioning on the provided initial image (I2V) or on the longest compatible suffix of a three‑second video (V2V). We run four seeds (42–45) and report the mean of the benchmark scores.
We also evaluate WorldModel‑Bench, PAI‑Bench‑G, and PhyGround in the I2V setting, resizing and cropping inputs to 1024 × 576, using best‑practice prompts, four seeds, and reporting mean scores.
Public‑model evaluation includes Wan2.2‑TI2V‑5B at 1280 × 704 (cropped to match our 1024 × 576 resolution) and Causal Forcing / Causal Forcing++ checkpoints, run with their default inference code and sampling settings; Stage 1 of Causal Forcing serves as a teacher‑forced baseline.