Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation
Songlin Yang, Zhe Wang, Xiaotong Zhao, Xi Wang, Ran Zhang, Shuai Li, Taiyi Wu, Songlian Li, Eric Liu, Alan Zhao, Anyi Rao
FAPL improves cinematic video generation by training MLLM latents to be actionable for temporal inference and factor-aware control.
How can we force MLLM-DiT video generators to actually use MLLM latents for temporal inference and factor-specific control instead of ignoring them in favor of easier, lower-quality shortcuts?
Multimodal Large Language Models (MLLMs) provide rich semantic priors, but Video Diffusion Transformers (DiTs) often bypass these latents in favor of simpler text or visual pathways, resulting in poor temporal inference and weak control over specific cinematic factors like camera motion or character identity. Factor-Aware Predictive Learning (FAPL) addresses this by isolating the MLLM pathway and using a three-stage training process: grounding the generator in MLLM latents, using paired editing examples to supervise factor-specific changes, and jointly optimizing the MLLM to predict video content from limited inputs. This approach consistently outperforms existing MLLM-conditioned baselines across sparse-to-video generation, factor-aware editing, and multi-shot consistency tasks.
Paper Primer
FAPL treats the MLLM-to-DiT connection as a learning bottleneck rather than a static interface. The core move is to force the generator to rely exclusively on MLLM latents during training, then use structured latent-span replacements to teach the model how specific prompt changes should manifest as visual edits.
The method functions like a specialized translator: it forces the MLLM to "speak" in a language the generator can act upon, then uses paired examples to teach the generator that changing a specific "word" in the latent sequence (e.g., the camera instruction) must trigger a corresponding visual change while keeping the rest of the scene stable.
FAPL significantly improves action sequencing and camera-trajectory fidelity compared to state-of-the-art MLLM-conditioned models.
In sparse-to-video generation, FAPL achieved a 61% Motion Order accuracy, compared to 48% for the next-best baseline (UniVideo 13B). A 13-percentage-point gain in action sequencing accuracy.
FAPL enables precise factor-aware editing without requiring fully disentangled latent representations.
On the FiVE benchmark, FAPL achieved 58% intersection accuracy (∩-Acc) for requested factor changes, outperforming the 42% baseline. A 16-percentage-point improvement in factor-specific editing accuracy.
Why does this approach use a three-stage training process instead of standard end-to-end fine-tuning?
Standard training allows the generator to bypass MLLM latents by relying on easier text or VAE pathways. FAPL’s stages isolate the MLLM pathway to force latent reliance, then introduce factor-specific supervision, and finally optimize the MLLM to produce generation-compatible representations.
What is the scope of this method — does it require a new architecture?
No, FAPL is a training framework that works with existing MLLM-conditioned Video DiT architectures. It improves performance by optimizing how the MLLM and generator interact, rather than by modifying the underlying model structure.
FAPL demonstrates that cinematic control is a training-signal problem, not just an architectural one; by explicitly supervising latent-to-visual mappings, researchers can unlock actionable control from existing MLLM priors.
The MLLM Latent Bypass Problem
We expose why MLLM latents are bypassed by DiT and outline our three‑stage solution.
Cinematic video generation must infer plausible action sequences from sparse inputs and keep character, scene, action, and camera factors independently controllable, yet current Video Diffusion Transformers (DiTs) often sidestep multimodal large‑language‑model (MLLM) latents, losing both temporal fidelity and factor disentanglement.
**Figure 1.** Cinematic video generation requires inference from limited inputs and factor-aware control. (i) The man turns toward the box, places his right hand on top, and steadies the bundle with his left hand. Text-only conditioning misses this sequence, while observing the complete video enables the same model to reproduce it. (ii) Keeping the descriptions of all other factors fixed and replacing only the character-related prompt, the non-character factors change as well. These cases illustrate failures in temporal inference and factor control, with editing providing a probe of what the generator changes and preserves.
**Figure 2.** MLLM latents are easily bypassed in existing MLLM–DiT models. (i) Under MLLM-only conditioning, Omni-Video 2 (1.3B DiT) produces noise, while Omni-Video 2 (14B DiT) and UniVideo produce artifacts and depict one person’s successive actions as two people acting differently. MLLM latents alone therefore fail to provide reliable generation guidance in these examples. (ii) Under conflicting conditions, VAE references dominate the output.
The core gap is that MLLM capabilities remain under‑utilized by DiT generators, limiting both temporal inference and factor‑aware control.
Cinematic Video Generation Context
We position our method among prior cinematic video generation and predictive learning approaches.
Cinematic video generation has attracted attention for producing shots with specified content, performances, and camera motions from limited inputs. Prior work separates the problem into (i) inferring content from sparse references—via reference‑driven or multi‑shot methods that preserve identity and scene consistency—and (ii) factor‑aware control, where camera motion, character replacement, action transfer, and scene modification are handled independently.
MLLM‑conditioned video generation builds on recent diffusion models that achieve high‑quality synthesis and support diverse editing tasks. Existing systems either prepend MLLM features through in‑context conditioning or attach an MLLM understanding branch to a generation branch, sharing a conditioning interface and supervising at the video level; however, they often let the generator rely on familiar text and dense visual‑reference pathways, and treat paired editing examples as extra samples without explicit factor‑level guidance.
Predictive representation learning uses joint‑embedding architectures to predict target features from partial observations, avoiding low‑level pixel reconstruction, and extends to action‑conditioned future prediction for planning. Unlike prior approaches that require task‑specific spaces or separate decoders, our method reuses a pretrained MLLM and Video DiT, supervises prediction through visual reconstruction with a fixed generator, and employs factor‑span replacement between paired examples to enforce factor‑aware changes and preservation.
Factor-Aware Predictive Learning
Method Overview explains the three‑stage Factor‑Aware Predictive Learning pipeline.
Current video generators often ignore the rich latent representations produced by large language models, taking easy shortcuts that limit factor‑level control. This shortcut undermines the ability to edit characters, scenes, actions, or camera motion reliably.
FAPL forces the Video DiT to depend on MLLM latents by structuring supervision around factor‑specific temporal prediction tasks.
How does FAPL differ from ordinary diffusion training that already uses MLLM embeddings?
Ordinary training treats the MLLM embedding as just another conditioning channel and optimizes reconstruction loss; FAPL adds explicit factor‑span interventions and a three‑stage schedule that forces the DiT to both use the embedding and respect factor‑level edits, preventing the model from ignoring the latent.
Each cinematic factor (character, scene, action, camera) is assigned a distinct token span in the MLLM hidden state; swapping those spans between paired videos teaches the DiT to change only the targeted factor while leaving the others untouched.
Identify the character span in both prompts (single token each).
Swap the character tokens, producing a new prompt “Bob | park | running | static”.
Pass the swapped prompt through the MLLM, obtain hidden state $H_{\theta}$, and map it with the connector $P_{\eta}$ to get $Z_{\theta,\eta}$.
Train the DiT with the editing loss so that the generated video reflects the new character while preserving the original scene, action, and camera.
Swapping only the character span forces the generator to attribute visual changes to that specific latent region, demonstrating that the latent truly encodes the factor.
Why not simply replace the entire latent vector instead of swapping a factor span?
Replacing the whole vector would also change all other factors, providing no signal about factor‑level disentanglement. Span‑level swapping isolates the effect of a single factor, allowing the model to learn that only the edited region should drive the corresponding visual change.
Stage I: Freeze the MLLM and connector; update the DiT $\phi$ using the flow‑matching loss $L_{\text{FM}}$ on observation data $D_{\text{obs}}$.
Stage II: Keep the MLLM frozen; replace the factor‑$k$ span in $Z$ with the span from a paired video; update the DiT $\phi$ using the editing loss $L_{\text{edit}}$ on $D_{\text{edit}}$; then freeze the DiT.
Stage III: Freeze the DiT; update the MLLM $\theta$ and connector $\eta$ using $L_{\text{FM}}$ on the same observation data, now with the DiT fixed, so that limited‑input conditioning learns to predict the full target.
**Figure 3.** Overview of Factor-Aware Predictive Learning (FAPL). Stage I freezes the MLLM and its original connector and trains the DiT using MLLM latents. Stage II retains this configuration and swaps factor-specific output spans between paired examples to supervise factor-aware changes. Stage III freezes the trained DiT and jointly updates the MLLM and connector through video reconstruction, and text-only and text-plus-complete-video inputs share the same generation target.
Training Stages I and II
Stages I–II train DiT to rely on frozen MLLM latents using mixed observations and factor‑span replacement.
The DiT can sidestep the MLLM latents, reducing generation loss by exploiting easier shortcuts. To compel the generator to actually decode the frozen latent representations, we introduce a two‑stage training regime.
Instead of feeding the DiT multiple content streams, we give it only the frozen MLLM latents, forcing it to learn to reconstruct video from that signal alone.
How does this training differ from ordinary supervised video prediction?
Standard supervision provides the raw video frames as direct conditioning inputs. Here the DiT never sees the frames; it must reconstruct them solely from the frozen MLLM latents, which removes the easy shortcut and forces the generator to internalize the latent semantics.
Freeze the pretrained MLLM and its connector; set DiT parameters $\phi$ as the only trainable variables.
Sample a video–description pair $(x,p)$ from the observation dataset $D_{\text{obs}}$.
Compute latents $Z_{\theta_0,\eta_0}(u_t)$ and $Z_{\theta_0,\eta_0}(u_{tv})$ for the two observation modes.
Feed each latent set to the DiT, obtain predicted videos $\hat{x}_t$ and $\hat{x}_{tv}$.
Evaluate $L_{\text{FM}}$ for both predictions, weight by $\rho_t$ and $\rho_{tv}$, and back‑propagate the summed loss to update $\phi$ only.
Having forced the DiT to depend on the frozen latents, Stage II adds factor‑level control by swapping only the latent slice that corresponds to a chosen cinematic factor.
We replace the latent slice that encodes a single factor while keeping all other slices unchanged, so the DiT learns a direct mapping between that factor and the resulting visual change.
Identify the color factor span (first entry) in each latent vector.
Replace $Z_a$’s first entry with $Z_b$’s first entry: $Z_{a\leftarrow b}^{\text{color}}=[0.7,0.5,0.1]$.
Feed $Z_{a\leftarrow b}^{\text{color}}$ to the DiT; the generated video shows a blue ball instead of a red one.
Similarly, $Z_{b\leftarrow a}^{\text{color}}=[0.2,0.5,0.1]$ yields a video with a red ball.
Swapping only the factor‑specific slice forces the DiT to associate that slice with the visual change, isolating the factor’s effect while leaving all other content unchanged.
Sample a pair of captions $(p_a,p_b)$ from $D_{\text{edit}}$ and compute their frozen latents $Z_a$, $Z_b$.
Select a factor $k$ (e.g., color, motion) and apply $R_k$ to obtain $Z_{a\leftarrow b}^{k}$ and $Z_{b\leftarrow a}^{k}$.
Feed each augmented latent to the DiT, generate videos $\hat{x}_{a\leftarrow b}^{k}$ and $\hat{x}_{b\leftarrow a}^{k}$.
Supervise $\hat{x}_{a\leftarrow b}^{k}$ with the target video $x_b$ and $\hat{x}_{b\leftarrow a}^{k}$ with $x_a$ using the same $L_{\text{FM}}$ loss.
Back‑propagate the losses to update $\phi$ while keeping $\theta_0,\eta_0$ frozen.
The Editing Objective
Fine‑tune the MLLM pathway while the DiT generator stays fixed.
Recall that the earlier stages taught the DiT to react to factor‑aware interventions. Stage III now leverages this frozen generator to shape the MLLM pathway itself.
Swap a factor latent from video $a$ into video $b$ (and vice‑versa) and ask the generator to reconstruct the target video, thereby teaching the model to be aware of individual cinematic factors.
Why does the loss include both swap directions instead of just one?
Training both $Z_{a\leftarrow b}^{k}$ and $Z_{b\leftarrow a}^{k}$ prevents the model from over‑fitting to a single direction; it forces the generator to learn a symmetric mapping that can both add and remove a factor, yielding robust factor awareness.
Train the MLLM and its connector to produce the same video output whether they see only the text prompt or the text plus the full video, while the DiT generator stays frozen at $\phi^{*}$.
Compute the weighted loss for the text‑only mode: $0.6 \times 2.5 = 1.5$.
Compute the weighted loss for the text‑plus‑video mode: $0.4 \times 1.0 = 0.4$.
Sum the two contributions: $1.5 + 0.4 = 1.9$.
Take the expectation over the single sample (trivial here), yielding $L_{\text{obs}} = 1.9$.
This toy calculation shows how the mixed‑observation loss balances the easier text‑only signal against the richer video signal, encouraging the MLLM to learn representations that succeed under both conditions.
Why share the same target video $x$ across both observation modes?
Using a common target forces the MLLM to produce a latent that is compatible with the fixed generator regardless of input richness. If each mode had a different target, the MLLM could simply memorize mode‑specific shortcuts, defeating the purpose of learning a unified, generation‑compatible representation.
At inference time the trained system supports three cinematic uses. (i) Generation from scripts: the MLLM encodes a textual script (and optional references) into latents that condition the frozen DiT, which then samples and decodes a video. (ii) Factor‑aware control: a user edits a single factor field in the prompt, the MLLM re‑encodes the edited prompt, and the corresponding latent span replaces the original in the DiT input, leaving all other spans untouched. (iii) Multi‑shot generation: each shot is generated independently from its description, while shared character or scene references are kept consistent across shots.
Implementation and Baselines
Implementation specifics of FAPL, training regime, and quantitative comparison to baselines.
We implement Factor‑Aware Predictive Learning (FAPL) on top of the UniVideo framework, using a 13B Diffusion Transformer (DiT) and compare against the original UniVideo and Omni‑Video 2 models (1.3B and 14B DiTs).
UniVideo is a video‑synthesis system that trains a Diffusion Transformer on large‑scale video‑text pairs using a standard diffusion pipeline.
Omni‑Video 2 extends the diffusion‑based video generation paradigm with larger model capacities (1.3B and 14B DiTs) and additional conditioning mechanisms.
Training runs for 10 K, 15 K, and 10 K steps on stages I–III respectively, using AdamW with learning rate $1 \times 10^{-5}$, zero weight decay, and a maximum gradient norm of $1.0$.
At inference we perform 50 diffusion steps with a classifier‑free guidance (CFG) scale of $5.0$.
Evaluation and Results
FAPL outperforms baselines across video generation tasks, with the strongest metric gains highlighted.
FAPL attains the highest FaceSim‑Arc score among evaluated video generation models.
Table 1 reports a FaceSim‑Arc of 0.523 for FAPL, versus 0.412 for Omni‑Video 2 (1.3 B), 0.468 for Omni‑Video 2 (14 B), and 0.451 for UniVideo (13 B).
The table compares the performance of different video generation methods across three categories: Sparse-to-Video, Factor-Aware Editing, and Multi-Shot Consistency. The methods evaluated are Omni-Video 2 (1.3B), Omni-Video 2 (14B), UniVideo (13B), and the proposed FAPL (Ours).
**Figure 4.** Under limited inputs, baseline models fail to predict temporal visual details. With only text and no visual references, we compare UniVideo with FAPL (Ours), highlighting representative baseline limitations across four cinematic factors. (i) Character. Long hair is generated as short hair, failing appearance details specified only by text. (ii) Scene. The background is not preserved and incorrect content appears. (iii) Action. The requested sequence, e.g., gesturing outward with the left hand and extended fingers, is missing. (iv) Camera. UniVideo ignores the instruction and keeps the camera static, while Ours orbits around the character. FAPL better realizes prompt-specified appearance, scene, and motion from text alone.
**Figure 5.** FAPL improves factor-aware control. For each factor, we modify only the corresponding description in the structured prompt and generate twice, comparing UniVideo with FAPL (Ours). UniVideo does not confine the change to the requested factor and alters unrelated content, whereas Ours modifies the selected factor while preserving the others. This indicates that factor-level latent augmentation enables selective control.
**Figure 6.** FAPL supports consistent multi-shot generation without cross-shot memory.
Ablation Study
Ablations isolate the impact of each FAPL training stage on motion ordering, camera control, and editing accuracy.
We evaluate three progressive training stages of Factor‑Aware Predictive Learning (FAPL) on the UniVideo (13B) model. Each stage adds a supervision component, allowing us to measure how that component contributes to motion ordering, camera control, and factor‑aware editing.
**Table 2.** Ablation of FAPL training stages on UniVideo (13B).
Removing factor swapping (Stage II w/o swapping) collapses editing accuracy back to the pretrained level, confirming that the swapping supervision is the driver of the 9 % gain. Adding full video observations (Stage III full) simultaneously lifts motion ordering to 61 % and cuts both translation and rotation errors, showing that complete‑video context benefits temporal coherence without harming factor‑aware editing.