Untitled

Guian Fang, Shengju Qian, Dongyue Lu, Junchao Huang, Lutao Jiang, Xin Wang, Yiren Song, Li Jiang, Mike Zheng Shou

A universal, few-step generative renderer that follows arbitrary structural inputs using a single frozen backbone.

How can we turn arbitrary structural inputs (like wireframes or skeletons) into temporally coherent, realistic video using a frozen generative backbone?

Video generation models often struggle to maintain consistent 3D geometry or temporal coherence when tasked with following specific scene layouts or camera paths. Existing renderers are typically tied to a single, engine-specific input format, requiring a new model for every new type of structural proxy. This paper introduces a renderer that feeds any structural video—from semantic maps to wireframes—directly into the native reference pathway of a frozen video backbone. By using a single control adapter and a structure-guided distillation process, the model achieves high-fidelity rendering in just four steps. The resulting system outperforms existing structure-conditioned renderers and generalizes to 11 unseen structural representations, while rendering 768p video in real time.

Paper Primer

The core mechanism hinges on treating structural inputs as untyped, frame-aligned references within the backbone's existing attention architecture. Instead of adding custom branches or type identifiers, the model uses a single control adapter to learn layout and motion from whatever structural representation occupies the target frame's position.

To enable fast inference, the authors employ structure-guided distribution matching: they post-train a four-step acceleration adapter by extrapolating the teacher's structural signal. This forces the student to prioritize the structural condition, effectively sharpening the output and preventing the blurriness typical of standard distillation.

The model achieves state-of-the-art performance on both in-distribution and generalization benchmarks.

Pairwise human-preference judgments (Elo ratings) against existing renderers like StreamRender-H3 and LynnReal-Omni. Outperforms existing renderers on 11 unseen structural representations, including pencil storyboards and point clouds, with a 12.3x speedup over 50-step sampling.

Why does this approach generalize to unseen structural representations when others fail?

Because the model uses untyped references and fixed text descriptions rather than hard-coded type identifiers or modality-specific branches, the control adapter learns to interpret structural layout and motion as a general visual task rather than a format-specific one.

What is the primary trade-off of using a causal streaming variant?

The causal variant enables real-time 768p video generation by processing clips in small, independent blocks, though it incurs a slight drop in overall visual quality compared to the bidirectional model.

This work demonstrates that a single, frozen video backbone can serve as a universal renderer for any structural input, provided the data is frame-aligned and the distillation process is explicitly guided by the structural condition.

Introduction

We expose the gap between visual realism and geometric consistency in video generation.

Current video generators excel at producing photorealistic footage, yet they often drift in geometry, dynamics, and temporal consistency, which limits their use in interactive content creation.

Realistic video is impressive, but without reliable 3D structure and motion the output cannot serve downstream tasks that require precise control.

**Figure 1** SolarRenderer renders structure of any kind into realistic video. Each wedge pairs a structure (inner ring) with its rendering, and seven of the ten structure types are unseen in training. Readers can click and play the animation using Adobe Acrobat.

The key gap is that visual realism alone does not ensure geometric and temporal fidelity, which is essential for controllable video generation.

Related Work

We review prior video rendering and control methods, noting their reliance on engine-specific representations.

Recent systems let a game engine, simulator, or coding agent dictate scene content while a video model renders it. Most train a renderer on a single structural representation—white‑box frames, semantic maps, depth/normal maps, projected 3D boxes, or proxy video of a code world model—and distill the renderer to a few streaming steps for real‑time use. However, each renderer is tied to its engine’s representation, so new engines or proxy types require new renderers.

Controllable video generation typically injects conditions either via branch‑based modules attached to the backbone (e.g., ControlNet and its video variants) or by placing condition tokens alongside target tokens and processing them with a small LoRA. Additional methods steer generation through warped noise, 3D tracking videos, reference videos, predicted LoRAs, or jointly generated intrinsic channels, and some training‑free approaches condition a model using structured first frames or a ControlNet‑style world model. These approaches either add parameters for control, require explicit labeling of condition types, or are trained on a fixed set of condition kinds, limiting flexibility.

Few‑step video generation is achieved either by distilling a multi‑step teacher into a few‑step student via distribution matching or consistency and flow‑map distillation, or by training a plug‑in acceleration LoRA on a base model and reusing it across derived models, with recent work enabling LoRA transfer between base and distilled models. For conditional models, some works distill the conditional model itself or re‑tune its control module after distillation, while MotionEcho restores motion control at inference using guidance from the slow teacher. Yet, distilling a large control model from scratch is costly, and generic few‑step adapters can degrade visual quality and first‑frame fidelity, whereas post‑training a four‑step LoRA on the frozen control model with a distribution‑matching teacher improves performance.

The SolarRenderer Architecture

Inject structural latents into a frozen video backbone via lightweight adapters.

Adapting a large pretrained video model to render arbitrary structural inputs is costly if we retrain the whole backbone; feeding the references through the model’s native pathway avoids that expense.

SolarRenderer treats a frozen video backbone as a black‑box renderer and injects structural information via tiny adapter modules, so the heavy appearance knowledge stays untouched while control comes from the adapters.

MiniMax‑H3 is the underlying DiT video model that provides the high‑fidelity appearance prior used by SolarRenderer.

Structural latents are the encoded representations of each frame of the structural video $S$, placed at the same RoPE coordinates as the corresponding target latent $\mathbf{z}_k$.

Instead of adding a new ControlNet branch, SolarRenderer feeds structural frames and the optional first frame $I$ through the backbone’s existing reference pathway, letting the frozen model read them as native inputs.

Assign RoPE coordinates: $\mathbf{s}_1$ gets position (t=1, h=0, w=0), $\mathbf{s}_2$ gets (t=2, h=0, w=0), $\mathbf{s}_3$ gets (t=3, h=0, w=0).

Insert reference tokens before target tokens, yielding the sequence [ref $s_1$, ref $s_2$, ref $s_3$, tgt $z_1$, tgt $z_2$, tgt $z_3$].

The attention layer computes interactions; because reference and target tokens share positions, the model can directly copy layout information from $\mathbf{s}_k$ to $\mathbf{z}_k$.

After the adapters $\phi$ and $\psi$ adjust the latent stream, the decoder produces video frames that respect the structural layout while adding appearance details.

Aligning reference and target tokens at identical RoPE positions lets the frozen backbone treat structural cues as native context, eliminating the need for a separate control branch.

How does the native reference pathway differ from adding a ControlNet branch?

ControlNet introduces a new set of parameters that must be learned from scratch to fuse structural inputs, whereas the native pathway reuses the backbone’s existing attention stack; only the lightweight LoRA adapters $\phi$ and $\psi$ are trained, so the parameter count and training cost are dramatically lower.

Injecting structural latents via the native reference pathway.

Decoupled Two-Stage Training

Decoupled two-stage training isolates control and acceleration to enable fast sampling with retained control.

Training the control and acceleration objectives together fails because updates to the acceleration adapter also disturb the learned structural control. Joint training therefore cannot keep the fine‑grained control while reducing sampling steps.

Separate the learning of structural control from the speed‑up of denoising so each can be optimized without hurting the other.

Generate random matrices $B\in\mathbb{R}^{4\times2}$ and $A\in\mathbb{R}^{2\times4}$.

Compute $\Delta W_{\text{ctrl}} = (\alpha/r)\, B A$, a $4\times4$ low‑rank update.

Add the update to the frozen weight: $W = W_0 + \Delta W_{\text{ctrl}}$.

The resulting $W$ differs from $W_0$ only in a rank‑2 subspace, yet it can steer the backbone’s attention and feed‑forward projections.

Even a tiny low‑rank update can steer the frozen backbone’s projections, providing the full‑step structural control.

Why can’t we simply train the control and acceleration LoRAs together in a single stage?

Training them jointly lets the acceleration updates modify the same linear layers that the control adapter uses, so the structural guidance is overwritten. By freezing the control adapter during Stage 2 we guarantee that the learned control signal remains unchanged while the acceleration LoRA learns to speed up sampling.

Initialize a LoRA adapter of rank $r$ for each linear projection in the frozen transformer.

Freeze the backbone weights $W_0$ and train only the LoRA factors $A$ and $B$ using the flow‑matching loss $L_{\text{FM}}$.

The loss is evaluated on target video tokens while reference tokens share the same adapted projections.

After convergence, keep the trained control adapter $\phi$ fixed for the next stage.

Load a pretrained four‑step LoRA (rank 128) that already predicts a clean sample after four denoising steps.

Initialize a higher‑capacity LoRA of rank 384, copying the pretrained weights into its first 128 ranks and zero‑initializing the remaining ranks.

Compose the acceleration LoRA $\psi$ with the already‑trained control adapter $\phi$, forming $W = W_{0} + \Delta W_{\text{ctrl}} + \Delta W_{\text{acc}}$.

Freeze $\phi$ and train only the acceleration factors of $\psi$ using the structure‑guided distribution‑matching loss.

Even after Stage 2 the student still deviates from the full‑step teacher, so we guide it with a distribution‑matching loss that emphasizes the structural signal. The teacher’s structural contribution is isolated by removing the structural video tokens, and a weight $w>1$ amplifies this signal to push the student toward a sharper target.

The teacher’s score is modified by a guidance term that isolates the structural contribution: $0 = \hat{z}_{0}(C) + (w - 1)\bigl(\hat{z}_{0}(C) - \hat{z}_{0}(C^{\emptyset})\bigr)$. Setting $w=1.5$ amplifies the structural signal, making the student aim for a target sharper than the teacher itself.

Causal Streaming and Data

Extends two‑stage training to incremental video generation via block‑causal streaming and aligned paired data.

Generating video incrementally requires a mechanism that can attend only to past information while still respecting the structural prompt. The authors extend the two‑stage design with block‑causal attention to meet this need.

Each video is broken into short blocks; a block sees the prompt, its own structure, and a fixed window of previously generated blocks, but never any future block.

Block 1 attends to the prompt and its own structure; no past blocks exist.

Block 2 attends to the prompt, its own structure, and Block 1 (the only preceding block).

Block 3 attends to the prompt, its own structure, and Blocks 1‑2 (two preceding blocks).

Block 4 attends to the prompt, its own structure, and Blocks 1‑3 (the five‑block window is not yet full).

Block 5 attends to the prompt, its own structure, and Blocks 1‑4 (window now contains five blocks).

Block 6 attends to the prompt, its own structure, and Blocks 2‑5 (the oldest block, 1, is dropped from the cache).

The cache slides forward each time a new block is generated, guaranteeing constant‑size attention while preserving all necessary past context.

How does Causal Streaming differ from ordinary causal attention used in language models?

Ordinary causal attention lets each token see all previous tokens, but it does not enforce a fixed‑size context window or structural conditioning. Causal Streaming adds a bounded window of five blocks, freezes a control LoRA that injects prompt‑guided structure, and trains a separate acceleration LoRA on teacher‑forced rollouts, all of which are absent in standard causal attention.

To train the streaming model, the authors need paired data where the structural condition exactly matches the target video at test time. Mismatched conditions would teach the model the wrong geometry.

The training set comprises 136 K samples (≈243 hours) of 158‑frame clips at 24 fps and 768p resolution, drawn from several sources that provide different structural cues.

Sources include Magpie (Unreal Engine gameplay with a grey block‑out layout), CARLA (semantic segmentation masks), engine recordings (depth, masks, poses, skeletons), Blender renders (white‑box scene layout), UnrealZoo (object masks + skeletons), YouTube videos (SMPL body pose), WildWorld (appearance only, first frame), and OmniWorld (appearance only, first frame).

Two checks filter each source. Temporal‑spatial agreement correlates motion to ensure zero offset and measures edge recall against a mirrored version; clips failing either test are discarded. Sufficiency checks replace coarse box proxies with depth shading (within 30 m) to provide enough geometric detail for the model to learn shapes.

Prompting and Input Combinations

Defines the fixed six‑section prompt format and input‑combination strategy that drive training.

During early experiments the model learned to infer missing structure from the surrounding prompt text, causing unintended leakage and unstable control.

The prompt is split into six fixed parts so the model receives a complete, self‑contained description of every required modality.

Section 1 inserts the two reference names.

Section 2 adds the task prefix and high‑level goal.

Section 3 lists the attributes to retain (wave height, sand texture).

Section 4 expands the visual details of the beach.

Section 5 appends the ambient sound description.

Section 6 finishes with the music cue.

The six‑section scaffold guarantees that every modality (visual, structural, audio) is explicitly mentioned, eliminating any hidden dependence on missing inputs.

How does this six‑section prompt differ from a typical free‑form text prompt used in conditional generation?

Typical prompts intermix content arbitrarily, leaving the model to infer which words refer to structure versus style. Here each modality occupies a dedicated slot, so the model never has to guess what information belongs where; the prompt’s shape itself encodes the control signal.

Each training clip is assigned a single, immutable combination of inputs, so the prompt never mentions an input that is absent.

During data construction the combo “structure + first frame” is recorded for this clip.

Each epoch the prompt generator reads the stored combo and produces a prompt that mentions both the structural video and the first frame.

The model therefore never sees a prompt that mentions structure while the structural video is missing.

Fixing the combo per clip removes any stochastic mismatch between what the prompt says and what inputs are actually provided.

Experimental Results

SolarRenderer beats all baselines on the generalization benchmark.

SolarRenderer injects structural latents into a frozen video backbone via lightweight adapters, then trains with a decoupled two‑stage process to balance control and speed.

SolarRenderer attains the highest Elo rating on the generalization benchmark, surpassing every competitor.

Rating 1441 versus 580 for StreamRender‑H3 and 1075 for Streaming CWM.

The benchmark evaluates how well a renderer preserves visual quality, temporal stability, structural control, and first‑frame fidelity when presented with structural inputs it has never seen during training.

**Figure 4.** Multiple appearances from unseen structural representations. The left column shows, from top to bottom, a 2D skeleton, a crayon drawing, a normal map, and a wireframe. The remaining columns show our generated appearances at the same time.

**Figure 5. Results by difficulty tag.** Net score of ours against each compared method on the generalization benchmark. Positive values favor ours.

**Figure 6. Results by representation.** Net score of ours against Stage 1 + LongLive-Plug (left) and our 50-step Stage 1 model (right) on the generalization benchmark, grouped by representation category. Positive values favor ours.

**Figure 7.** Visual comparison with other methods. Top to bottom: structural condition, results from other works, and ours. Left to right: SceneCraft, Programmable World Model, Cosmos-Transfer2.5, Magpie, and CWM, with frames synchronized within each example.

**Figure 8.** A gallery of structure-conditioned renderings. Each image comes from a different case, spanning human motion, everyday scenes, and imagined environments across seen and unseen structural representations.

**Figure 15.** Pairwise margin of ours over each compared method. Net score of ours against each indicated method for each judge question on the two test sets. Positive values favor ours.

**Figure 16.** Results by representation and source on the generalization benchmark. Each bar splits the cases of one representation into wins, ties, and losses of ours against Stage 1 + LongLive-Plug (left) and against our 50-step Stage 1 model (right), with the number of cases in parentheses and the net score at the end of the bar. Bold rows match the categories in Figure 6, with individual representations and sources below. Positive values favor ours.

**Figure 17.** Visual comparison with other methods. Overall net score of ours against the outputs of 15 works on demonstration cases from their project pages. Positive values favor ours.

SolarRenderer outperforms baselines across unseen structural representations, delivering higher quality, stability, and control while running up to 12× faster.

Implementation Details and Ablations

Ablation details, evaluation metrics, and method comparisons for the appendix.

The frozen MiniMax‑H3 33B Ref2VA backbone receives a LoRA adapter of rank 384 on both attention and feed‑forward layers. The control LoRA is trained for 38 400 steps with batch size 4 and learning rate 10⁻⁴, then frozen for structure‑guided post‑training. Subsequent LoRAs (four‑step, causal control, causal acceleration) use smaller batch sizes and distinct learning rates as described.

Metrics are computed per case and averaged across the test set; structural video frames $I_t$ are embedded with DINOv2 ViT‑S/14 as $f(\cdot)$. Visual quality includes DOVER, Aesthetic Score, and VideoScore2, while temporal stability uses Flicker and SDINO, and structural adherence uses MotionError and Edge F1. Scores are normalized to 0–100, and pairwise judgments are aggregated into Elo ratings via a Bradley–Terry model.

Training captions embed structural references and a fixed definition line, then describe the target video in 350–500 words. The caption pipeline validates six fields, reference tags, language constraints, and shot ordering, retrying up to three times on failure. Caption validation uses the Kimi K2.6 model to enforce these rules.

A source audit records dataset provenance, preprocessing steps, and any modifications applied before training. The audit ensures reproducibility and flags potential biases in the structural video inputs. Detailed audit logs are provided in the supplementary material.

Extended results include per‑metric breakdowns and further ablation studies omitted from the main text. These ablations isolate the impact of each LoRA component and training schedule on the automatic score. The findings corroborate the trends reported in the primary experiments.

Data Audit and Limitations

Limits and detailed audit of data sources, plus extended experimental analyses.

In the source audit (B.3) the synchronization filter kept 32,027 of 44,266 clips; rejected clips were mainly due to shot changes, static content, structural mismatches, and darkness, with additional losses from near‑duplicates, black frames, and unmatched jumps.

Engine recordings were aligned to RGB timestamps for depth, masks, camera poses, and skeletons; 80 of 84 motion windows aligned best at zero offset, and depth‑camera reprojection showed a median error of 0.06 %.

The near‑range depth conversion retained 13,700 of 13,702 clips. A finer proxy built from 0.25 m blocks was examined but discarded because its flicker was four to seven times higher than the chosen proxy.

For YouTube videos the pipeline kept 137 of 559 videos that featured a clear protagonist; the target stream and the SMPL‑fitted stream differ by roughly two grey levels per pixel, while mesh videos match the target in crop, resolution, length, and frame rate.

Detailed results (C.1) focus on 74 unseen structural representations (Table 6). Our method attains a structure rating of 1360 versus 1275 for LynnReal‑Omni, an overall rating only six points behind the competitor, higher quality, and a slightly lower first‑frame rating. The four‑step rendering improves all four questions over Stage 1, yielding a total automatic score of 70.6 compared to 70.4.

Pairwise margins are non‑negative on both test sets; against LynnReal‑Omni the structure margin is +0.27, and all four questions favor our approach on the in‑distribution set. Compared with Stage 1 + LongLive‑Plug, we see positive margins across all broad groups, especially on pose inputs and shaded geometry.

When evaluated per representation, the four‑step model wins on six of eight agent‑built skeleton cases and six of seven silhouettes, but loses on colour‑mapped depth, voxels, and pencil storyboards, illustrating that structure‑guided post‑training benefits many formats while some still favor full‑step sampling.

Additional ablations (C.2) examine post‑training settings (Table 3) and structure‑guidance strength (Table 2). Guidance improves quality and control from every starting point, though stability and first‑frame preservation drop, slightly lowering the equally‑weighted total despite higher Elo margins.

Post‑training with DMAD raises quality and control but reduces stability and first‑frame scores; increasing guidance weight to w=2 further lowers stability with little change to control. Using eight sampling steps improves quality marginally but reduces first‑frame preservation, leaving the total unchanged, while two steps achieve the highest control but lower first‑frame scores. Four steps strike the best balance, attaining the highest overall, quality, and first‑frame Elo ratings.

Conditioning design analysis (Table 7) shows that the native full‑resolution pathway improves stability and control over the ControlNet branch, with only a small quality reduction. Raising structural resolution boosts control from 46.6 to 48.0 and the total from 66.2 to 66.5, while half‑resolution with shared RoPE improves quality and control compared to separate positional encodings.

Training‑data experiments reveal that using only structured samples raises control but lowers stability and first‑frame preservation, keeping the total unchanged. Removing misaligned structural supervision improves quality, stability, and control, raising the total from 66.2 to 66.5. Reducing the dataset to one‑tenth of its size has a smaller impact than the variation between full‑data runs, underscoring the importance of alignment over sheer volume.

The highest net‑score list highlights Safka blog (+1.00) and StreamRender‑H3 (+0.88), while the lowest net‑score list shows Code Video Model (−0.67) and other methods with negative margins, summarizing the relative performance extremes across the benchmark.

**Table 3.** Ablation on post-training settings. Gemini Elo ratings on the generalization benchmark from one fit over this table, comparable only within it. Each block varies one setting of our model, listed in the last row.

**Table 2.** Ablation on structure guidance. Gemini Elo ratings on the generalization benchmark, fitted per starting point and comparable only within a block. $w = 1$ means no guidance, and bold marks the best value per block.

**Table 1.** Comparison of various methods across Generalization benchmark and In-distribution test set metrics.

Conclusion

Summarizes the final performance, ablations, and design insights of the method.

Our method delivers state‑of‑the‑art video quality while rendering 12.3× faster than the traditional 50‑step sampling pipeline. The block‑causal variant further enables real‑time 768p video generation by respecting temporal causality within each block.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers