Generative World Renderer at the Speed of Play

Guixu Lin, Zheng-Hui Huang, Siqi Yang, Ming-Hsuan Yang, Kaipeng Zhang, Zhixiang Wang

AlayaRenderer-Flash enables real-time, prompt-controllable generative rendering by distilling offline diffusion into a streaming, four-step autoregressive pipeline.

How can we generate high-fidelity, prompt-controllable stylized video frames from game engine G-buffers in real-time?

Generative renderers can transform raw physics-engine data into stylized visuals, but their high computational cost and bidirectional processing make them too slow for live, interactive gameplay. AlayaRenderer-Flash reformulates this process as an autoregressive streaming model, distilling the original 50-step denoising schedule into a four-step process and replacing heavy codecs with lightweight, distilled alternatives. This pipeline sustains 31.54 FPS on a single GPU, enabling continuous, prompt-controllable rendering that keeps pace with live game engine output.

Paper Primer

The core mechanism hinges on a three-tier history compression hierarchy: the model retains recent frames at full fidelity while progressively compressing older history, allowing the renderer to maintain temporal consistency over unbounded streams. To ensure prompt controllability during these long rollouts, the model uses a persistent text sink in every self-attention layer, effectively treating the style prompt as a continuous conditioning signal.

The distillation process follows a three-stage trajectory: guidance distillation converts classifier-free guidance into a single-pass velocity field, progressive step reduction stabilizes the model as it shrinks the denoising budget, and Mean Flow Distillation (MFD) refines the final four-step trajectory. Lightweight GAN heads are attached to the final layers to recover high-frequency textures lost during this aggressive step reduction.

AlayaRenderer-Flash achieves real-time performance without sacrificing structural fidelity or prompt controllability.

Comparison against the original AlayaRenderer and external baselines (RGB↔X, FrameDiffuser) on a 5-second G-buffer benchmark. Throughput increases from 0.56 FPS to 31.54 FPS, while peak VRAM usage drops from 30.1 GB to 16.2 GB.

Why is the autoregressive streaming formulation necessary for this system?

The original AlayaRenderer used a bidirectional, fixed-window approach that required future context, making it impossible to process the unbounded, real-time G-buffer streams produced by a live game engine.

How does this method handle prompt changes during a live session?

Because the model processes the stream autoregressively with a persistent text sink in its attention layers, users can update the style prompt at any time without restarting the rendering pipeline or interrupting the game.

Introduction and Motivation

We expose the gap: text‑only video generators lose scene structure, prompting a G‑buffer‑conditioned solution.

Generative video models that rely solely on text prompts or control hints often lose the underlying scene structure, material properties, and gameplay logic as the world evolves.

We want a renderer that keeps the physics‑engine’s exact world state while letting a generative model dictate only the visual appearance.

AlayaRenderer demonstrates high‑quality, G‑buffer‑conditioned generation but its 50‑step denoising pipeline limits throughput to 0.56 FPS, making it unsuitable for interactive use.

AlayaRenderer‑Flash reforms the model into a few‑step autoregressive streaming architecture, distills the denoiser to four steps, and replaces heavyweight encoders/decoders with lightweight distilled codecs, boosting speed to 31.54 FPS while keeping the original G‑buffer and text‑prompt interfaces; it outputs frames in Red Green Blue (RGB) format.

The shift from text‑only generation to G‑buffer‑conditioned generation enables real‑time, structure‑preserving, prompt‑controllable video rendering.

The AlayaRenderer-Flash Pipeline

AlayaRenderer‑Flash turns synchronized G‑buffer streams into real‑time stylized video.

Real‑time stylized rendering demands both geometric fidelity and low latency, yet naïve diffusion pipelines stall at dozens of denoising steps and require multiple full‑resolution VAE passes per frame.

A G‑buffer stream is a set of per‑frame geometry‑and‑material maps (albedo, depth, metallic, normal, roughness) that the game engine emits in lockstep with the RGB video.

Encode the albedo map → latent vector $a_1\in\mathbb{R}^4$; repeat for depth $d_1$, metallic $m_1$, normal $n_1$, roughness $r_1$.

Stack the five vectors vertically to obtain $g_1 = [a_1; d_1; m_1; n_1; r_1]\in\mathbb{R}^{5\times4}$.

Take the noisy RGB latent $x_1\in\mathbb{R}^{1\times4}$ (single channel for the toy) and concatenate: $[x_1, g_1]\in\mathbb{R}^{6\times4}$.

Apply the linear patch projection (e.g., $W\in\mathbb{R}^{8\times6}$) to obtain $h_1 = W\,[x_1, g_1]$, an 8‑dimensional token ready for the transformer.

Concatenation lets the diffusion model attend jointly to appearance (RGB) and scene geometry, so the generated frames inherit correct depth and material cues without extra training.

How does G‑buffer conditioning differ from ordinary text‑only cross‑attention?

Text‑only conditioning injects a prompt vector that influences every token uniformly. G‑buffer conditioning injects per‑frame geometry tensors that are spatially aligned with the RGB latents, allowing the model to resolve occlusions and lighting consistently across time.

The pipeline streams latent video chunks, compresses distant history, runs a four‑step diffusion model, and decodes with ultra‑lightweight codecs, all while staying synchronized with a live G‑buffer feed.

Divide the latent video into fixed‑length chunks (4 frames each).

After generating a chunk, store it in the highest‑fidelity tier.

When a tier exceeds its capacity, compress its oldest chunk into the next coarser tier (e.g., apply a learned down‑sampling encoder).

Repeat compression up to the third tier, which holds the oldest history at the coarsest resolution.

During denoising of the current chunk, insert all retained history tokens (full, medium, coarse) into the self‑attention context together with the per‑chunk G‑buffer conditioning.

Prepend the first generated latent frame as a global appearance anchor to the highest‑fidelity tier.

Generate C₁ → store in Tier 1.

Generate C₂ → Tier 1 now contains C₂; C₁ is moved to Tier 2 after applying a lightweight encoder.

Generate C₃ → Tier 1 holds C₃; C₂ is moved to Tier 2; C₁ is further compressed and moved to Tier 3.

When denoising C₄, self‑attention receives tokens from Tier 1 (C₃), Tier 2 (compressed C₂), and Tier 3 (coarsely compressed C₁) plus the global anchor C₁.

The hierarchy lets the model attend to long‑range scene context without paying the full cost of storing every past frame at full resolution.

We shrink a 50‑step diffusion schedule to four steps by first teaching the model to mimic classifier‑free guidance with a single network pass, then gradually shortening the schedule, and finally refining the four‑step dynamics with flow‑based matching.

Why not simply train a four‑step diffusion model from scratch?

Training from scratch forces the network to learn a long‑range trajectory in a single jump, which is unstable and yields severe artifacts. The staged distillation supplies a warm‑start at each intermediate schedule, ensuring the model can approximate the teacher’s dynamics at the target budget.

Instead of five separate VAE passes for the G‑buffer channels, a single shared encoder predicts all five latent streams; similarly, a compact decoder replaces the full‑size VAE decoder, cutting latency dramatically.

**Figure 2** System overview of AlayaRenderer-Flash. During gameplay, the player independently controls game interaction through standard game inputs and visual appearance through text prompts. The game engine continuously updates the world state and exports synchronized G-buffers, which, together with the text prompt, are processed by AlayaRenderer-Flash to generate stylized RGB frames in real time. The rendered frames are immediately displayed to the player, forming a closed-loop interactive rendering system.

Receive the next G‑buffer frame batch from the engine.

Encode the batch with the shared tiny encoder to obtain $g_k$.

Run the four‑step diffusion transformer on $[x_k, g_k]$ to produce the denoised RGB latent.

Decode the latent with the tiny decoder into an RGB frame.

Display the frame and feed the generated latent into the history hierarchy for the next chunk.

Experimental Setup

AlayaRenderer‑Flash achieves real‑time 31.54 FPS throughput.

AlayaRenderer‑Flash reaches $31.54$ FPS, enabling real‑time playback.

Table 1 reports $31.54$ FPS for the Flash configuration, far surpassing the $0.56$ FPS baseline.

The experiments use the Black Myth: Wukong dataset captured at $1280\times720$ and $30\text{ FPS}$, providing an RGB frame plus five aligned G‑buffer channels (albedo, depth, normal, roughness, metallic). Evaluation follows five complementary metrics: SCLIP‑I for semantic content preservation, tLPIPSwarp for temporal flicker, Boundary MSE/SSIM for long‑horizon consistency, and MCLIP for prompt controllability.

Table 1 isolates each design stage: autoregressive streaming improves SCLIP‑I (0.846 vs. 0.836) but raises tLPIPSwarp; 4‑step distillation restores temporal stability and lifts throughput to $6.30$ FPS; finally, lightweight codecs deliver the reported $31.54$ FPS and $16.2$ GB memory while preserving or slightly improving all quality metrics.

Performance and Capability Comparison

AlayaRenderer‑Flash outperforms baselines in quality, speed, and streaming capabilities.

Recall that AlayaRenderer‑Flash preserves scene structure via G‑buffer conditioning; it runs at 31.54 FPS, enabling real‑time interactive streaming while baselines lag far behind.

Table 3 reports 31.54 FPS for AlayaRenderer‑Flash versus 1.30 FPS for RGB↔X and 0.31 FPS for FrameDiffuser.

**Figure 3.** Qualitative comparison with external baselines on 5-second clips. Reference denotes the original RGB frames rendered by the game engine during data collection. Since prompt-conditioned rendering admits multiple valid outputs, these frames serve as visual references rather than ground-truth targets.

These methods generate video frames from game‑engine G‑buffer streams, but vary in how they handle time and prompt control.

How does AlayaRenderer‑Flash differ from the other G‑buffer baselines?

AlayaRenderer‑Flash combines causal streaming with prompt‑controlled generation and runs at real‑time speeds, whereas RGB↔X lacks temporal modeling, FrameDiffuser cannot switch prompts, and DiffusionRenderer requires offline windowed processing.

AlayaRenderer‑Flash maintains superior structural fidelity while delivering real‑time streaming.

Prompt Switching and Long Rollouts

Evaluating prompt switching over long streaming rollouts shows seamless style transitions.

AlayaRenderer‑Flash supports seamless prompt updates during continuous streaming inference, unlike offline renderers that must restart after each change. We evaluate this capability on long rollouts of 637 frames, switching prompts every five chunks across eight distinct styles.

**Figure 4.** Prompt-controlled visual style transitions during a single long streaming rollout. Reference denotes the original RGB frames rendered by the game engine during data collection and is shown for visual comparison.

Across the entire sequence AlayaRenderer‑Flash consistently tracks each prompt change, preserving scene structure and camera path without visible ghosting or boundary artifacts. This demonstrates robust, continuous prompt‑controlled rendering over long streaming rollouts.

Interactive Deployment

AlayaRenderer‑Flash runs at real‑time speed in a live game demo.

AlayaRenderer‑Flash meets the 30 FPS real‑time target on a single NVIDIA H200 GPU.

Runs at 31.54 FPS, exceeding the 30 FPS requirement while preserving gameplay logic.

**Figure 1** AlayaRenderer-Flash restyling a live *SuperTuxKart* session in our interactive web demo. The game engine continuously exports synchronized G-buffers to AlayaRenderer-Flash, which generates stylized RGB frames at playback rate. Players can modify the visual style of the rendered game through arbitrary text prompts while the underlying gameplay logic remains unchanged.

AlayaRenderer‑Flash demonstrates real‑time feasibility for interactive game environments.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers