Multiplayer Interactive World Models with Representation Autoencoders

Anthony Hu, Václav Volhejn, Adrien Ramanana Rahary, Chris Mulder, Aditya Makkar, Alyx Liao, Amélie Royer, Manu Orsini, Adam Jelley, Eloi Alonso, Florian Laurent, Fredrik Norén, James Swingos, Jan Hünermann, Kent Rollins, Lucas Hosseini, Matthieu Le Cauchois, Maxim Peter, Pim de Witte, Tim Brown, Vincent Micheli, Moritz Böhle, Gabriel de Marmiesse, Viktoriia Sharmanska, Lucia Specia, Michael Black, Patrick Pérez

A 5-billion-parameter latent diffusion model that simulates four-player Rocket League matches in real time.

How can we build a world model that simulates complex, multi-agent physical interactions by operating in a learned latent space rather than raw pixels?

Most world models treat multi-agent environments as single-player problems, failing to capture how multiple agents jointly reshape a shared scene. This leads to models that ignore player actions or entangle their effects, making them unsuitable for interactive planning or control. MIRA conditions a latent diffusion model on the simultaneous action streams of four players, tiling their views into a single grid to learn consistent, multi-agent dynamics. It uses a representation autoencoder built on a frozen DINOv3 feature extractor to compress video into a compact latent space, enabling real-time inference. The model generates stable, four-player matches at 20 frames per second on a single Nvidia B200 GPU, with rollouts remaining coherent for over five minutes.

Paper Primer

The core mechanism is a representation autoencoder that maps raw video into a latent space, where a flow-matching transformer predicts future states. By tiling four player views into a single grid and conditioning on all action streams simultaneously, the model learns to attribute scene changes to the correct agent.

Multiplayer conditioning significantly improves predictive accuracy over single-player baselines.

The model reduces uncertainty about future states by leveraging multiple views and action streams, with warm-starting from a single-player model yielding better performance than training on multiplayer data from scratch. The model maintains distributional quality for over five minutes of rollout, with practical stability observed for hours.

Why is a representation autoencoder used instead of predicting directly in pixel space?

Predicting in pixel space is computationally expensive and wastes capacity on high-frequency textures that are irrelevant to physical dynamics. The autoencoder compresses the video into a compact latent space, which keeps rollouts stable and enables real-time generation.

How does the model handle the "agency" problem where multiple players act at once?

The model uses tiled views and per-player action embeddings that are broadcast to every latent position. Because the model is trained with action dropout—where some players' actions are randomly withheld—it learns to infer which actions affect a specific view and which do not.

This work demonstrates that world models can scale to complex, multi-agent physical environments by shifting from single-agent perspectives to joint, action-conditioned latent prediction.

Introduction

We expose why pixel‑space models falter in multiplayer dynamics and motivate a latent‑space solution.

Pixel‑space world models tie visual reconstruction to dynamics, so visual errors corrupt physical predictions; this makes them brittle in multiplayer settings. Pixel‑space prediction is like trying to forecast a soccer match by looking at a single photograph — you see the current layout but have no sense of momentum or player intent. Shifting to a latent representation decouples rendering from physics, enabling consistent multi‑agent rollouts.

Pixel‑space models entangle appearance and physics, so any visual artifact can break causal consistency; a Representation Autoencoder learns a compact latent where dynamics can be modeled independently of rendering.

**Figure 1.** World model imagination. Rows are the four players' viewpoints (Players 1–4) and columns show three timesteps in the trajectory ($t_0 arrow t_0 + 6\text{s} arrow t_0 + 8\text{s}$). From the initial state at $t_0$ and the players' action streams, the model imagines a dynamic game scene that captures the physical interplay between the players, the ball, and the scene. The rollout is temporally consistent: the in-game clock counts down with elapsed time (4:41 $arrow$ 4:35 $arrow$ 4:33), and the players' views stay mutually coherent. For instance, Players 2 and 4 at $t_0 + 6\text{s}$, who are contesting the ball near the goal, each render the ball and the cars in a matching spatial configuration. This is best experienced interactively: try our live demo at mira-wm.com.

**Figure 21** Multiplayer rollout with game-state probe. A rollout from the multiplayer world model; the game-state probe (Section 6.2) reads ball and car positions and velocities from the model's pre-head activations.

The core insight is that moving prediction from raw pixels to a learned latent space decouples visual fidelity from dynamics, enabling stable, physically consistent multiplayer world modeling.

The MIRA Architecture

The method encodes player views into a compact latent space and predicts future latents conditioned on actions.

Pixel‑space models quickly lose physical consistency when multiple agents interact, because visual dynamics entangle each player’s influence.

The RAE compresses each view into a low‑dimensional latent $z$, letting the world model operate on a clean, dynamics‑focused space while the decoder later restores full‑resolution video.

Each 4 × 4 × 3 frame is first passed through a frozen feature extractor, yielding a 2 × 2 × 3 feature map.

The linear bottleneck projects the 2 × 2 × 3 map to a 1 × $C$ vector; with $C=2$ we obtain $z_1 = [0.12, -0.07]$ for frame 1.

Repeating for frame 2 gives $z_2 = [0.15, -0.04]$.

The two latents are stacked to form the initial context window $[z_1, z_2]$.

This tiny example shows how spatial‑temporal down‑sampling collapses a 4 × 4 × 3 image into a 2‑dimensional code, dramatically reducing the amount of data the world model must predict.

How does this Representation Autoencoder differ from a conventional autoencoder?

Standard autoencoders learn both encoder and decoder jointly for reconstruction. Here the encoder is pretrained and frozen; only the latent dynamics are learned via diffusion. The decoder is used solely at inference to render predicted latents, so the latent space is optimized for dynamics rather than pure reconstruction quality.

Encode each player’s view with the Codec Encoder, producing per‑player latents.

During the initial warmup, feed a short real clip into the encoder to seed a fixed‑length context window.

Collect the current action streams from all four players and concatenate them.

Condition the Latent World Model on the concatenated actions and the context window; predict the next latent frame at 10 Hz.

Decode the predicted latent with the Codec Decoder, generating two future frames (50 ms apart) for each player at 20 fps.

Append the new latent to the context window and drop the oldest entry, preserving a constant window size for the next rollout step.

Roll‑forward loop that maintains the fixed‑size context window.

**Figure 2. Method overview.** MIRA simulates four-player Rocket League in the latent space of a video representation codec. During an initial context warmup, the codec encoder maps a short clip of each player's view into a stack of latent frames that seed a fixed-length context window (Section 4.2). Conditioned on the per-player action streams (Section 4.5) and the latents currently in the context window, the latent world model predicts the next latent frame causally at 10 Hz (Sections 4.3 and 4.4). Few-step diffusion distillation, a streaming KV-cache, and a rolling context window keep inference fast enough to generate 20 frames per second on a single Nvidia B200 GPU (Section 5). The codec decoder then renders each predicted latent back into the four player views at 20 fps (Section 4.2). Appending the predicted latent to the context window and dropping the oldest (roll forward) autoregressively extends the rollout over long horizons (Section 5).

**Figure 3** Codec. A frozen pretrained feature extractor (DINOv3-L) extracts per-frame features, mixed across several intermediate layers, and a learned linear bottleneck downsamples them by $2 \times 2$ in space and $2 \times$ in time and projects to a latent $z$ with $C$ channels. The decoder reverses these steps: a spatial upsampling restores the spatial resolution, a causal spatio-temporal vision transformer reconstructs the video, and a temporal upsampling restores the frame rate.

We gathered 10 000 hours of multiplayer Rocket League play from RL policies, pairing each video stream with per‑player actions and privileged game state. Training a 5 B‑parameter diffusion world model on this data yields 20 fps inference on a single Nvidia B200 GPU.

Related Work

Survey of world‑model approaches and design choices relevant to multiplayer video prediction.

World‑model research splits into two families: latent dynamics models that never render pixels, and generative simulators that output video frames. This paper belongs to the latter, tackling the hardest regime—real‑time multiplayer games where several agents simultaneously reshape a shared environment.

Section 2.1 surveys latent‑space models used for control, from early recurrent Dreamer variants to recent transformer‑based world models and joint‑embedding predictive architectures (JEPA). These methods prioritize compact state representations and accurate dynamics, often discarding pixel reconstruction entirely.

Section 2.2 reviews generative world models that synthesize observations. Early works generate static novel views or camera‑controlled videos; later systems produce fully interactive game worlds, conditioning on a single player’s actions for titles ranging from DOOM to Minecraft.

Section 2.3 focuses on multiplayer extensions, where multiple agents jointly reshape a shared scene. Prior approaches either treat other players as part of the environment or condition on several agents in grid‑world or voxel settings; none address the continuous‑physics, high‑speed dynamics of a sport like Rocket League.

Section 2.4 describes the visual backbone: latent video diffusion built on transformers. Diffusion and flow‑matching techniques provide high‑fidelity video synthesis, but by default lack causality and action conditioning—properties required for a usable world model.

Section 2.5 examines representation choices. Predicting in a compact latent (often distilled from a self‑supervised encoder such as DINO) reduces dimensionality, stabilizes long rollouts, and shifts the quality burden onto the latent space rather than raw pixels.

Section 2.6 tackles long‑horizon stability. Exposure bias between training (clean frames) and inference (self‑generated frames) is mitigated by techniques like diffusion forcing, history guidance, and few‑step consistency objectives, enabling real‑time rollout at 10 Hz latent prediction.

Section 2.7 surveys evaluation metrics beyond per‑frame fidelity, including gFID, Action‑Adherence Reward (ARR), and probing of physical consistency. These metrics capture temporal drift, action recoverability, and alignment with underlying game physics.

Models that predict raw video frames directly, treating each pixel as part of the state to be forecast.

The table describes the data streams used, their contents, and their respective sample rates. It includes three rows: Video (720p, 30 fps), Game state (Ball + per-car physics, scores, events, 120 Hz), and Actions (Nexto output: 3-valued axes + buttons, 15 Hz).

World Model Dynamics

We describe the latent world model and its training pipeline that enable real‑time multiplayer prediction.

Predicting directly in pixel space wastes capacity on high‑frequency detail and cannot run at real‑time rates, so we shift prediction to a compact latent space.

The model predicts the joint future latent of all players from their past latents and actions, treating the four viewpoints as a single tiled grid.

How does this differ from modeling each player separately?

By tiling the latents into one grid the transformer sees all views simultaneously, allowing it to enforce a shared physical state across perspectives, whereas separate models would have no guarantee of consistency.

During training each latent frame is blended with Gaussian noise at a random flow time, so the model learns to denoise from any noise level.

Compute the interpolation weight (1‑$\tau$)=0.5.

Form the noised latent: z_$\tau$ = 0.5·z₀ + 0.5·z₁ = 0.5·0 + 0.5·2 = 1.

The model must now predict a velocity that moves z_$\tau$=1 back toward the clean target z₁=2.

This mixed latent contains both clean signal and noise, forcing the model to learn a mapping that works for any proportion of corruption.

Why not use teacher forcing where the model always sees clean context?

Teacher forcing conditions on perfect past latents, which never occur at rollout; diffusion forcing exposes the model to its own noisy predictions, reducing compounding errors.

We train the model to predict the average velocity over a larger time step, allowing a single evaluation to replace multiple small integration steps.

How does the self‑consistency term ensure the model remains accurate with fewer steps?

It forces the large‑step prediction to match the average of two sequential half‑step predictions, so the learned velocity field stays consistent when the integration granularity changes.

A learned conditioning vector that combines action embeddings and flow‑time embedding modulates the layer‑norm parameters of every transformer block, injecting player actions and temporal context into the model.

Why embed actions and flow time separately before summing?

Separate embeddings let the model learn distinct representations for the discrete action space and the continuous time variable; summing them yields a single conditioning vector that can be efficiently applied via AdaLN to all tokens.

Embed each latent vector $z$ with a linear layer to obtain token embeddings.

Form the conditioning vector $c$ by summing the action embedding and the flow‑time embedding.

Apply space‑time attention: first a bidirectional spatial attention within each frame, then a causal temporal attention across frames at each spatial location.

Modulate the layer‑norm of the feed‑forward sub‑layer with AdaLN using $c$ (compute $\gamma,\beta$ and apply $(1+\gamma)\text{LN}(x)+\beta$).

Predict the velocity $v_{\theta}(z_{\tau},\tau)$ for the current noised latent.

Update the latent by stepping along the predicted velocity (e.g., $z_{t+1}=z_{t} + v_{\theta}\Delta$).

Repeat for the desired rollout horizon.

**Figure 4.** World model training and architecture. Top: the training pipeline, read left to right. Each of the $P$ players' frames is mapped by a shared, frozen codec encoder (Section 4.2) to a per-player latent $z_1^P$, and the $P$ latents are concatenated into a single clean joint latent $z_1$. Together with Gaussian noise $z_0 \sim \mathcal{N}(0, \text{I})$ of the same shape, diffusion forcing (Section 4.3) forms the noised latent $z_\tau = (1-\tau)z_0 + \tau z_1$, drawing an independent flow time $\tau \in [0, 1]$ for each frame so that every frame is at once clean context and a target to denoise. The trained latent world model, conditioned on all players' actions $a^{1:P}$ and the flow time $\tau$, predicts the flow velocity $v_\theta$, supervised against $z_1 - z_0$ by the flow-matching loss $\|v_\theta - (z_1 - z_0)\|_2^2$ (Section 4.3). Bottom: one of the $N$ stacked AdaLN space-time transformer blocks inside the world model. The noised latent $z_\tau$ passes through space attention (bidirectional, within a frame), time attention (causal, across frames), and an AdaLN-modulated SwiGLU feed-forward layer. Conditioning enters once per block: the action embedding $\text{emb}(a^{1:P})$ (a per-player embedding of the keyboard controls, concatenated over players) and the flow-time embedding $\text{emb}(\tau)$ (a sinusoidal encoding followed by an MLP) are summed into a single vector $c$, from which an MLP predicts the scale $\gamma$ and shift $\beta$ that modulate the feed-forward layer normalization, $(1+\gamma)\text{LN}(x) + \beta$; the same $c$ is broadcast over every spatial position. Here $p$ indexes the $P$ players ($P=4$ in our 2v2 setting).

**Figure 5.** World model imagination of a demolition event. Rows are the four players' viewpoints (Players 1–4) and columns show three timesteps ($t_0 arrow t_1 arrow t_2$) of an imagined rollout in the desert arena. From the initial state at $t_0$ and the players' action streams, the model imagines a fast, sub-second collision: a blue car boosts into an opponent and demolishes it. The event stays mutually coherent across viewpoints. At $t_2$, Player 1 renders the “DEMOLITION” callout for its successful hit, while Player 3 (the demolished car) sees the “BOOM!” explosion fill its own view, and Player 2 renders the very same blast as a distant burst of smoke on the pitch. The in-game clock (4:12) is consistent across all players and barely advances, reflecting how brief the event is.

**Figure 6.** World model imagination of a goal. Rows are the four players' viewpoints (Players 1–4) and columns show three timesteps ($t_0 arrow t_1 arrow t_2$) of an imagined rollout in the temple arena. From the initial state at $t_0$ and the players' action streams, the model imagines the ball being driven into the net for a goal. The event stays mutually coherent across viewpoints: at $t_1$ the “SCORED!” callout and the blue goal-explosion appear simultaneously in all four players' views, each rendering the same blast from its own camera, and by $t_2$ every player sees the aftermath on the pitch. The rollout is also temporally consistent: the scoreline updates from 2–1 to 3–1 for the blue team as the goal registers.

Empirical Results

Key quantitative results show latent world modeling vastly outperforms pixel‑space baselines.

Latent world modeling attains gFID 10.7, an order‑of‑magnitude improvement over pixel‑space baselines (104.9 and 81.0).

Table 2 compares the latent pipeline against two pixel‑space models trained with equal total steps.

ARR quantifies how faithfully a generated rollout reproduces the actions it was conditioned on, by comparing action detection on the generation to a ceiling set by the codec reconstruction.

How does ARR differ from a plain action‑detection accuracy metric?

A raw detection accuracy would conflate failures of the visual encoder with failures of the world model. ARR first measures the best achievable detection on the codec reconstruction (the ceiling) and then reports the ratio of generation to that ceiling, thereby isolating the world model’s controllability.

**Figure 23** Pixel-space rollouts drift. A rollout from the plain pixel-space world model, shown left to right. The scene stays coherent for the first frames, then rapidly degrades into warped, unstructured texture, illustrating why pixel-space generation trails the latent baseline (Table 2).

**Figure 27.** Human evaluation: quality. Pairwise Elo (Bayesian, 95% CI) from the quality study of Section 6.2, on accuracy and on visual quality, across a subset of comparisons.

**Figure 9. Objective and drift.** gFID across a 5-minute rollout (log time) on the baseline codec. Past the 4 s training window (dashed), teacher forcing degrades sharply and stays about 10x higher, while diffusion forcing stays low and flat.

**Figure 10. Context noise at inference.** gFID as a function of the past-noise standard deviation (vertical) and rollout time (horizontal), for three representative codecs; cell values are gFID and color runs green (low) to red (high, clipped at 30). The distilled codec's quality collapses over the rollout when the past is fed clean (noise 0) and is rescued by a small amount of noise, while the pretrained and from-scratch codecs are flat in noise. The other pretrained feature extractors and the gFVD/gFDD metrics show the same pattern (Section H).

Diffusion forcing reduces generation gFID to 10.7, a three‑fold improvement over teacher forcing’s 32.5.

Table 8 reports gFID, gFVD, and gFDD for both training objectives on the baseline codec.

Scaling and Data Efficiency

Scaling analysis quantifies how data, model size, and training regime affect multiplayer world‑model performance.

Diffusion forcing keeps generative video quality stable over long rollouts, while teacher forcing degrades to roughly ten‑times higher gFID.

gFID remains flat for diffusion forcing across horizons up to 12 s, whereas teacher forcing rises sharply.

Larger models consistently reduce ball‑position readout error, dropping from 2130 units at 100 M parameters to 1448 units at 5 B.

Linear probe error measured on frozen features shows monotonic decline across five model sizes.

**Figure 11.** Few-step sampling. Generation quality against the number of flow-matching steps, for the baseline and the PSD self-distilled model (trained from scratch).

**Figure 12** ARR tracks human judgment of controllability. Automatic ARR against the human action-adherence preference (an Elo delta, predicted vs. context) for the models rated in both, spanning pixel-space, feature-extractor, and training-checkpoint comparisons. The two are strongly correlated (Pearson $r = 0.84$) and rank the models almost identically (Spearman $\rho = 0.93$); the line is a linear least-squares fit.

**Figure 13. Controllability.** (a) Per-action ARR (faint) and their aggregate (bold black) against training step for the single-player model. (b) Per-action ARR at the final checkpoint against training frequency, with bootstrap 95% confidence intervals and a log-linear fit (r = 0.82). Colors identify the nine controls across both panels (shared legend); the dashed line marks a faithful reconstruction (ARR = 1).

**Figure 15.** Data scaling at fixed compute. (a) gFID and (b) controllability ARR of the single-player model against the amount of unique training data seen, at a fixed 100k-step budget (smaller subsets are revisited more, so unique data and repetition vary together). Below about 50 hours both metrics collapse as distinct data runs short. Above it gFID saturates while ARR keeps climbing, so more unique data buys action fidelity that gFID no longer registers.

**Figure 16** Larger models improve generation quality and encode physical state more faithfully. (a) Ball-position readout error of the linear game-state probe (L2, in Unreal units) against model size, and (b) validation FVD throughout training, for five multiplayer world models spanning 100M to 5B parameters (light to dark), with the data and training recipe held fixed. Lower is better on both axes; the FVD y-axis is logarithmic, cropped to the converged regime. Larger models converge faster and to lower error, their ordering by size holds for almost the entire run, and the probe error decreases monotonically with model size. Returns diminish beyond 2.5B, where the two largest models reach comparable quality.

**Figure 17** The multiplayer model attends across views to shared objects and state. Cross-view spatial attention on a rollout generated by the multiplayer world model. Each column is one player's generated view. After the generated views, each row queries the token named on the left and shows where its attention lands in the other views, warm colors marking strong attention and cool colors weak. Player 1's car, queried through Player 2's view, is picked out in the other views that see it and stays cooler where it is off-screen. The ball, queried through Player 1's view, activates in every view. The timer, queried through Player 1's view, attends to the shared top-center HUD band of every view. Cross-view attention is spatially structured; averaged over layers 4–7.

**Figure 18. Humans play unlike the training policies.** Distribution of actions per minute across clips for the nine controls (forward, backward, left, right, jump, boost, powerslide, air-roll left and right). Both curves come from recorded gameplay, and neither is produced by the world model: the automated policies behind the training data (bots) act fast and consistently, in a narrow band near 390 actions per minute, while humans act more slowly and far more variably, from near-idle to bursts and spread broadly below 350. The human curve comes from a separate corpus of publicly shared human gameplay clips on the Medal.tv platform (Medal.tv, 2024) (roughly 5,000 hours). The plot documents the distribution shift the model must handle; in our live demo it stays coherent under human control.

**Figure 19** The clock drifts from real time. A generated rollout (single view), sampled once per second, with the score-and-clock HUD boxed in red and magnified at the lower left of each frame. Over five seconds of real rollout the clock should count down five seconds, yet it barely moves, reading 4:54, 4:53, 4:53, 4:52, 4:54, 4:53: it advances far too slowly and even ticks back up, one instance of the clock losing track.

**Figure 20** A car merges into the ball in a single-player world model. A rollout from a 1B single-player model, shown left to right. A non-playable car (blue) sits beside the ball and is drawn into it as the two overlap; once the ball moves on, the car does not re-emerge as a distinct object. We do not observe this failure in the multiplayer model.

Ablation Studies

We quantify how each design choice affects reconstruction and generation quality.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers