GameWAM: A World Action Model for Video Games

Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Weijia Li

GameWAM unifies visual world modeling and native keyboard-mouse control for closed-loop gameplay.

How can we build a unified world model that jointly predicts future visual observations and native keyboard-mouse actions to enable closed-loop gameplay?

Existing game agents map visual input to actions without modeling how the world evolves, while world models predict visual futures without serving as task policies. This disconnect prevents agents from learning the fine-grained temporal and metric control required for native keyboard-mouse interaction. GameWAM bridges this gap by jointly generating future visual observations and executable action trajectories using parallel flow-matching branches. It decouples long-horizon planning from short-horizon execution, allowing the model to predict far ahead while maintaining frequent closed-loop feedback. On the Minecraft Universe benchmark, GameWAM achieves state-of-the-art success rates while requiring fewer executed native actions than previous agents.

Paper Primer

The core mechanism is a block-causal World-Action Model (WAM) that uses parallel Video and Action Diffusion Transformers (DiTs) to predict future states and controls. It functions like a dual-track navigator: one branch simulates the visual consequences of potential moves, while the other selects the specific keyboard-mouse inputs to achieve the task, both conditioned on a shared history of realized observations.

GameWAM improves interaction efficiency in complex, long-horizon game tasks.

Across the Minecraft Universe (MCU) benchmark, the model achieves higher average success rates while consistently reducing the number of native actions required per successful episode compared to baseline agents. State-of-the-art performance on MCU Mini and full task sets.

The authors identify a failure mode called Low-Frequency Action Source Imprinting (LASI), where low-frequency components of the sampled action source can induce persistent, pathological camera bias. They mitigate this by resampling the action source at each replanning step rather than reusing it across the entire rollout.

Why is a unified World-Action Model necessary for video games compared to standard behavioral cloning?

Standard agents often use compressed or abstract action representations that weaken the fine temporal structure needed for concurrent key control and continuous camera motion. By explicitly modeling visual dynamics, GameWAM provides dynamics-aware supervision that improves control precision.

How does the model handle the difference between gameplay and GUI interaction?

GameWAM uses a timestep-wise router that selects between gameplay- and GUI-specific prediction branches, applying separate normalization statistics to the shared keyboard-mouse action space to account for their distinct conditional distributions.

GameWAM demonstrates that joint world-action modeling is viable for native, closed-loop game control, provided the architecture accounts for heterogeneous interaction modes and the specific risks of source-dependent generative bias.

Introduction and Motivation

We frame the gap: current agents miss joint visual‑action dynamics, prompting GameWAM.

Modern video games require fast visual perception, persistent world state, and native keyboard‑mouse control. Existing agents either react to observations without predicting future dynamics, or predict visual futures from supplied actions but cannot serve as task policies. The World‑Action Model (WAM) unifies these objectives, yet it has not been explored under the rapid, open‑ended interaction of games.

Game agents need a model that simultaneously predicts how the visual world will evolve and decides which native actions to take, because neither pure reaction nor pure prediction suffices for long‑horizon gameplay.

**Figure 1.** **Overview of GameWAM.** GameWAM jointly models future visual observations and native actions for closed-loop gameplay and GUI control.

The key shift is from reactive policies to world‑action models that predict and act jointly.

Related Work

We situate GameWAM among prior game agents, world models, and world‑action approaches.

Game agents have leveraged video games as scalable testbeds for native control and long‑horizon interaction. Early work introduced large‑scale environments, demonstrations, and open‑ended evaluation.

Later agents emphasized multimodal reasoning, memory, hierarchical control, and generalist pre‑training. Recent systems such as OpenHA, Game‑TARS, and Lumine target native keyboard‑mouse action spaces at scale.

World‑Action Models unify visual prediction with executable action generation.

Recent generative WAMs like DreamZero, Fast‑WAM, and FFDC‑WAM couple video forecasting with closed‑loop control, motivating the GameWAM approach.

VPT learns generic video representations by predicting masked future frames from large unlabeled video corpora, providing a foundation for downstream visual prediction tasks.

The GameWAM Architecture

We describe GameWAM’s block‑causal world‑action architecture and its hierarchical memory system.

Long‑horizon gameplay demands that an agent keep future predictions aligned with the actions it will actually execute. Naïve training would expose the model to future visual frames that never materialize, wasting capacity and corrupting the learned world‑action dynamics. Block‑causal visibility solves this by restricting each planning block to only the clean context that truly stems from executed history.

Think of reading a story where you can only see the pages you have already turned; the current page is visible, but any future pages remain hidden until you actually turn to them.

Block 0: $P_{c,0}=H_c\cup C_{c,0}$ contains the initial two frames; $A_0$ is generated and executed.

Block 1: $P_{c,1}=H_c\cup C_{c,1}$ now includes the frame produced by $A_0$, but $A_1$ remains masked.

Block 2: $P_{c,2}=H_c\cup C_{c,2}$ adds the frame from $A_1$ (once executed) while $A_2$ stays hidden.

Only the actions that have actually been executed become part of the clean prefix; predicted but unexecuted actions never influence the context, preserving causality.

How does Block‑Causal Visibility differ from standard causal masking used in language models?

Standard causal masking hides all future tokens, regardless of modality, but it still allows a predicted token to attend to its own noisy version. Block‑Causal Visibility additionally separates clean and noisy variables: the clean prefix $P_{c,k}$ is shared, while each modality’s noisy variable ($V_j$ or $A_j$) is masked from the other, preventing cross‑modal leakage of unexecuted predictions.

WAM treats future video frames and native keyboard‑mouse actions as two sides of the same joint distribution, letting the agent plan by sampling coherent world‑action trajectories.

Imagine a planner that proposes a long sequence of actions but only commits to the first few; after execution the planner revises the remaining tail based on the new observation, while a separate memory keeps track of what has already happened.

Training Objective

The section defines the multi‑loss training objective that aligns visual prediction, action prediction, mode routing, and history consistency.

Jointly training visual prediction and native‑action prediction is tricky because each branch can dominate the gradient, and the compressed history must stay informative for long‑horizon planning. The authors therefore construct a multi‑loss objective that balances these signals while respecting block‑causal visibility.

The loss combines four terms—video flow, action, mode routing, and history consistency—each supervising a distinct prediction target, and the weighted sum forces the model to learn all of them together.

Video flow loss: $L_v = \|[1.0,0.5]-[0.8,0.6]\|_2^{2}= (0.2)^2+( -0.1)^2 = 0.05$.

Continuous action loss: $L_{\text{cont}}^{a}= (0.2-0.2)^2 = 0$.

Discrete action loss: $L_{\text{disc}}^{a}= (1-1)^2 = 0$.

Combined action loss: $L_a = \lambda_{\text{cont}} \cdot 0 + \lambda_{\text{disc}} \cdot 0 = 0$ (using $\lambda_{\text{cont}}=\lambda_{\text{disc}}=1$).

Mode loss: with a valid label ($\nu=1$) and perfect prediction ($\rho=1$, $r^{*}=1$), $L_{\text{mode}} = \text{BCE}(1,1)=0$.

History loss: assume $F(\text{Pool}(H_c))=[0.9,0.4]$, $s_g(q_c)=[1.0,0.5]$, cosine similarity $=0.998$, squared error $=0.02$, so $L_{\text{hist}}\approx0.011$.

Total loss with all $\lambda$ set to 1: $L = 0.05 + 0 + 0 + 0.011 = 0.061$.

Even with perfect action and mode predictions, the visual and history terms dominate the gradient, illustrating why separate weighting and masking are essential to keep all branches learning.

Why does the objective reduce continuous and discrete action coordinates separately instead of a single joint loss?

Continuous coordinates typically have larger numeric ranges than discrete one‑hot vectors; if summed together, the continuous term would dwarf the discrete term, preventing the model from learning accurate categorical actions. By computing $L_{\text{cont}}^{a}$ and $L_{\text{disc}}^{a}$ separately and scaling them with $\lambda_{\text{cont}}$ and $\lambda_{\text{disc}}$, each branch receives a balanced gradient signal.

Empirical Evaluation

GameWAM’s results on MCU and ViZDoom benchmarks demonstrate superior efficiency and success.

GameWAM attains the highest average success rates while using fewer steps per successful episode on the MCU benchmark.

Table 1 shows GameWAM outperforms all baselines on ASR Mini, ASR All, and steps across Embodied, GUI, and Combat categories.

**Figure 4.** Evaluation results in FPS ViZDoom benchmark across four maps. The reported average reward for each map is calculated over 50 episodes.

Ablation and Latent Analysis

We quantify each component’s impact by ablating it and measuring performance on MCU Mini.

We assess each design choice by removing it and reporting the resulting drop in the four MCU Mini metrics.

Removing future‑video supervision reduces the average score by 30 points.

Full GameWAM average 64.0 vs Action‑only supervision average 34.0.

Beyond the ablations, we investigate how low‑frequency components of the sampled action source influence generated camera motion.

LASI captures the systematic effect of low‑frequency source coefficients on coarse camera motion while keeping visual, historical, and proprioceptive context fixed.

Replacing source modes 0–2 makes the yaw DCT₀ output follow the donor in $94.8\%$ of trials.

Intervention study reported $94.8\%$ trial alignment.

Zeroing the low‑frequency band removes $99.25\%$ of the yaw output variance.

Variance reduction measured after zeroing modes 0–2.

Limitations and Qualitative Analysis

We discuss the limits of GameWAM’s closed‑loop control and its cross‑game transfer performance.

The rollout visualizations (Figures 9‑18) expose interaction patterns that cannot be captured by aggregate success metrics alone. They show how GameWAM maintains a mining action on an initially hidden object, pursues moving entities, manipulates structured GUIs, and recovers from intermediate errors.

**Figure 9.** Closed-loop rollout for “Mine the white bed.” The white bed is not visible in the initial observation. GameWAM explores the nearby environment, enters a structure, acquires visual access to the target, and completes the mining interaction. The sequence illustrates visually guided local exploration and target acquisition under closed-loop interaction.

**Figure 10.** Closed-loop rollout for “Collect obsidian for crafting.” Obtaining obsidian requires a sustained mining interaction rather than a brief action. After aligning with the target block, Game-WAM maintains the mining behavior across successive replanning steps until the block breaks, illustrating temporal consistency of low-level control.

**Figure 11.** Closed-loop rollout for “Hunt a horse.” The horse requires repeated successful attacks before task completion and changes its relative position throughout the interaction. GameWAM repeatedly reacquires the target, adjusts movement and viewpoint, and continues the interaction until completion. The sequence highlights persistent closed-loop control over an extended moving-target interaction, where consistent task intent must be maintained across multiple observation–action cycles.

**Figure 12.** Closed-loop rollout for “Kill the salmon.” GameWAM tracks and attacks a moving target in an underwater environment, where both visual appearance and locomotion differ from ordinary land-based gameplay. The rollout demonstrates successful closed-loop control under a qualitatively different gameplay condition.

**Figure 13.** Closed-loop rollout for "craft item purple concrete powder." GameWAM progressively moves the required ingredients from the inventory into their corresponding crafting-grid locations and completes the target recipe. The sequence illustrates precise cursor control and structured multi-step placement through the native GUI action interface.

**Figure 14.** Closed-loop rollout for “Create a dispenser in the crafting table.” GameWAM makes an intermediate placement error while constructing the recipe. After observing the resulting GUI state, subsequent actions adjust the crafting-grid configuration and complete the target item. The rollout illustrates error recovery enabled by repeated closed-loop observation and replanning.

**Figure 15.** Closed-loop rollout on ViZDoom Battle 1 for “Fight off the monsters and stay alive as long as you can.” GameWAM navigates through the environment, reacts to newly visible enemies, adjusts its view toward threats, and repeatedly engages them while continuing the survival-oriented rollout.

**Figure 16.** Closed-loop combat and health recovery on ViZDoom Battle 2 for “Fight off the monsters in the maze and stay alive as long as you can.” GameWAM navigates and fights through the maze. After taking damage, the policy approaches and collects a health pack, recovers health, and subsequently continues the survival-oriented combat rollout. The sequence illustrates adaptation to realized agent state in addition to reactive enemy engagement.

**Figure 17.** Closed-loop rollout for “Defend the line: shoot the monsters advancing toward you from across the room.” GameWAM repeatedly acquires approaching enemies, adjusts horizontal aim, and fires as the active threat changes. The sequence highlights reactive closed-loop control under a frontal-defense objective.

**Figure 18.** Closed-loop rollout for “Defend the center: shoot the enemies closing in from all directions.” Threats can appear from different directions around the agent. GameWAM repeatedly redirects its view and switches targets as enemies approach, illustrating reactive multi-directional defense from a central position.

These visualizations underscore why closed‑loop replanning matters: actions must adapt to realized feedback rather than follow a single pre‑computed plan. Maintaining a mining action, tracking a moving horse, or correcting a GUI error all require conditioning on the environment state at each step.

Cross‑game zero‑shot transfer to VoxeLibre serves as a diagnostic of generalization, not a new benchmark. We deploy the Minecraft‑trained checkpoint unchanged, mapping the native keyboard‑mouse controls deterministically.

**Figure 20.** **Single-forward LASI across analytic noise levels.** (a) Regression gain from within-condition yaw-DCT0 source differences to the corresponding reconstructed clean-action differences. (b) Pearson correlation between the same source and reconstructed action differences. (c) Sign agreement of the paired differences. Each point uses one denoiser evaluation at an analytically constructed noisy state, with no integration update before measurement.

**Figure 19.** **LASI source-space controls.** (a) Correlation between the change imposed on a low-frequency source coefficient and the corresponding change in the predicted clean-action coefficient. (b) Condition-induced variance divided by source-induced variance; values below one indicate larger source variation for the measured mode. (c) Antithetic cancellation ratio for paired $Z$ and $-Z$ sources, measuring source-odd structure. P0–P2 and Y0–Y2 denote pitch and yaw DCT modes 0–2.

Extended LASI analysis (Sections D.5.1‑D.5.8) provides quantitative diagnostics of low‑frequency source influence, frequency selectivity, and amplification along the model path. Key findings include strong low‑frequency donor‑follow rates (94.8 % for yaw DCT 0) and substantial variance reduction when zeroing low‑frequency components.

Single‑forward source transfer shows that the regression gain peaks at 0.709 for yaw DCT 0 ($\sigma$ = 0.68), with Pearson correlation rising from 0.480 to 0.853 as noise decreases. Model‑path amplification later boosts the gain up to 6.9 × for the same mode.

Closed‑loop alignment experiments confirm that the source schedule aligns best with the executed actions (alignment score 0.33763), significantly above all circular‑shift and permutation baselines (p = 0.001). Resampling the action source at each replanning step mitigates coherent bias accumulation.

Scope limitations: GameWAM operates as a low‑level closed‑loop controller and does not maintain symbolic task graphs, inventory plans, or high‑level search. Consequently, tasks requiring explicit procedural knowledge or long‑horizon planning lie outside its intended capability.

Interpretation of LASI diagnostics: the evidence confirms low‑frequency source‑action coupling but does not pinpoint a single internal layer. Mitigation via source resampling is effective at deployment time, while removing the coupling at training time remains an open research problem.

Extended Method Details

Detailed extensions of the training, rollout, and memory mechanisms underlying GameWAM.

Teacher‑forced training and online rollout share the same block‑causal information boundary but differ in how the clean causal prefix becomes available.

During training, complete trajectories provide ground‑truth causal prefixes for each prediction anchor; future variables are corrupted according to the flow objective while observations beyond the prediction horizon stay masked.

Online rollout uses the same dependency structure sequentially: the model predicts a $P$‑step action plan, commits the first $E$ actions, receives the resulting observation, and then replans from the updated clean prefix.

The execution horizon $E$ defines the block‑cycle grid; the prediction horizon $P$ determines how far ahead each anchor is supervised, allowing overlapping targets when $P > E$.

Because complete trajectories are available, overlapping supervision can be extracted without autoregressive generation of intermediate blocks, though larger $P$ still adds loss terms.

During action‑only online evaluation, future video denoising is omitted; the realized visual prefix remains the context for action generation.

A.2 introduces two temporal levels of context: the persistent cross‑cycle state $H_c$ and the within‑cycle realized visual context $C_{c,j}$.

A.3 details the hierarchical memory that stores executed segments in a recent buffer $R_c$ and a long‑term memory $M_c$.

A.4 contrasts the default modality‑decoupled block‑causal mask with a joint within‑block alternative.

A.5 describes how a single native action representation is conditioned on interaction mode via a timestep‑wise routing prediction.

During training, the interaction‑mode label selects the supervised branch; during rollout, the router’s prediction chooses which branch generates the native action vector.

Continuous camera coordinates are normalized per mode, while binary keyboard/mouse bits share a common normalization; validity masks hide unavailable coordinates.

Table 3 lists the 11 native controls used across Minecraft gameplay and GUI interaction.

A.6 clarifies supervision details: continuous and binary action coordinates are reduced separately, teacher forcing applies only to causally complete blocks, and future video generation is omitted during action‑only evaluation.

Conclusion

We wrap up findings, ethical considerations, and reproducibility plans for GameWAM.

We introduced GameWAM, a World–Action Model for native closed‑loop gameplay and GUI control that jointly models future visual observations and executable keyboard‑mouse actions. Across Minecraft and ViZDoom, GameWAM achieves competitive closed‑loop performance while using fewer executed native actions than the compared agents in Minecraft.

We further identify Low‑Frequency Action Source Imprinting (LASI), showing that sampled low‑frequency action‑source structure can induce persistent control bias when reused across replanning steps.

Together, these results support joint world‑action modeling as a promising direction for interactive game control while exposing a source‑sensitivity failure mode specific to generative action policies.

Generative AI tools were used for language polishing and editorial assistance, including improving grammar, clarity, concision, and academic phrasing, as well as refining the presentation of author‑produced experimental results. The core research questions, method, model design, data construction, implementation, experimental results, and numerical measurements were developed or produced by the authors. All AI‑assisted text was manually reviewed and checked against the implementation, experimental records, and underlying results, and was revised by the authors where necessary.

The authors take full responsibility for the final content of this work, including all claims, analyses, and artifacts.

This work studies learned closed‑loop control in simulated video‑game environments. We conduct no new human‑subject experiments and collect no sensitive personal data as part of this study. The reported experiments are restricted to digital game environments and do not involve deployment in physical systems.

While GameWAM is developed for research on native game control, models capable of operating general keyboard‑mouse interfaces could in principle be adapted to other interactive software; deployment beyond the controlled environments studied here should therefore consider authorization, safety, and potential misuse.

We provide detailed descriptions of the model formulation, training objectives, and closed‑loop control procedure in the main text and Section A. Section B documents trajectory alignment, native action interfaces, training data construction, and training‑sequence semantics, while Section C reports the architecture, temporal and history configurations, training‑data mixture and clip‑sampling settings, optimization details, objective weights, sampling configuration, and evaluation protocols. Additional quantitative and qualitative analyses are provided in Section D.

We will publicly release the training and evaluation code, data‑processing scripts, constructed datasets, trained GameWAM model weights, and associated configuration files. The released resources will include the complete procedures required to reproduce the reported experiments, subject to applicable third‑party licensing and redistribution requirements.

Together with the detailed methodological descriptions and experimental protocols provided in this paper, these resources are intended to facilitate reproduction of our results and further research on world‑action models for interactive game control.

Data and Training Configuration

Data pipelines, training setup, and evaluation details for GameWAM.

This section details how all raw recordings are unified into a common observation‑state‑action timeline, how the native action spaces are defined, and the exact training‑data mixture, optimizer settings, and evaluation protocols used for GameWAM.

**Table 4.** ViZDoom action interface used in our experiments. A unified 9-D representation covers both map families. Battle maps use continuous horizontal turning, whereas the two defense maps use binary left/right turning.

The table lists various controls, their types, and the maps in which they are used.

**Table 5.** Model scale and reported training-token consumption. Token categories follow the accounting reported by each work. For two-stage methods, modality-level counts are shown as Stage 1 / Stage 2. A dash indicates that the corresponding count is not separately reported or is not applicable.

**Table 6.** Cross-modal masking, closed-loop performance, and online inference efficiency. We report MCU Mini and MCU All ASR together with executed environment steps averaged over successful episodes under the closed-loop protocols in Section C.6, and separately report online execution frequency.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers