Player-1: Learning to Play from Synthetic Experience
Mingyu Ouyang, Yiran Wang, Siyuan Hu, Xingyilang Yin, Huiqiang Sun, Guangzhi Wang, Junfu Pu, Cheng Cheng, Honghao Chen, Ying Shan, Mike Zheng Shou, Hwee Tou Ng
PLAYER-1 uses synthetic experience and verifiable rewards to teach foundation models complex, transferable game control.
How can we improve computer-use agents to master complex game mechanics by combining supervised demonstration learning with reinforcement learning in synthetic environments?
Computer-use agents struggle to master games because gameplay demonstrations are limited in state coverage and scalable, interactive training environments are difficult to obtain. PLAYER-1 addresses this by initializing models on curated demonstrations and then training them in a suite of sixteen synthetic 2D and 3D environments designed around recurring control demands like aiming, combat, and driving. On the MineDojo-300 benchmark, the 9B model improves success from 10.54% after supervised fine-tuning to 38.33% after reinforcement learning, significantly outperforming existing baselines.
Paper Primer
The core mechanism is a post-training framework that extends standard computer-use action spaces—which typically rely on absolute screen coordinates—to include relative pointer motion, sustained key holds, and concurrent composite actions. This allows the model to execute coordinated maneuvers like running jumps or aiming while firing, which are essential for dynamic game environments.
The training pipeline uses a "synthetic experience" approach: the model learns in PLAYER-1-GYM, a suite of environments with programmatically verifiable rewards and randomized visual and control dynamics. The routing layer is like a mail sorter—it reads the task instruction and current screen, then routes the action through an extended space that supports concurrent inputs, ensuring the model learns to handle sustained, multi-step control rather than just static target selection.
Learning from synthetic experience significantly improves game-control performance over supervised initialization alone.
On the MineDojo-300 benchmark, the Player-1-9B model achieved a 38.33% success rate, a 27.79 percentage point gain over its supervised fine-tuning (SFT) baseline. The 9B model outperformed the strongest evaluated external baseline (12.63%) by more than 3x.
Control capabilities acquired in synthetic environments generalize to external games not seen during reinforcement learning.
On ViZDoom-10, the Player-1-35B-A3B model achieved a scaled return of 0.7236, surpassing the strongest external baseline (GameWAM) by 83.7%.
Why is this approach necessary instead of just training on more gameplay demonstrations?
Demonstrations are inherently limited in their coverage of dynamic game states and often lack the variety required for robust control. By using synthetic environments with verifiable rewards, the agent can explore and learn from its own failures in a controlled, scalable way that demonstrations cannot provide.
What is the scope of this framework—does it require a custom engine for every new game?
No; the framework is designed around recurring control demands (aiming, combat, driving, navigation, UI) rather than specific game titles. The synthetic environments use a shared foundation that allows for consistent observation and action interfaces, meaning the learned control policies transfer to new games that share these fundamental mechanics.
This work demonstrates that synthetic experience is a viable, scalable alternative to massive demonstration datasets for building generalist game-control agents. Researchers can now focus on defining "control demands" rather than collecting thousands of hours of human gameplay.
Introduction and Motivation
We expose why current game agents fall short and introduce PLAYER‑1 to bridge the control gap.
Computer-use agents are models that interpret screenshots and issue keyboard or mouse commands. While they excel at generic UI tasks, games demand precise aiming, sustained movement, and coordinated multi‑key inputs—capabilities these agents currently lack.
Existing game‑agent pipelines train on large‑scale demonstration datasets that only sparsely cover the dynamic state space of open‑ended games, and building scalable interactive environments is notoriously hard. This raises a central question: can synthetic experience push game‑control abilities beyond what supervised fine‑tuning (SFT) provides, and do those gains transfer to unseen games?
PLAYER‑1 is a post‑training framework that starts from a supervised fine‑tuned model and then learns game control by interacting with synthetic environments.
**Figure 1.** Overview of **PLAYER-1**: learning game control beyond demonstrations. A shared game-control action space connects supervised initialization from gameplay demonstrations with multi-environment RL in **PLAYER-1-GYM**. Learning from synthetic experience with verifiable rewards advances control capabilities beyond SFT, with gains that generalize beyond the RL training suite.
Table 1 (referenced) contrasts PLAYER‑1 with prior methods, emphasizing its focus on “beyond demonstrations,” the use of sixteen synthetic environments, and its public availability.
The core gap is that UI‑level interaction does not provide the fine‑grained, sustained, and concurrent control required for competent game play.
Experimental Results and Generalization
PLAYER‑1 models set new performance records on external game benchmarks.
PLAYER‑1 models achieve state‑of‑the‑art performance on external commercial games, with Player‑1‑35B‑A3B reaching a scaled return of 0.7236.
Table 3 reports a scaled return of 0.7236 for Player‑1‑35B‑A3B, exceeding the next best baseline.
**Table 2.** Task success and coverage on MineDojo-300. Results use 8 trials per task. Coverage counts tasks meeting the success thresholds; brackets show 95% task-bootstrap confidence intervals.
**Table 1.** Generalization beyond the RL training suite. The same policy improves performance on external games. On ViZDoom-10, Player-1-9B increases scaled return from 0.4361 after SFT to 0.6922. Player-1-35B-A3B achieves an overall scaled return of 0.7236, surpassing GameWAM, the strongest external baseline at 0.3938, by 83.7%. On GameWorld's 3 × 170 episodes, progress rises from 31.43% to 34.45%. Together, these results show that control experience acquired in PLAYER-1-GYM transfers to first-person shooting and distinct browser games, beyond the synthesized environments for RL. These gains support using synthetic experience to develop transferable control capabilities without reinforcement learning in each target game.
**Figure 4.** Player-1-9B in the held-out commercial game *Watch Dogs 2*: on-foot navigation, car approach, vehicle entry, and city driving. The sequence illustrates coordinated movement and camera control during navigation, and specified inputs for vehicle entry and driving. Reasoning and action text has been lightly shortened and reformatted for readability.
**Table 8.** 3D environments, controls, and verifiable score components. State fields are read from `gameGymAPI.getState().metrics`; `primary_score` gives the resulting cumulative score.
PLAYER‑1 delivers large performance gains on external commercial games, confirming the value of synthetic experience.
The PLAYER-1 Framework
We initialize policies via curated gameplay demonstrations and scale them through reinforcement learning in randomized synthetic environments.
Computer-use agents often struggle with the precise, multi-key coordination required for modern games. We address this by extending the standard computer-use action space to support dynamic game control, initializing the policy with human demonstrations, and scaling it through reinforcement learning in a diverse synthetic suite.
Standard computer-use models treat actions as isolated events, but games require simultaneous inputs—like running while jumping or aiming while firing. Our composite action space allows the model to execute multiple keyboard or mouse inputs concurrently within a single control step.
The policy selects the composite action: $\{\text{hold}(\text{'W'}) \parallel \text{tap}(\text{'Space'})\}$.
The environment receives both inputs in the same control step $\Delta t$.
The character initiates the jump while maintaining forward velocity.
Without composite actions, the model would have to choose between running or jumping, making fluid movement impossible.
**Figure 2.** Curated gameplay demonstrations for supervised initialization. (a) Recorded controls summarized by control type, number of inputs per control step, and pointer-motion mode in our demonstration trajectories. Two or more concurrent inputs form a composite action. (b) A demonstration sample pairing recorded actions with an annotated task instruction and visual reasoning.
Supervised Fine-Tuning (SFT) is the initial training phase where the model learns from human gameplay demonstrations. By predicting the reasoning and actions recorded by expert players, the policy learns the "how-to" of game control before it ever touches the synthetic environment.
**Figure 3.** **PLAYER-1-GYM**: synthetic experience with verifiable rewards. (a) Shared control demands across five task families. (b) Visualization of sixteen 2D and 3D environments. (c) Examples of appearance, camera, and scene randomization applied to the same environment.
Learning Dynamics and Ablations
PLAYER‑1 starts from human demos and refines control via RL in synthetic games.
The paper’s core claim is that initializing from human demonstrations and then scaling with reinforcement learning in a diverse synthetic suite yields stronger game control.
RL refinement takes a policy already seeded by supervised fine‑tuning and improves it by maximizing a reward signal in many synthetic environments.
How does RL refinement differ from standard RL training?
Standard RL typically starts from a random policy, whereas RL refinement begins from a supervised‑initialized policy and optimizes it in a suite of synthetic, randomized environments that expose the same control primitives the agent will later use.
Removing supervised initialization (Base‑RL) cuts Minecraft success by 35 percentage points compared with SFT‑RL after 100 RL steps.
SFT‑RL reaches 51.67 % success while Base‑RL only attains 16.67 % at the same training horizon.
Composite‑action usage rises by 28.5 percentage points (from 64.7 % to 93.2 %) over 100 RL steps, showing that RL naturally discovers coordinated inputs.
Bar charts in Figure 8 record the share of turns with composite actions at 0, 50, and 100 steps.
Expanding the training suite from 0 to 16 environments adds 34.0 percentage points to Minecraft success (17.67 % → 51.67 %).
Figure 7 plots success versus the number of environments, showing a steep early gain that tapers off.
**Figure 5.** Training dynamics and the effect of supervised initialization. (a,b) Mean episode returns and game scores across 16 environments for 9B and 35B-A3B over 200 RL steps; faint traces show raw values and bold curves show EMA. (c) Base→RL and SFT→RL for 9B over 0–100 RL steps.
**Figure 6.** Per-environment episode returns for 9B and 35B-A3B over 200 RL steps. The curves show differences in learning progress, return scales, and relative model performance across diverse synthetic environments.
**Figure 7.** Environment scaling at a fixed budget of 25,600 episodes over 100 RL steps. Zero denotes the SFT reference.
**Figure 8.** Evolution of action composition during PLAYER-1-GYM validation. Composite-action use rises from 64.7% to 93.2% over 100 RL steps. Bars show each checkpoint's most frequent actions as shares of all turns. K-H: key hold; M-M: mouse movement; M-LC: mouse left click.
Game Control Semantics
Appendix details the game‑control interface, data alignment, and dataset statistics.
A.1 COORDINATES AND COMPOSITE ACTIONS – A composite action groups multiple control inputs that are applied concurrently within one game step.
**Table 5.** Game-control semantics. Inputs in a composite action execute concurrently within one step; bindings are restricted to those supported by the environment.
A.2 EXECUTION AND VISUAL FEEDBACK – The game engine interprets mouse displacements as camera rotations, scaling them by a per‑pixel gain before applying any built‑in limits.
A.3 OBSERVATION AND INTERACTION HISTORY – Training data includes the two most recent visual–action pairs and two earlier action‑only frames, with omitted screenshots and reasoning for privacy.
B GAMEPLAY DEMONSTRATIONS: CONSTRUCTION AND ANNOTATION – The dataset aggregates thousands of human‑recorded trajectories across multiple games, providing both raw control logs and curated annotations.
B.1 SOURCES AND STATISTICS – The full collection comprises 6,044 trajectories (≈16.4 hours of control) drawn from 2,544 recording sessions and 172 distinct games.
B.2 ACTION STANDARDIZATION – Recorded inputs are temporally aligned to a uniform 200 ms grid and spatially calibrated to a common pixel scale, enabling consistent composite actions across sources.
B.3 GOAL AND REASONING ANNOTATION – Automated pipelines assign game‑specific goals and annotate per‑turn reasoning using Gemini 3.8 Flash, ensuring uniform semantic coverage.
**Table 6.** Source-specific mouse-action mapping. Relative inputs use calibration coefficients bound to recording geometry; UI operations use absolute endpoints.
Task Environments and Rewards
Appendix C–D Part1 documents task specs, reward formulas, randomization settings, and post‑training procedures.
This appendix details the concrete task definitions, reward calculations, randomization parameters, and the fine‑tuning procedures applied after the main training phases.
**Table 12.** Evaluation panels, budgets, and reporting units. Decision limits are per episode (per role in GameWorld). The three external benchmarks use environments outside PLAYER-1-GYM.
**Table 8.** 3D environments, controls, and verifiable score components. State fields are read from `gameGymAPI.getState().metrics`; `primary_score` gives the resulting cumulative score.
**Table 9.** Appearance, camera, scene, and control-response randomization configurations. Multipliers are relative to each game’s native setting. $U$ denotes uniform sampling and Tri is parameterized by lower bound, mode, and upper bound. Rig parameters affect only camera components used by the game. Scene variation is task-specific and may already be present in native settings.
**Figure 9.** Sources and game distribution of the 6,044 gameplay demonstrations used for SFT. (a) Sources and scale, with sampled control hours. (b) Game distribution.
Because some control actions (e.g., mouse clicks) appear rarely in the raw recordings, the training pipeline deliberately oversamples those actions to give the model enough supervision on them.
Each controllable parameter k in an environment is assigned a random multiplier at reset, drawn from a triangular distribution that centers on the native value, and then kept fixed for the whole episode.
During supervised fine‑tuning (SFT) the model predicts token‑level actions using cross‑entropy loss on the recorded human turns; action‑balanced sampling ensures that rare control inputs receive sufficient gradient signal.
In the reinforcement‑learning phase the agent is trained with synchronous PPO, using a dense critic initialized from a large pretrained model; the critic is warmed up for five iterations before any actor updates.
Penalties are applied per turn: an invalid‑action penalty of 1 and a length‑penalty that caps excessive reasoning tokens, both subtracted from the raw game‑score increment before computing $r_{e,t}$.
Environment Randomization Details
Details of environment randomization, loss normalization, and training hyperparameters.
This appendix enumerates the concrete randomization knobs applied to each game environment, the loss‑normalization scheme used during reinforcement learning, and the hyper‑parameter recipe shared by the two model sizes.
**Figure.** Qualitative examples of the environment variations. The rows represent different tasks (e.g., 2D aiming, 3D navigation), while the columns represent different types of variations applied to the environment (Original, Appearance, Camera, Scene, Joint 1, Joint 2, Joint 3).
Table 10 lists the shared hyper‑parameter recipe for the 9 B and 35 B model variants. Both use 32 NVIDIA H200 GPUs for the supervised fine‑tuning (SFT) stage (≈ 24 h) and 64 GPUs for the 200‑step RL stage (≈ 36 h). The interactive‑learning phase runs 200 PPO steps with a batch of 512 concurrent environments, each managed by a pool of 32 workers.
Environment scaling follows the protocol of Section D.4: each of the 16 environments receives an equal share of the total turn budget, and the randomization mixture is fixed at 100 % across appearance, camera, scene, and control variations. This uniform allocation guarantees that no single visual factor dominates the learning signal.
Training Hyperparameters
Comprehensive details on evaluation environments, baselines, formulas, and cost estimates.
Scaling experiments use the 9B policies that share the same SFT checkpoint, PPO hyper‑parameters, and randomization settings (Table 10). Table 11 enumerates the environment subsets and per‑environment episode budgets used for these runs.
**Table 12.** Evaluation environment subsets and episode budgets.
**Table 13.** Lists the ten selected maps, control categories, native rewards, and fixed return scales. ViZDoom 1.3.0 provides 640 × 480 RGB observations. Each decision advances two simulator tics. Basic permits 150 decisions, Deathmatch 2,100, and the other maps 1,050, subject to earlier native termination. PLAYER-1's camera adapter uses 0.00196 radians per observation pixel with native pitch constraints.
E.1 BASELINE INFERENCE AND ACTION ADAPTERS describes how each baseline predicts a single action per decision. JARVIS‑VLA uses an action‑token decoder with two history entries; UI‑TARS‑1.5‑7B uses three screenshots and a 400‑token output cap; NitroGen aggregates 18 native actions into six benchmark decisions; GameWAM predicts 16 actions and replans after eight; Open‑P2P conditions only on visual input.
E.2 PLAYER‑1‑GYM IN‑SUITE VALIDATION reports the mean episode return $G_{e,1}$, averaging over trials and environments. The return subtracts invalid‑action and length penalties defined elsewhere.
Mean episode return is first averaged across trials within each environment, then equally across the sixteen environments.
E.3 MINEDOJO‑300 comprises 200 Combat and 100 Harvest tasks. Each decision advances a 50 ms simulator tick, and the camera adapter uses a gain of $0.00196$ rad per observation pixel, respecting native pitch limits.
Example tasks include “Hunt a Bat in Night Plains with an Iron Sword, Shield, and Diamond Armor”, “Fight a Husk in Night Desert with an Iron Sword, Shield, and Diamond Armor”, and “Obtain One Log in Plains”.
E.4 VIZDOOM‑10 describes ten maps grouped into categories such as Precision shooting, Goal navigation, Mobile combat, and Survival. Raw rewards are scaled by $c_m$ to obtain the final metric.
Evaluation cost estimates a full‑budget run (≈ 3.14 million decisions) at roughly USD 110 k using GPT‑6 Astra’s token rates, before any caching discounts.
E.5 GAMEWORLD evaluates 170 tasks across 34 browser games. Each task is run for three trials (seeds 0, 1, 2). Success is the fraction of trials that finish, and progress is the normalized achievement toward the task’s target.
Sample GameWorld tasks are OVO (“load into level 3, clear it, and reach level 4”), Temple Run 2 (“reach 4000 points without stumbles”), and Another Gentleman’s Adventure (“gather 4 coins in the current level”).