HiSTEVE 1.0: Harnessing Human-Interface VLA

Zecong Tang, Zian Guan, Yuncheng Guo, Zhanqiu Zhang, Yiwen Guo, Bo Dai

HiSTEVE integrates long-horizon planning with native-interface control for Minecraft agents.

How can we build an embodied agent that reliably performs long-horizon tasks in open-ended games using native keyboard-and-mouse controls?

Minecraft agents struggle to bridge the gap between high-level planning and low-level execution, often failing to maintain task state or handle the distinct visual and action requirements of GUI versus non-GUI interactions. HiSTEVE uses a hierarchical harness that coordinates a VLM planner with a single VLA policy, employing regime-specific normalization and a policy-coupled completion detector to manage subtask transitions. This approach achieves an 85.0% success rate on MineStudio Bench and significantly improves long-horizon performance, enabling the agent to craft complex items from scratch.

Paper Primer

The agent harness acts as an orchestration layer: it tracks verified task progress in a structured memory and uses a VLM planner to select atomic subtasks, while a policy-coupled detector signals when the VLA has successfully completed its current instruction.

To handle the heterogeneous nature of Minecraft, the VLA policy uses Regime-Specific RMSNorm (RS-Norm) and Dual-Mouse token embeddings: the model is like a bilingual speaker who switches between two distinct vocabularies—one for camera-based movement and one for cursor-based interaction—ensuring that shared parameters do not suffer from negative transfer.

HiSTEVE achieves state-of-the-art atomic task performance.

Evaluation on MineStudio Bench across mining, killing, crafting, and smelting tasks.

The harness significantly improves long-horizon coordination.

Comparison of final-goal success on LongBench against the strongest baseline. 28.6 percentage point improvement in Resource & Crafting tasks.

Why is a specialized completion detector necessary instead of using a general-purpose VLM?

General-purpose VLMs are not trained for task-completion detection and often exhibit high false-positive or false-negative rates; HiSTEVE’s detector conditions on the VLA policy context to provide a more accurate, balanced signal for the harness.

What is the primary bottleneck for this agent in long-horizon tasks?

Robust multi-step GUI execution remains the key challenge; while the agent can reach and interact with interfaces, it frequently breaks down during complex item placement or intermediate-state correction.

Reliable long-horizon agency requires more than just a stronger low-level executor; it demands a harness that organizes local execution around a persistent, verified task state.

The Challenge of Embodied Agency

Introducing the planning‑execution gap that HiSTEVE aims to close.

Building agents that can act in open‑ended games like Minecraft demands two complementary abilities: the capacity to pursue goals over long horizons and the ability to translate first‑person visual input into precise keyboard‑and‑mouse actions.

Current Vision‑Language‑Action (VLA) models excel at native control but lack mechanisms for persistent task‑state tracking, while large‑language‑model planners reason over long horizons but depend on environment APIs or hand‑crafted skills, leaving a disconnect between high‑level plans and low‑level execution.

At the planning timescale the agent must continuously track progress and decide when a subtask has truly finished; at the action timescale it must handle heterogeneous interaction regimes—GUI actions that move a cursor versus non‑GUI actions that rotate the camera—without the two interfering.

The image illustrates a sequence of Minecraft gameplay actions performed by an agent, labeled 1 through 4. Step 1 shows the agent mining wood from a tree. Step 2 shows the agent mining stone. Step 3 shows the agent smelting iron in a furnace. Step 4 shows the agent engaging in combat with a zombie.

HiSTEVE addresses this gap with a hierarchical VLA architecture that couples a capability‑aware harness—structured memory plus a vision‑language planner—to a single VLA policy, and adds a policy‑coupled task‑completion detector to close the planning‑execution loop.

The core gap is the mismatch between high‑level planning and low‑level motor control in open‑world games.

The Agent Harness Mechanism

The harness ties a VLM planner to a VLA policy via capability constraints and structured memory.

Long‑horizon tasks in Minecraft break when a high‑level plan drifts from what the low‑level controller can actually do. The capability‑aware harness closes that gap by constantly checking feasibility and feeding back verified outcomes.

A thin orchestration layer that sits between the planner and the controller, turning abstract subtask intents into concrete keyboard‑and‑mouse actions while logging outcomes.

How does this harness differ from earlier agent‑harness designs that simply forward observations to a language model?

Earlier harnesses act as passive routers; our harness actively constrains the planner to only those atomic subtasks the VLA can execute, and it maintains structured memory that records verified state transitions, enabling recovery and replanning.

The planner selects the next atomic subtask only from schemas that the VLA is proven to execute, guaranteeing feasibility at every step.

Check applicability: “place‑item” predicates are satisfied, “move‑forward” is also applicable.

The VLM planner selects “place‑item” because it directly advances the goal; it outputs $u_j$ = “place torch at (1,2)”.

The harness converts $u_j$ to instruction $I_j$ = “use torch on wall at (1,2)”.

The VLA executes $I_j$, the completion detector reports 0.92 probability of success, exceeding the confidence threshold.

Memory updates: working memory records the torch placement, episodic memory logs the successful subtask, procedural memory stores the “place‑item” dependency for future reuse.

This tiny example shows how the planner never proposes an unreachable action; the schema predicates filter out infeasible options before the VLM even sees them.

Why not let the VLM planner generate any subtask and rely on the VLA to fail gracefully?

Unconstrained generation leads to many dead‑end attempts, wasting compute and causing the agent to get stuck. By restricting to capability‑compatible schemas, the harness guarantees that every generated subtask has a realistic chance of succeeding, dramatically reducing wasted interactions.

Gather the latest visual observation $O_j$ and retrieve relevant memory $R_j$.

Construct the applicable schema set $S_j$ by evaluating each schema’s predicates against the current planning state $h_j$.

Invoke the VLM planner: compute $u_j = \text{PVLM}(G, O_j, h_j, S_j, R_j)$.

Translate $u_j$ into a native‑interface instruction $I_j$.

Execute $I_j$ with the VLA policy; the completion detector returns a probability.

If the probability exceeds the confidence threshold or the execution budget expires, hand control back to the harness.

Verify progress using $O_{j+1}$; record the outcome $v_j$ (success, failure, or interrupted) in structured memory.

Update $h_{j+1}$ based on verified progress; loop to the next decision step.

**Figure 1.** System overview of HiSTEVE, illustrating the agent harness mechanism.

VLA Policy and Native Control

Describes the VLA policy architecture, action encoding, and three-stage training for reliable visuomotor control.

The policy turns a subtask instruction plus a first‑person view into native keyboard‑and‑mouse actions, handling GUI and non‑GUI regimes separately.

When the policy predicts a non‑GUI step, it selects token $2$ from the first set, which decodes to a rightward mouse move.

When the policy predicts a GUI step, it selects token $5$ from the second set, which decodes to a cursor‑right click.

The interaction‑type token at the sequence start tells the decoder which subset to interpret, preventing the same token from being ambiguous.

The split eliminates the conflict where the same token would otherwise mean either camera rotation (non‑GUI) or cursor movement (GUI), ensuring deterministic decoding.

Native‑Interface Control encodes Minecraft’s keyboard‑and‑mouse actions as discrete tokens, enabling the policy to predict actions with the same next‑token objective used for language modeling.

How does the interaction‑type token prevent ambiguity between GUI and non‑GUI mouse actions?

The token explicitly tells the decoder which mouse‑displacement subset to interpret, so the same numeric token cannot be misread as both camera rotation and cursor movement.

Reinforcement‑learning stage: branch sampling, advantage computation, and DPO‑style pairwise update.

The detector predicts whether the current subtask has been completed, allowing the harness to pause the VLA policy when the goal is reached.

**Figure 2.** Architecture of the VLA policy model and task completion state detector.

Performance and Evaluation

HiSTEVE sets new performance records on MineStudio Bench and related benchmarks.

HiSTEVE attains an 85.0 % average strict pass@1 on MineStudio Bench, a +20.0 pp gain over the OpenHA Text/CoA baseline.

Table 1 shows HiSTEVE scores of 93.3 % (Mine), 86.7 % (Kill), 66.7 % (Craft), 93.3 % (Smelt), yielding an overall 85.0 % average versus 65.0 % for Text/CoA.

On OpenHA, HiSTEVE reaches a 37.4 % average success, mainly driven by a jump from 37.0 % to 46.0 % on Embodied tasks; GUI and Combat remain slightly below the Text/CoA baseline. LongBench final‑goal success rises to 42.9 % (Resource & Crafting) and 28.6 % (Exploration & Memory), improving macro‑average goal success by up to 28.6 pp over Text/CoA.

**Atomic task execution.** On MineStudio Bench, HiSTEVE achieves strict pass@1 scores of 93.3%, 86.7%, 66.7%, and 93.3% on Mine, Kill, Craft, and Smelt, respectively, corresponding to an overall average of 85.0%. Compared with OpenHA Text/CoA, HiSTEVE improves the four-family average from 65.0% to 85.0%, with gains of 20.0, 13.4, 26.7, and 20.0 percentage points on Mine, Kill, Craft, and Smelt, respectively. The largest absolute gain occurs on Craft, although Craft remains the weakest MineStudio category, indicating that multi-step GUI manipulation remains more challenging than first-person mining and combat.

**Table 2.** Performance comparison of various models on task-completion detection.

**Figure 3.** Qualitative examples of HiSTEVE. The left panel shows successful atomic-task executions, while the right panel illustrates multi-stage long-horizon execution.

HiSTEVE’s 85.0 % average on MineStudio Bench sets a new state‑of‑the‑art, surpassing prior native‑control agents by up to 20 pp.

VLA Architecture and Gradient Analysis

Limits of the VLA design are examined through gradient‑conflict analysis and action‑encoding details.

The central premise—that high‑level plans must stay aligned with low‑level motor execution—remains critical when extending VLA to mixed GUI and non‑GUI regimes.

Gradient conflict measures how much the direction of parameter updates diverges when the model sees GUI versus non‑GUI data, revealing whether a shared decoder can simultaneously satisfy both interaction regimes.

Across 30 000 training steps the mean conflict gap over all layers and components stays around 0.20, with a fitted slope of $-5.4\times10^{-7}$ per step (≈ $-0.016$ total), indicating virtually no reduction in misalignment.

When averaged by decoder depth the gap drops from 0.257 in the early layers (L0–L8) to 0.124 in the deepest layers (L19–L27), a per‑layer decline of $-0.007$, showing stronger regime tension near the input side.

Component‑wise analysis reveals the highest gaps in the two RMSNorm modules (0.606 for input RMSNorm, 0.317 for post‑attention RMSNorm); the remaining projections and feed‑forward network range from 0.225 (K) down to 0.038 (FFN).

The VLA policy reuses the Qwen2.5‑VL‑7B backbone unchanged except for regime‑specific RMSNorm branches and separate mouse‑token embeddings; the regime token $c_t$ routes the forward pass to the appropriate branch.

Native actions are encoded as discrete tokens: keyboard keys occupy reserved vocabulary slots, while mouse displacements are µ‑law‑companded, quantized into 21 bins per axis, and mapped to distinct token sets for GUI and non‑GUI regimes (see Table 4).

**Figure 4.** Overview of the cross-regime gradient-conflict analysis. (a) Mean conflict gap over all layers and components throughout training; the shaded region denotes ±1 standard deviation across components, and the dashed line is a linear fit. (b) Mean conflict gap by decoder depth, averaged over training steps and components. (c) Mean conflict gap by shared component, averaged over training steps and decoder layers. All panels are computed from the same set of gradient probes.

Table 4 lists the 15 native Minecraft actions (e.g., Forward, Jump, Attack, mouse‑move tokens) and their corresponding keyboard or mouse inputs, providing the concrete action vocabulary used by the VLA policy.

Training and World Knowledge

How HiSTEVE acquires Minecraft knowledge, imitates human play, and refines behavior with RL.

HiSTEVE’s training proceeds in three stages: first it learns Minecraft world facts, then it copies human gameplay, and finally it refines the policy with reinforcement learning.

The language decoder is taught the names, properties, and crafting rules of every Minecraft entity so that a subtask description can be grounded in the visual scene.

HiSTEVE’s action vocabulary mirrors Minecraft’s native controls; Table 4 lists the 15 token types, split between movement, interaction, and GUI cursor displacement.

Starting from the world‑knowledge checkpoint, the policy copies human gameplay trajectories, learning to map a single‑frame observation and a subtask instruction to the exact keyboard‑mouse token sequence.

**Table 9.** Representative imitation-learning examples from the non-GUI and GUI interaction regimes. Each example contains a single-frame observation, an atomic-subtask instruction, and its aligned target action sequence.

After imitation learning, HiSTEVE is further refined with reinforcement learning that rewards true environment progress rather than model‑predicted success.

Success is measured by observable changes in the game state (e.g., inventory count) instead of the model’s internal confidence.

Samples are re‑weighted by how likely the old behavior policy would have taken them, then the new policy is updated with a clipped importance‑sampling ratio to keep updates stable.

Task Completion Detector Construction

Implementation specifics for the completion detector and its integration.

We build a completion‑detection dataset from VPT human‑gameplay trajectories by extracting atomic‑task segments, detecting their completion times, and sampling temporal windows that never cross trajectory boundaries.

Each sampled window is marked “complete” exactly when it occurs at or after the true task‑completion moment, otherwise “incomplete”.

After the detector’s decoder layers, we inject additional attention heads that can attend to the VLA’s hidden state, allowing the detector to incorporate visual‑language context.

We keep only the most active feed‑forward channels, discarding the rest to reduce compute and memory.

We fine‑tune only the newly added cross‑attention heads, the classification head, and LoRA adapters while keeping the frozen VLA backbone untouched.

At inference time the detector runs before the VLA decides the next action, and a completion signal is emitted only after two consecutive high‑confidence predictions.

Requiring two successive predictions above a confidence threshold dramatically reduces spurious completions.

Experimental Protocol and Benchmarks

Describes the atomic and long‑horizon evaluation protocols and the harness ablations.

This appendix details how we measure low‑level VLA performance, long‑horizon composition, and the effect of the orchestration harness. Atomic tasks isolate the VLA, while LongBench stresses persistent behavior across many invocations. The harness ablation in Table 3 isolates orchestration from policy quality.

MineStudio Bench comprises 20 atomic tasks split evenly among Mine, Kill, Craft, and Smelt, each run three times. We report strict $pass@1$, i.e., success only on the first execution attempt. Category scores $SMine$, $SKill$, $SCraft$, $SSmelt$ are averaged to obtain $SMineStudio$ as in Equation (35).

**Table 1.** Ablation study of HiSTEVE components.

LongBench evaluates whether atomic VLA capabilities can be composed into persistent behavior when only a final goal is visible. The suite contains 14 goal‑only tasks organized into four families that stress resource chaining, memory‑based navigation, recovery, and open‑ended strategy selection.

**Table 5.** LongBench task suite. Each episode exposes only the final goal to the agent; the decomposition and diagnostic checkpoints are withheld during execution.

Each episode defines a set $D_i$ of hidden diagnostic nodes; the subset $R_i\subseteq D_i$ actually reached yields diagnostic progress $P_i=|R_i|/|D_i|$ (Eq. 36‑37). This metric is recorded for analysis but does not affect success.

The primary LongBench metric is final‑goal success rate $GoalSR=\frac{1}{N}\sum_i g_i$ (Eq. 38), where $g_i$ indicates whether the terminal predicate is satisfied. Table 1 reports family‑level $GoalSR$, and Table 3 reports the macro‑average across families.

To isolate orchestration effects, the harness ablation keeps the native‑control VLA and semantic model fixed while varying only the handoff logic. The fixed‑horizon variant returns control after a predetermined boundary; Full HiSTEVE uses state‑dependent handoff, visual verification, and optional recovery.

**Table 6.** Ablation of the state detector design. Each group changes one design factor while holding the remaining settings fixed.

Ablation Study of Design Choices

Ablations reveal which design choices impact detection accuracy and guide the final HiSTEVE configuration.

Recall that HiSTEVE’s core premise is to constrain planning to feasible native‑interface actions, using a capability‑aware harness that aligns high‑level plans with low‑level motor execution.

Reducing the FFN width to one‑third retains accuracy (≈ $0.863$) compared with the half‑width baseline ($0.865$), while a quarter‑width drops accuracy sharply to $0.739$.

Accuracy changes: $0.865 arrow 0.863$ (one‑third) and $0.865 arrow 0.739$ (one‑quarter) on the balanced validation set of 4,674 samples.

Naïve prefix truncation of channels reduces accuracy to $0.768$, whereas selecting channels with Wanda‑inspired activation‑weight scores keeps accuracy at $0.863$.

At the one‑third FFN width, naive truncation yields $0.768$ accuracy versus $0.863$ with Wanda scores.

Increasing cross‑attention density beyond every fourth layer offers no accuracy gain: inserting every two layers reaches $0.869$, every layer $0.859$, while the every‑fourth‑layer placement attains $0.863$, within $0.006$ of the best.

Accuracy: every‑two layers $0.869$, every layer $0.859$, every fourth layer $0.863$.

Broadcasting the final‑layer policy context $H^{\pi}_t$ to all cross‑attention modules preserves accuracy ($0.863$), whereas using depth‑matched intermediate policy representations reduces accuracy to $0.690$.

Accuracy drops from $0.863$ to $0.690$ when replacing final‑layer broadcast with layer‑wise pairing.

**Table.** Final-Layer Broadcast versus Layer-Wise Pairing. Using depth-matched intermediate policy representations reduces accuracy from 0.863 to 0.690. This substantial gap supports broadcasting the final-layer policy context to all cross-attention modules, as it integrates information from the current observation and subtask. We therefore broadcast the same final-layer policy context $H_t^{\pi}$ to all cross-attention modules in the final detector.

Harness Implementation Details

Implementation specifics of the agent harness: runtime loop, atomic interface, verification, and memory.

Appendix A supplies the low‑level implementation details of the agent harness introduced earlier, covering its runtime loop, atomic subtask interface, verification and recovery logic, and the structured memory it maintains.

The harness repeatedly alternates between a high‑level planner that proposes a semantic subtask and a VLA that executes that subtask until a handoff condition signals it should stop.

Each atomic schema defines a bounded, capability‑aware operation that the VLA can execute directly via native keyboard‑and‑mouse actions.

The harness separates the moment the VLA stops (handoff) from the moment visual evidence confirms success, failure, or the need for recovery.

The harness maintains a three‑part memory that stores episodic observations, procedural knowledge, and world‑level facts, which the planner queries to stay coherent over long horizons.

Discussion and Future Directions

We reflect on the harness’s role, current limits, and paths toward continual, transferable embodied agents.

The core difficulty for long‑horizon embodied agents is keeping task state coherent between high‑level planning and low‑level motor execution. The agent harness addresses this by persisting task state, constraining plans to native‑interface actions, and feeding execution outcomes back into decision making.

When the Vision‑Language‑Action (VLA) controller is held constant, the harness only modestly shifts aggregate atomic performance yet yields a large boost in LongBench goal‑success rates. This reveals that long‑horizon capability hinges not just on the strength of the local controller but on how existing capabilities are orchestrated across repeated invocations.

A design insight is to separate control handoff from task‑success verification. The completion detector emits a candidate handoff signal, after which the harness confirms completion using visual evidence, preventing detector errors from corrupting persistent task state.

The current system has two main limitations. First, atomic subtask interfaces are manually defined and bounded by the existing VLA policy; extending capabilities demands new training data and policy updates, and neither the harness nor the VLA performs parametric continual learning. Second, high‑level planning depends on an off‑the‑shelf VLM, making overall performance contingent on that backend’s abilities.

Future work could let the VLA learn continually from failure and recovery trajectories while the harness updates its planning constraints and recovery strategies from experience. Additionally, using Minecraft as a training ground may reveal how planning, memory, and recovery skills transfer to other interactive domains.

**Figure.** A first-person perspective screenshot from the game Minecraft showing a player character holding a shield while facing a zombie in a dimly lit cave environment. A pool of lava is visible in the background, and the player's inventory bar is displayed at the bottom of the screen.

**Figure.** A screenshot from the game Minecraft showing a desert village biome. The scene features sandstone buildings, sandy ground with patches of terracotta, and a player's inventory bar at the bottom containing tools and food items.

**Subtask instruction:** Craft the fishing rod at the crafting table.

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers