Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents
Jiaming Zhang, Xuan Wang, Fuyao Zhang, Yang Cao, Lingjuan Lyu, Wei Yang Bryan Lim
LGWM verifies GUI agent transitions by comparing predicted and observed latent representations, bypassing expensive rendering.
How can GUI agents detect malicious interface transitions by predicting the expected next screen in latent space rather than pixel space?
GUI agents are vulnerable to "hidden-transition" attacks where a malicious interface produces a legitimate-looking screen after an unexpected action. Because the screen itself is plausible, existing monitors that inspect only the final image cannot distinguish a safe transition from a hijacked one. The authors introduce LGWM, an action-conditioned world model that predicts the next screen's representation directly in the same latent space as the observation. By comparing the predicted and observed vectors using cosine similarity, the system detects integrity violations in 17 ms without needing a decoder or a secondary judge. On the RSWT-BENCH diagnostic, where outcome-only detection is provably at chance, this training-free approach reaches 0.987 AUC, matching the performance of frontier closed-source VLMs at three orders of magnitude lower latency.
Paper Primer
LGWM functions as a "latent gate": it maps the current screen and a structured action to a spatial map of the expected next screen's representation. The core move is to treat the prediction as a direct geometric target for the observed future, using a change-weighted objective to focus the model on the sparse regions of the GUI that actually shift during a transition.
LGWM achieves state-of-the-art transition integrity detection without task-specific training.
On RSWT-BENCH, the training-free integrity score reaches 0.987 AUC, performing on par with Gemini 3.7 Flash and GPT-5.6 Luna. The system operates at 17.1 ms per decision, which is over 60x faster than the fastest VLM baseline tested.
The residual vector between prediction and observation separates harmful attacks from benign surprises.
A linear head trained on the residual direction achieves 0.953 AUC at identifying harmful violations, whereas magnitude-based anomaly scores (and prompted VLMs) perform near chance. This directional readout allows the system to ignore benign events like push notifications or layout shifts that would otherwise trigger false positives.
Why is rendering the future a bottleneck for GUI agent safety?
Rendering requires a second model (a judge) to compare the generated artifact against the observed screen, which is computationally expensive and introduces a new point of failure. LGWM eliminates this by comparing vectors directly in a shared latent space.
What makes RSWT-BENCH a "strict" test for GUI detectors?
It uses donor-paired construction where the same credential screen appears as both a legitimate and a hijacked outcome. This makes any detector relying solely on screen appearance mathematically incapable of scoring above 0.5 AUC.
For GUI agents, world models should shift from generating human-readable simulations to providing machine-readable verification signals. Predicting in representation space allows for real-time, step-by-step integrity monitoring that is both cheaper and more precise than VLM-based reasoning.
The RSWT Problem
We expose the transition‑based safety gap of GUI agents and introduce LGWM for verification.
GUI agents can be tricked by a malicious transition that reuses a legitimate screen, so safety must be judged at the level of the action‑to‑screen transition. Existing world models render predictions as text, code, or images, which forces a second model to compare the rendered output with the observed screen, incurring large latency. Predicting directly in the latent representation of the observation eliminates the decoder and enables a fast vector similarity check.
A screen that looks identical can be safe or unsafe depending on the action that produced it; the same pixel image after a legitimate tap is benign, but after a malicious tap is an attack.
**Figure 1.** **The future as an interface.** (a): Existing GUI world models render the future as text, pixels, or code, at a cost of seconds to minutes per decision. LGWM predicts the future in representation space, in milliseconds. (b): A pixel-identical credential screen is legitimate after *Sign in* but hostile after *View order*. Safety is a property of the transition, not the screen.
Safety is a property of the transition, not the screen.
Latent GUI World Model
LGWM predicts the next GUI state in latent space and uses residuals to verify transitions.
Integrity of a GUI transition cannot be judged from the resulting screen alone; we must compare the observed outcome with what a faithful world model expects given the prior screen and action.
LGWM learns a compact latent representation of the GUI and predicts the next latent state directly, so downstream checks can operate on vectors instead of costly pixel reconstructions.
Patch 1: $[1,0] + [0,1] = [1,1]$ → $\hat{z}_{t+1,1}$
Patch 2: $[0,1] + [0,1] = [0,2]$ → $\hat{z}_{t+1,2}$
Patch 3: $[1,1] + [0,1] = [1,2]$ → $\hat{z}_{t+1,3}$
Patch 4: $[0,0] + [0,1] = [0,1]$ → $\hat{z}_{t+1,4}$
The EMA encoder reads the true next screen and outputs $\bar{z}_{t+1}=\{[1,1],[0,2],[1,2],[0,1]\}$, matching the prediction exactly.
This toy example shows how the predictor operates token‑wise in the same latent grid that the target encoder produces, making a direct element‑wise comparison possible.
Because most GUI patches stay unchanged, we up‑weight the few patches that actually change so the loss focuses on the transition signal.
At runtime we compare the predicted latent future with the observed latent future; the residual’s magnitude flags unexpected changes, while its direction tells whether the change is benign or harmful.
**Figure 2.** One predicted future, two closed-form readouts. (a) Training: an online screen encoder and spatially grounded action tokens condition a latent predictor; an EMA encoder supplies the next-state target, and changed patches receive higher loss weight. There is no decoder anywhere in the model. (b) Runtime: because predicted and observed futures share one space, the alignment gap gives a training-free integrity score (surprise detects), while the directional residual feeds a linear head that separates harmful from benign violations (direction interprets). Training-only paths are dashed.
Runtime verification – compute residual, surprise, and harm scores.
Performance on RSWT-BENCH
LGWM detects malicious transitions with near‑human AUC while running in milliseconds.
We pre‑train LGWM on 1,845,659 transitions from five GUI corpora (training = 1,824,824, validation = 20,835) using AdamW, bfloat16, batch size = 512 on four A100‑80GB GPUs. The model contains $261\,\text{M}$ parameters, of which $172.86\,\text{M}$ are active at runtime. Loss coefficients are $\lambda_{\text{chg}}=8$, $\lambda_{\text{cov}}=0.01$, and $\lambda_{\text{inv}}=0.5$.
On RSWT‑BENCH the integrity score (Eq. 4) attains $0.987$ RSWT AUC, correctly ordering $219$ of $222$ donor pairs, while three blind annotators reach $0.958$ AUC on the same $40$ examples. Generative world‑model baselines (gWorld‑8B, Code2World) fall $ \approx 10$ points lower and require $ \geq 92.9$ s and $ \geq 182\times$ FLOPs per decision. The learned dynamics prior is the only source of signal; a detector that sees only the outcome is forced to $0.500$ AUC (Prop. 1).
A benchmark that pairs a hijacked screen with its legitimate counterpart, differing only in the transition that produced the identical final screen.
LGWM attains $0.987$ RSWT AUC, surpassing all closed‑source and open VLM baselines.
Orders $219$ of $222$ donor pairs correctly; the next best closed‑source system (Gemini 3.7 Flash) trails by $0.008$ AUC (p = 0.07).
**Table 1.** **Main results on RSWT-BENCH.** Detection is pairwise AUC: RSWT compares hijacked and legitimate occurrences of identical screens; Swap and Cred compare benign mismatches with paired safe transitions and legitimate credentials, respectively. Cost is measured per decision at batch size one on one A100-80GB. Blind human annotators reach 0.958 RSWT on 20 hijacked and 20 legitimate examples.
LGWM achieves near‑human detection AUC with millisecond latency.
Ablations and Degenerate Explanations
We verify that the predictor truly anticipates future screens rather than copying or memorizing.
The verifier checks whether the latent prediction matches the true next screen, confirming that LGWM’s integrity signal is not a shortcut.
Before trusting a high RSWT score we must rule out simple cheats: a predictor that merely copies the current screen, memorizes training transitions, ignores the action, or only works for a single step.
We now evaluate three design components by removing them one at a time and measuring the metric each component was intended to improve (Table 2).
Removing rollout training hurts the ten‑step recursive cosine agreement (‑10 %), confirming that exposure to its own imperfect forecasts is essential for stability. Dropping inverse dynamics reduces the copy‑score gap (‑32 %), showing that the auxiliary loss forces the representation to encode the action that caused the transition. Without VICReg the effective rank collapses (‑49 %), indicating that variance‑covariance regularization prevents representation collapse and preserves discriminative power.
**Table 2.** Design ablations (ViT-S, 15k steps, one seed). Each row removes one component and measures the metric its design targets.