Focused Omni-Modal GUI Agents
Fuyao Zhang, Wenwen He, Jiaming Zhang, Yurong Hao, Xuan Wang, Guowei Guan, Che Wang, Longtao Huang, Wei Yang Bryan Lim
FocusGUI dynamically prioritizes multimodal inputs to prevent compounding errors in long-horizon GUI agents.
How can we improve GUI agent performance by dynamically filtering and prioritizing multi-modal inputs instead of feeding the entire context to the model at every step?
GUI agents often fail in multi-step tasks because they treat all input modalities—screenshots, audio, and history—as equally relevant at every step, leading to "Sequential Distraction" where errors compound over time. FocusGUI solves this by dynamically scoring and prioritizing these input channels at each step, compressing the most relevant evidence into a compact latent that guides the model's attention without altering its frozen backbone. This approach significantly improves action accuracy, particularly in deeper interaction steps where standard models typically drift.
Paper Primer
The core mechanism is a zero-initialized residual adapter: FocusGUI decomposes the observation into separable channels, scores their relevance against a step-specific decision reference, and injects the compressed result as a residual at the action boundary. It acts like a spotlight operator: it doesn't change the scene (the input), but it forces the decoder to focus its attention on the most critical evidence for the current action.
FocusGUI significantly outperforms frozen backbones and existing spatial-refinement methods on multi-step GUI tasks.
Across four omni-modal backbones, FocusGUI raised Type Match by +33.2 and Exact Match by 3.0× over frozen baselines. It reaches 1.9× the Type Match of the strongest existing enhancement method.
The gains are depth-dependent, confirming the mechanism effectively mitigates Sequential Distraction. While frozen models lose half their accuracy by the fourth step of an interaction, FocusGUI retains 86.8% of its initial performance.
Why is spatial cropping (the common alternative) insufficient for these agents?
Spatial cropping only addresses visual saliency, ignoring decisive signals from audio, video, or device states. Furthermore, it often discards the surrounding context necessary for grounding precise coordinates and text parameters.
Does this method require expensive fine-tuning or specialized labels?
No. FocusGUI is trained from action supervision alone, requiring no modality-relevance labels, and it leaves the underlying backbone parameters completely frozen.
The Challenge of Omni-Modal GUI Agents
We expose the transient, sparse relevance problem that hampers omni‑modal GUI agents.
Omni‑modal GUI agents ingest full screenshots, audio, video, interaction history, and device states at every step, yet the signal that determines the next action is often confined to a tiny, transient slice of that observation. Because the executed action feeds back into the observation, any misread of relevance propagates, leading to the compounding failure we call Sequential Distraction.
**Figure 1.** Relevance is transient across steps and sparse within them. Across multi-step episodes, we systematically ablate individual modalities from the input of Qwen2.5-Omni and record the resulting change in the log-likelihood of the ground-truth action. A modality is useful if its ablation drops this likelihood, and interfering if its ablation increases it; the dominant modality is the one whose absence causes the steepest drop. (a) The dominant modality switches across 45.3% of adjacent steps, with 94.0% of episodes containing at least one switch. (b) 90.1% of steps contain at least one interfering modality. (c) Based on annotations, the target element occupies a median of a mere 0.87% of the screen area, and video content changes by a median of just 7.41% within a step.
Agents need a step‑specific, compact representation that captures only the modalities that actually influence the current action, because most of the streamed observation is irrelevant and can mislead the decoder.
The performance gap in long‑horizon GUI tasks stems from agents processing full, noisy context without a step‑wise focus signal.
The FocusGUI Mechanism
FocusGUI adds a compact conditioning residual to guide the frozen decoder at each step.
The frozen backbone $F_{\Theta}$ cannot be altered, so the decoder must be guided to attend to the most relevant parts of the multimodal observation at each interaction step. FocusGUI provides a lightweight conditioning path that scores and compresses the observation into a compact focus latent, which is injected as a residual before action generation.
Each modality and auxiliary source is kept as a separate channel so that its relevance can be scored independently at every step.
Project each source: $x_{\text{text}} = (1,0)$, $x_{\text{img}} = (0,1)$, $x_{\text{audio}} = (1,1)$, $x_{\text{state}} = (0,0)$.
Collect the four channel embeddings into $X_t = \{x_{\text{text}},x_{\text{img}},x_{\text{audio}},x_{\text{state}}\}$.
These embeddings now reside in the common $d=2$‑dimensional space and are ready for scoring.
Separate channels preserve modality‑specific information while enabling a uniform scoring mechanism.
Why not simply concatenate all raw modalities into one vector instead of keeping separate channels?
Concatenation mixes the statistics of heterogeneous sources, making it impossible to assign a meaningful relevance score to any single modality. Separate channels let the model compute a per‑modality priority $\alpha_{m,t}$ that can change step‑by‑step.
A joint context vector summarizing the whole observation is computed so each channel can be compared against a common reference.
How does the decision reference differ from the usual query vector in standard attention?
In standard attention the query is derived from the current decoder state. Here $q_t$ is derived from the *entire* multimodal observation, providing a global task‑aware reference that is independent of the decoder’s hidden state.
The scored channels are weighted, compressed into a low‑dimensional focus vector, reshaped into a residual, and added to the decoder’s embedding window just before the action tokens.
Compute softmax priorities: $\alpha_{1}= \frac{e^{0.8/0.5}}{e^{0.8/0.5}+e^{1.2/0.5}+e^{0.4/0.5}} \approx 0.21$, $\alpha_{2}\approx 0.66$, $\alpha_{3}\approx 0.13$.
Weighted sum $c = 0.21\cdot2 + 0.66\cdot5 + 0.13\cdot1 \approx 3.73$ (scalar for illustration).
Project to $z\in\mathbb{R}^{2}$ via $P_s$: $z = (3.73,\,3.73)$ (identity projection for simplicity).
Linear map $W_p$ (zero‑initialized) yields $W_p z + b_p = \mathbf{0}$, so $R$ is initially a $2\times4$ zero matrix.
After a few training steps $W_p$ learns a non‑zero vector; suppose $W_p z = [0.5, -0.3, 0.2, 0.1]$ per row, giving $R$ a small residual that is added to the two positions before the action token.
The residual starts empty and only grows where the learned focus indicates a useful bias, ensuring the model never loses the original multimodal evidence.
Why add the focus residual instead of replacing the original embeddings at $J_t$?
Replacing would discard the untouched multimodal evidence, which the frozen backbone still provides. Adding preserves all original information while giving the model a lightweight, learnable bias that can steer the decoder without breaking the backbone’s invariants.
**Figure 2** FocusGUI. The frozen backbone’s original pathway is left intact (top). From the same observation, FocusGUI exposes perceptual channels, scores them against a decision reference recomputed at every step, compresses the result into a compact focus latent, and adds it as a zero-initialized residual at the K positions preceding the action boundary.
Empirical Evaluation
FocusGUI delivers large accuracy gains while keeping trajectories stable.
FocusGUI improves performance across all backbones, adding +33.2 TM, +22.3 EM, and +6.6 GP points over direct inference and beating the strongest baseline by +30.1 TM, +23.6 EM, +8.0 GP.
See Table 1 for the full breakdown across Qwen2.5‑Omni, VITA‑1.5, MiniCPM‑o 4.5, and Baichuan‑Omni‑1.5.
The benchmark stitches together screenshots, audio, video, and action history for each step, so any modality can carry the decisive signal.
Why does the benchmark include both visual and auditory streams if many tasks seem visual?
Some actions depend on spoken cues (e.g., “pause when you hear ‘childhood’”), so a purely visual model would miss the trigger entirely. The benchmark forces agents to consider all streams.
Qwen2.5‑Omni is a large‑scale transformer that ingests text, images, audio, and video in a unified token stream.
How does Qwen2.5‑Omni differ from a standard vision‑language model?
It processes audio and video tokens alongside text and image tokens, enabling true omni‑modal reasoning rather than just vision‑language fusion.
**Figure 3.** FocusGUI holds accuracy as trajectories lengthen. Per-step Exact Match (left) and Type Match (right) on Qwen2.5-Omni, with steps binned by their index within the episode and 3+ collecting the fourth step onward. Shaded regions are 90% episode-clustered bootstrap intervals.
**Figure 4.** Learned channel priorities reorder across interaction phases. (a) Mean priority $\bar{\alpha}^{(m)}$ per channel and phase on Qwen2.5-Omni; uniform allocation would be 14.3%. (b) Deviation of each cell from that channel's all-step mean. FocusGUI trains on action annotations alone, with no modality-relevance supervision.
**Figure 5** Case study: a decision that never appears on the screen. The instruction conditions the action on what is heard, so the screen looks the same at every step while the deciding evidence sits in audio. FocusGUI waits and taps once the engine sounds arrive; the frozen backbone acts early on visual salience and cannot recover, since its own action re-enters the context of every later step. Radar plots show the channel priorities $\alpha_t^{(m)}$ per step.
**Figure 11** Case study on cross-modal temporal reasoning. The agent is instructed to pause a video based on specific auditory cues ("past/history/childhood"). The visual interface remains largely static, forcing the model to rely on streaming audio comprehension over multiple steps. FocusGUI successfully avoids premature visual actions, dynamically shifting priority to the audio channel and executing the spatial action (tapping pause) precisely when the semantic condition is met.
**Figure 12** Case study on multi-step semantic planning and spatial localization. The agent executes a complex sequence of interface interactions based on a detailed text instruction. It successfully navigates through dynamic UI changes—locating a small filter icon, understanding the popup menu, and sequentially selecting "Like count", "Liked", and "This Week". This demonstrates FocusGUI's capacity for precise spatial grounding and robust state tracking in visually dense environments.
FocusGUI maintains accuracy across long trajectories where baselines decay.
Related Work
Survey of prior GUI agent approaches and their handling of omni‑modal perception.
GUI agents translate user intentions and interface observations into actions, forming a perception–action loop that spans mobile, web, and desktop platforms. Recent work extends this loop from isolated grounding decisions to long‑horizon interaction, but most agents still organize observations around the current screenshot with prior screens as auxiliary context. Temporal GUI benchmarks introduce dynamic visual content, and OmniGUI further interleaves screen, video, audio, and interaction history, shifting the problem from sequential interfaces to evolving omni‑modal states.
Perception is crucial because instructions must be linked to actionable interface elements, prompting research on GUI‑specific representation learning and large‑scale grounding data. Selective perception methods such as visual search, test‑time refinement, and active perception prune irrelevant screen regions, yet they focus solely on spatial localization. FocusGUI expands selective perception to the omni‑modal setting by dynamically scoring perceptual channels and constructing step‑dependent representations, complementing existing grounding mechanisms.
Extended Experimental Analysis
FocusGUI compresses noisy perceptual channels into a task‑relevant focus, boosting TM, EM, and GP.
We probe whether each component of FocusGUI is essential by removing it and measuring the impact on TM, EM, and GP.
**Table 3.** Sensitivity to the boundary window $K$ and the compact focus dimension $d_z$ on Qwen2.5-Omni. The default ($K = 4$, $d_z = 64$, shaded) is fixed across all backbones and never tuned per benchmark.
**Figure 6.** The capacity grid is flat around the default. Overall TM, EM, GP, and Temporal GP as a function of the boundary window $K$ and the action-sufficient dimension $d_z$ on Qwen2.5-Omni. Circles mark measured configurations, and the surface is a thin-plate interpolation between them; the diamond marks the default $K = 4, d_z = 64$ used throughout the paper. The interpolated regions carry no measurement and are shown only to convey the shape of the grid.
**Figure 7** MiniCPM-o 4.5 across interaction depth. Per-step Exact Match (left) and Type Match (right) with steps binned by their index within the episode and 3+ collecting the fourth step onward. Shaded regions are 90% episode-clustered bootstrap intervals. As on Qwen2.5-Omni, the frozen backbone declines with depth while FocusGUI holds, so the margin widens rather than staying constant.
Dynamic prioritization raises Exact Match by +3.54 points.
Table 5 (budget‑fixed control) reports the improvement when per‑step scoring replaces uniform averaging.
Goal Progress improves by +3.44 points under dynamic prioritization.
Table 5 shows the same configuration yields the largest gain on the GP metric.
Type Match gains +0.98 points with the dynamic scoring mechanism.
Table 5 records a modest but consistent uplift for TM when channels are scored per step.
Table 5 fixes the parameter budget (within 79 parameters) and replaces the decision reference with a static concatenation, confirming that the observed gains arise from dynamic prioritization rather than extra capacity.
Formal Algorithm Specification
FocusGUI compresses multimodal inputs via step‑adaptive channel prioritization and residual alignment.
Processing every perceptual channel at each interaction step forces the model to waste compute on irrelevant signals and dilutes the decision signal.
Instead of treating all modalities equally, the model scores each channel at the current step and blends them proportionally – like a spotlight that brightens the most relevant sensors while dimming the rest.
Compute raw scores: $s^{(v)} = v^\top W_q q_t = 2$, $s^{(a)} = a^\top W_q q_t = 0.5$, $s^{(t)} = t^\top W_q q_t = 1$.
Apply temperature $\tau=1$: $\exp(s/\tau)$ yields $[e^{2}, e^{0.5}, e^{1}] \approx [7.39, 1.65, 2.72]$.
Normalize: $\alpha^{(v)} = 7.39/(7.39+1.65+2.72) \approx 0.66$, $\alpha^{(a)} \approx 0.15$, $\alpha^{(t)} \approx 0.19$.
Form focus vector $c_t = 0.66\,v + 0.15\,a + 0.19\,t$, a compact blend that heavily favors the visual channel.
Even with a simple linear score, the softmax automatically suppresses irrelevant modalities, demonstrating how the model can allocate compute where it matters.
How does this differ from the softmax attention weights used in standard Transformers?
Standard attention computes a softmax over dot‑products between a query and each key, weighting *values* from the same sequence. Here the softmax operates over *independent modality scores* that combine a modality‑specific embedding with a global decision reference; the resulting weights blend whole‑modality embeddings rather than individual token‑level values.
Forward pass of FocusGUI (Algorithm 1)
**Algorithm 1** FocusGUI forward pass at interaction step $t$ **Require:** Observation $\mathcal{O}_t$; frozen backbone $\mathcal{F}_{\Theta_0}$; trainable $\phi = \{P_m, P_q, \{r_m\}, W_e, W_q, v, P_s, W_p, b_p\}$; hyperparameters $K, d_z, \tau, \gamma$ **Ensure:** Modified embeddings $\tilde{E}_t$; channel priorities $\{\alpha_t^{(m)}\}$ 1: $E_t, \{x_t^{(m)}\}_{m \in \mathcal{M}}, e_t^{(\text{all})} \leftarrow \mathcal{F}_{\Theta_0}.\text{ENCODE}(\mathcal{O}_t)$ $\triangleright$ prompt embeddings, channels, integrated state 2: **for all** $m \in \mathcal{M}$ **do** $\triangleright$ channel decomposition (CD) 3: $e_t^{(m)} \leftarrow P_m(x_t^{(m)}) \in \mathbb{R}^d$ 4: **end for** 5: $q_t \leftarrow P_q(e_t^{(\text{all})})$ $\triangleright$ decision reference, recomputed every step 6: **for all** $m \in \mathcal{M}$ **do** $\triangleright$ step-adaptive prioritization (SP) 7: $s_t^{(m)} \leftarrow v^\top \tanh(W_e(e_t^{(m)} + r_m) + W_q q_t)$ 8: **end for** 9: $\alpha_t^{(m)} \leftarrow \frac{\exp(s_t^{(m)}/\tau)}{\sum_{j \in \mathcal{M}} \exp(s_t^{(j)}/\tau)}$, $c_t \leftarrow \sum_{m \in \mathcal{M}} \alpha_t^{(m)} e_t^{(m)}$ 10: $z_t \leftarrow P_s(c_t) \in \mathbb{R}^{d_z}$ $\triangleright$ action-sufficient projection (AP), $d_z < d$ 11: $R_t \leftarrow \text{RESHAPE}(W_p z_t + b_p) \in \mathbb{R}^{K \times d_{\text{model}}}$ $\triangleright$ $W_p, b_p$ zero-initialized $\Rightarrow R_t = 0$ at the start of training 12: $J_t \leftarrow \{b_t - K, \dots, b_t - 1\}$ $\triangleright$ $K$ prompt positions before the action boundary 13: $\tilde{E}_t \leftarrow E_t + \mathcal{S}_{J_t}(\gamma R_t)$ $\triangleright$ additive residual alignment (RA); $\mathcal{O}_t$ is neither pruned nor extended 14: **return** $\tilde{E}_t, \{\alpha_t^{(m)}\}_{m \in \mathcal{M}}$
Training and inference procedures (Algorithm 2)
The two case studies illustrate that the same prioritization loop handles both a purely auditory temporal task—where visual channels stay dormant until a sound cue—and a spatial‑semantic UI navigation task—where visual and textual channels dominate.
Theoretical analysis shows that the residual $R_t$ influences the decoder through two distinct pathways: (V) injects new content via the value vectors, altering *what* the decoder reads; (A) perturbs the attention scores, changing *where* the decoder looks. Early layers only affect attention (A), while deeper layers also contribute new values (V), making the additive residual essential for preserving evidence while steering attention.
Notation and Definitions
Provides symbol definitions and discusses scope and future directions.
**Table 1.** Notation summary.
FocusGUI is designed as a lightweight mechanism for improving how frozen omni‑modal GUI agents organize step‑dependent evidence, and it is complementary to advances in perception, grounding, and planning. Our current formulation computes channel priorities from the observation available at each interaction step, which keeps the method simple and model‑agnostic; extending this mechanism to incorporate anticipatory signals about which evidence may become important in subsequent steps is a natural direction for future work. Our experiments span four omni‑modal backbones and five interaction categories; extending focused perception to broader computer‑use settings is a natural next step.
Implementation Details
Implementation specifics: prompts, training targets, and configuration tables for FocusGUI.
The system prompt is fixed for all backbones and methods; it defines the task, the action vocabulary, and the required JSON output format. The action space follows OmniGUI and contains thirteen primitives, including three pointer actions, four swipe actions, a text entry, four navigation/termination actions, and NONE for observation steps. Coordinates are normalized to [0, 1000] and emitted as text through the frozen language‑model head, allowing FocusGUI to reuse the base model’s action vocabulary without discretizing the screen.
During training the ground‑truth action is serialized as a compact JSON string and appended only as the assistant target; loss is computed exclusively over these assistant tokens while all system and user tokens are masked. At inference the target is absent and the original LM head generates the action autoregressively using greedy decoding with beam size 1. Both raw backbones and FocusGUI receive identical prompts, observations, action histories, and decoding settings.
Step‑template pseudocode – ordering of multimodal blocks and residual injection.
We train on episode‑disjoint splits: 1 807 257 steps for training, 257 000 for validation, and 508 steps for testing, drawn from 495, 71, and 142 trajectories respectively. All feature transformations and normalization statistics are fitted on the training split only. FocusGUI adds only 1.445 M trainable parameters versus 8.932 B frozen Thinker parameters, i.e., 0.0162 % trainable proportion.
Table 5 compares a parameter‑matched concatenation baseline with the Dynamic prioritization variant (FocusGUI) on Qwen2.5‑Omni. Both configurations share the same trainable budget (≈1.44 M parameters). Dynamic prioritization improves Type Match by +0.98 %, Exact Match by +3.54 %, and Goal Progress by +3.44 %.
Table 6 lists the default FocusGUI implementation configuration: shared channel dimension $d=128$, action‑sufficient dimension $dz=64$, residual slots $K=4$, residual scale $\gamma=1.0$, temperature $\tau=1.0$, dropout 0.05, AdamW optimizer with learning rate $2\times10^{-4}$, weight decay $10^{-3}$, gradient clipping 1.0, and greedy decoding (beam 1). These hyper‑parameters are used for all experiments.
Table 7 details the inference inputs for the frozen backbone versus FocusGUI. All entries up to the residual window are identical (system prompt, multimodal observations, action history, action schema, decoding). The only difference is that FocusGUI receives the residual $E_t,J_t$ at the action boundary, while the frozen backbone does not.
**Figure 8** VITA-1.5 across interaction depth. Per-step Exact Match (left) and Type Match (right) VITA-1.5 is the weakest of the four backbones at 13.98 Overall TM, and the depth-dependent separation appears at that level as well.
**Figure 9.** Baichuan-Omni-1.5 across interaction depth. Per-step Exact Match (left) and Type Match (right). The same widening margin holds on a third backbone family, indicating that the decay is a property of the interaction regime rather than of a particular model.