Focus Like a Human: Efficient GUI Grounding via Coarse-to-Fine Visual Attention and Parallel Verification

1 Introduction

Large Visual Language Models enable recent agents to interact with Graphical User Interfaces (GUIs) in a human-like and intuitive manner [Zheng et al., 2024; Cheng et al., 2024; Gou et al., 2025; Wu et al., 2025c; Zhang et al., 2025b; Bai et al., 2025b]. While these agents have demonstrated impressive performance on synthetic or low-resolution benchmarks [Xu et al., 2025a; Toyama et al., 2021; Zhang et al., 2024], operating real-world, information-dense interfaces remains a challenge. GUIs of professional software are often populated with a large number of visually or functionally similar elements, posing a critical bottleneck to GUI grounding-the ability to accurately locate functional elements corresponding to natural language instructions [Li et al., 2025; Qian et al., 2024]. To accommodate diverse input resolutions and capture fine-grained details in these scenarios, prior works have proposed various solutions, including resizing to fixed scales, grid-based slicing [Bai et al., 2023; Li et al., 2024; Vasu et al., 2025; Wang et al., 2024b]. Increasingly, the prevailing paradigm has shifted toward processing inputs at native high resolutions, scaling to 4K and beyond. While this strategy benefits visual fidelity, this paradigm incurs excessive visual redundancy, leading to higher computational cost and attention dispersion. As shown in Figure 1, we observe that excessive resolution inputs cause attention dispersion, biasing the model towards trivial pixel-level variations and superficial details instead of the global spatial layout. Consequently, the model lacks the context to anchor its predictions, resulting in coordinate instability and ungrounded hallucinations, where bounding boxes appear in arbitrary locations detached from any actual UI elements. We term this phenomenon Semantic Fragmentation, where the excessive density of visual tokens dilutes the model's global receptive field, compromising its representation of the global semantic layout as it fixates on local pixel-level details. In contrast, we identify that low-resolution inputs function as effective Semantic Aggregators. By reducing visual token redundancy, downsampling serves as a form of structural abstraction, forcing the model to bypass superficial details and instead align with the global spatial layout.

This computational insight finds a profound theoretical parallel in cognitive neuroscience, specifically the coarse-to-

∗Co-corresponding authors.

Task: Install the github copilot extension.

Answer: [1511,331,1576,360]

(B) Low Resolution Attention (Concentration)

Figure 1: The Resolution-Attention Conflict in GUI Grounding. Contrary to the intuition that "higher is better", processing GUIs at native high resolution (Right) often leads to Semantic Fragmentation, where visual attention is distracted by details (e.g., text strokes, edges), losing the object's center. In contrast, FastFocus (Middle) leverages low-resolution inputs as a "Semantic Aggregator", functioning as a filter to form a cohesive, robust attention prior for precise localization.

fine processing dynamics of the human visual system [Schyns and Oliva, 1994; Hochstein and Ahissar, 2002]. To manage the immense bandwidth of visual information [Carrasco, 2011], the biological system employs an efficient division of labor: Peripheral Vision operates at low resolution to rapidly extract the global spatial layout and structural gist [Larson and Loschky, 2009], while Foveal Vision is serially deployed for the high-acuity identification of specific details [Bar, 2004]. Our finding that low-resolution tokens effectively encode global semantics aligns perfectly with the functional role of peripheral vision. Driven by this biological plausibility, we propose FastFocus, a coarse-to-fine framework that mimics this dual-process strategy of human visual system. Specifically, FastFocus comprises two synergistic modules: (1) Global Proposal Module: leveraging downsampled whole-image views to act as a high-recall semantic aggregator, filtering out background noise to identify potential regions of interest; and (2) Local Verification Module: processing these candidate regions at native resolution, utilizing high-fidelity visual features to perform fine-grained verification and precise localization. FastFocus allows the model to concentrate on fine-grained details within relevant semantic regions, effectively decoupling the semantic understanding from pixel-level localization. Extensive experiments on the ScreenSpot-Pro benchmark demonstrate that FastFocus establishes a new state-of-the-art GUI grounding accuracy. Analysis confirms that our method maintains robust focus in visually cluttered environments, effectively resolving the trade-off between grounding precision and inference efficiency. Our main contributions are summarized as follows:

• We identify "Semantic Fragmentation" as a primary failure mode in high-resolution GUI grounding and empirically demonstrate that low-resolution perception serves as a superior cognitive prior by effectively aggregating semantic features.
• We propose FastFocus, a biomimetic framework that decouples grounding into attention-based proposal generation and parallel saccadic verification. This design effectively breaks the serial latency bottleneck while mitigating the visual hallucinations common in direct regression paradigms.
• We validate FastFocus on the challenging ScreenSpotPro benchmark, where it achieves state-of-the-art and demonstrates exceptional robustness in high-density professional GUIs.

2 RelatedWork

2.1 GUI Agent

The development of GUI agents has transitioned from relying on structured metadata to leveraging direct visual perception. Early approaches predominantly utilized text-based reasoning, parsing HTML, DOM trees, or accessibility XMLs to interact with Web and mobile environments [Zhang et al., 2025b; Wang et al., 2024a; Yan et al., 2023]. While efficient for structured data, these methods struggle with "visualcentric" applications where accessibility metadata is often missing or incomplete. Recent advances in Multimodal Large Language Models (MLLMs) have enabled a shift towards vision-based agents that perceive raw pixels directly [Liu et al., 2023; Chen et al., 2024; Wang et al., 2024b; Bai et al., 2025b]. By bridging the visual perception gap, these agents can operate in open-ended desktop environments, achieving end-to-end visual grounding and action prediction [Cheng et al., 2024; Qin et al., 2025; Hong et al., 2024; Gou et al., 2025; Xu et al., 2025b]. However, capturing fine-grained details in dense interfaces remains challenging. To mitigate these issues, recent works have introduced domain-specific pretraining and modality-specific finetuning [Wu et al., 2025c; Sun et al., 2024], as well as reinforcement learning approaches to improve agent performance [Lu et al., 2025; Zhou et al., 2025; Liu et al., 2025]. Nevertheless, these methods often require extensive training data and computational resources. Unlike prior works that attempt to optimize the backbone itself, our framework focuses on a generic perception strategy to resolve the conflict between high-resolution accuracy and inference latency.

2.2 GUI Grounding

Precise element localization is the fundamental capability underpinning vision-based agents [Qian et al., 2024]. Existing methodologies can be broadly categorized into three paradigms:

Supervised Regression. The dominant paradigm involves fine-tuning VLMs on large-scale datasets to directly regress bounding box coordinates [Cheng et al., 2024; Wu et al., 2025c; Gou et al., 2025]. Models like SeeClick and OSAtlas [Wu et al., 2025c] have demonstrated strong performance by learning from massive interaction data. However, these regression-based methods often suffer from hallucinations in visually cluttered regions and lack interpretability. Furthermore, their performance is heavily dependent on the scale and quality of annotated data.

Visual Test-Time Scaling (TTS). To handle high-resolution details without extensive retraining, recent works have introduced test-time visual refinement strategies. Methods such as RegionFocus and Iterative Narrowing adopt a "crop-andzoom" mechanism, iteratively updating the model's visual focus based on history or initial predictions. While effective, these methods inherently suffer from a "Serial Accumulation Bottleneck". Since each refinement step requires a sequential forward pass, the inference latency scales linearly with the number of refinement iterations. In contrast, FastFocus employs a parallel verification mechanism, assessing multiple candidate regions simultaneously to achieve fine-grained precision without the latency penalty of serial interactions.

Visual Attention and Priors. Attention mechanisms have long been fundamental to visual understanding. Recent methods such as TAG [Xu et al., 2025a] and GUI-Actor [Wu et al., 2025b] avoid coordinate regression by leveraging internal attention signals from VLMs. However, they typically utilize attention primarily for coarse localization without analyzing failure causes under challenging conditions. Diverging from the convention of operating on high-resolution features-which we identify as susceptible to "Semantic Fragmentation"-we revisit the potential of low-resolution attention. We demonstrate that low-resolution inputs serve as a superior "Semantic Aggregator," providing a robust, trainingfree prior for proposal generation that aligns better with object-level layouts than high-resolution counterparts.

3 Methodology

In this section, we present FastFocus, a biomimetic coarseto-fine framework designed for efficient and precise GUI grounding. Unlike prevailing methods that rely on computationally expensive native-resolution processing, FastFocus mimics the efficient attention mechanism of the human eye.

The overall pipeline is illustrated in Figure 2.

3.1 Biological Plausibility

The architectural design of FastFocus is theoretically rooted in the Reverse Hierarchy Theory of visual perception [Hochstein and Ahissar, 2002]. Neuroscientific evidence indicates that the human visual system avoids processing the entire visual field with uniform high fidelity, a strategy that would otherwise overwhelm the optic nerve's transmission bandwidth [Carrasco, 2011]. Instead, it employs a resourceefficient dual-process strategy:

Peripheral Processing as Semantic Aggregation: The human peripheral visual field, despite its low acuity, is highly sensitive to spatial layout and global statistics (the "gist" of the scene) [Oliva and Torralba, 2006]. It acts as a rapid scanning mechanism that filters out high-frequency noise to identify potential regions of interest [Larson and Loschky, 2009]. In our framework, this maps to the Global Proposal Module, where low-resolution visual tokens serve as semantic aggregators to stabilize localization anchors.

Foveal Processing as Detail Verification: Once a salient region is detected, the eyes perform a saccade to bring that region into the fovea, which processes high-frequency details for precise identification [Bar, 2004]. This biological saccade parallels our Local Verification Module, which crops and processes high-resolution patches only where necessary.

Driven by this biological plausibility, we operationalize the Reverse Hierarchy Theory into the FastFocus framework. Specifically, we decouple the grounding process into two synergistic stages that mirror the human visual system: a Global Proposal Module acts as the peripheral processing to rapidly identify potential regions of interest from low-resolution inputs; and a Parallel Local Verification Module functions as the foveal processing, executing high-resolution saccades to resolve fine-grained details.

3.2 Global Proposal Module

ProblemFormulation. The primary objective of this stage is to serve as a Coarse-Grained Proposal Generator. Formally, given an image Inative ∈ RH×W×3 and a natural language instruction T , our goal is to generate a set of candidate regions P = {b1, b2, . . . , bK}, where each bk represents a potential target location. We prioritize recall over precision here, aiming to capture the target within P with a low computational cost.

Intrinsic Attention Extraction. We leverage the intrinsic cross-attention mechanism of the VLM as a spatial prior. As shown by Zhang et al. and Xu et al., the attention weights in intermediate transformer layers often exhibit a strong correlation with the model's focus on relevant image regions. Building on this insight, we resize Inative to a lower resolution Ilow and input it into the VLM. We extract the attention maps averaged across the intermediate layers (layers lstart to lend). Let A ∈ Rh×w denote the aggregated attention map, where each scalar Ai,j approximates the semantic relevance of patch (i, j) to the instruction T :

Algorithm 1 FastFocus Inference Process

Require: Native Image Inative, Instruction T, Number of Proposals K

Ensure: Target Bounding Box b∗

Stage I: Global Proposal Module

• 1: Ilow ←Resize(Inative, 448)
• 2: A←ExtractIntrinsicMap(Ilow, T)
• 3: P ← ∅
• 4: for k ← 1 to K do
• 5: ck ← argmax(i,j) Ai,j // Peak identification
• 6: P.append(GetProposalBox(ck))
• 7: A←A⊙ (1-SpatialMask(ck)) // Spatial suppression
• 8: end for

Stage II: Parallel Local Verification

• 9: C ←CropHighFidelity(Inative,P)
• 10: while |C| > 1 do
• 11: Bpairs ← FormPairs(C)
• 12: Windices ←VLMcompare(Bpairs, Tpair) // Parallel tournament round
• 13: C ← SelectWinners(C,Windices) // Filter survivors for next round
• 14: end while
• 15: Cwinner ← C[0] Stage III: Precise Localization
• 16: blocal ←VLMloc(Cwinner, T) // Regression on optimal patch
• 17: b∗ ←MapToGlobal(blocal,Cwinner)
• 18: return b∗

where L is the set of selected intermediate layers. This heatmap A serves as a dense probability distribution of the target's existence.

Greedy Proposal Generation. To select Region of Interest candidates from the attention map A, we design an iterative peak extraction strategy with spatial suppression. This ensures the model captures diverse salient regions without redundancy. We initialize the proposal set P = ∅. For k = 1 to K:

Peak Identification. We locate the global maximum in the current heatmap as the proposal center ck:

Proposal Extraction. We generate a bounding box bk centered at ck with a predefined anchor scale and project it back to the coordinate space of Inative.

Spatial Suppression. To prevent repeated selection of the same region, we apply a hard spatial mask around ck. The heatmap for the next iteration is updated as:

where r is the suppression radius and I(·) is the indicator function, which returns 0 if the coordinate lies within the radius and 1 otherwise.

This greedy process efficiently yields a ranked list of candidates P based on intrinsic semantic saliency, without requiring additional training or external bounding box supervision.

3.3 Parallel Local VerificationModule

Problem Formulation. The objective of this stage is to identify the precise target from the coarse candidates generated in the Global Proposal Module. Formally, taking the proposal set P and the native high-resolution image Inative as input, the module functions as a fine-grained discriminator to determine the optimal target b∗ and refine its coordinates.

High-Fidelity Cropping. We first extract high-resolution visual details to recover the information lost during the downsampling in Stage I. For each candidate bounding box bk ∈ P, we crop the corresponding region from the native image Inative to obtain a set of high-fidelity patches C = {C1, C2, . . . , CK}. These patches preserve the original pixel density required for reading small text and distinguishing fine icon details.

Recursive Pairwise Verification. To robustly identify the target among these patches, we adopt a Recursive Pairwise Verification strategy inspired by [Zhang et al., 2025a]. We formulate the selection process as a multi-round tournament rather than independent scoring, which forces the model to discriminate the target from distractors based on relative visual evidence.

In each round t, we pair the current candidates {Ci,Cj} and construct a comparison prompt Tpair. These pairs are stacked into a single tensor batch to be processed in parallel:

The VLM predicts the winner for each pair in a single forward pass. The winners form the candidate set for the next round, and this process recursively continues until a single optimal patch Ck∗ remains. This mechanism effectively suppresses hard negatives by leveraging the model's stronger capability in relative comparison versus absolute verification.

Precise Localization. Upon determining the winning candidate k∗ , the model performs a final coordinate regression on the patch Ck∗. The predicted local coordinates are then projected back to the global coordinate system of Inative to yield the final prediction b∗ .

4 Experiments

4.1 Experimental Setup

Datasets and baselines. Our primary evaluation utilizes ScreenSpot-Pro [Li et al., 2025], a benchmark comprised of high-resolution screenshots (larger than 1920 × 1080) from professional desktop applications (e.g., Blender, VS Code), which poses significant challenges for current agents due to element density and layout complexity. To assess generalization across platforms, we also report results on the ScreenSpot V2 benchmark. Baselines. We compare our method against three classes of models: (1) Generalist VLMs including GPT-4o, Claude Computer Use, Qwen2.5-VL and Qwen3-VL [OpenAI, 2024; Bai et al., 2025b; Bai et al., 2025a]; (2) Specialist Agents such as SeeClick [Cheng et al., 2024], OS-Atlas [Wu et al., 2025c], UGround [Gou et al., 2025] and UI-Venus [Gu et al., 2025], which leverage Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to align visual perception with actionable intent. These models typically undergo domain-specific training on large-scale interaction datasets to achieve precise element localization and robust task execution; and (3) Test-Time Scaling Methods like RegionFocus [Luo et al., 2025], DiMo-GUI [Wu et al., 2025a], and BAMI [Zhang et al., 2025a], which represent the current state-of-the-art in Test-Time visual refinement.

Implementation Details. We utilize UI-Venus-7B [Gu et al., 2025], UGround-V1-7B, and UI-TARS-1.5-7B as the backbonemodels. For the Global Proposal Module, input images are resized to 448×448. We extract the intrinsic semantic attention by averaging the cross-attention weights from layers 12 to 28 of the vision encoder. To generate diverse region proposals, we employ an iterative greedy sampling strategy with the spatial suppression radius set to 0.15× the image size, selecting the top K = 4 candidates. In the Parallel Local Verification Module, we extract high-resolution crops with a scale of 0.3× the original image dimensions, enforcing a minimum shortest edge of 448px, to preserve sufficient local context. These candidates are processed in a single parallel batch to ensure efficient verification.

4.2 Main Results

As shown in Table 1, we present a comprehensive evaluation on the ScreenSpot-Pro benchmark. Unlike static baselines, we assess FastFocus as a model-agnostic perception module applied to three distinct base agents: UGround-V1, UITARS-1.5, and UI-Venus-Ground. The results demonstrate consistent performance gains across all backbones, indicating that FastFocus is not backbone-specific and transfers effectively across diverse agents. Notably, applying FastFocus to UI-Venus-Ground-7B achieves a new state-of-the-art average accuracy of 61.8%, surpassing the previous best Test-Time Scaling method, BAMI-7B (56.2%), by a significant margin.

This confirms that our dual-stage strategy effectively complements existing SFT and RL-based optimization schemes without requiring model-specific tuning.

A granular analysis reveals a distinct performance divergence: while text grounding sees moderate gains, the improvement in Icon and Widget subsets is disproportionately significant (reaching up to +28.3% in Office/Icon for UIVenus). We attribute this to the differing perception granularity required. Text relies on explicit features, whereas icons depend heavily on regional layout semantics-the understanding of an element's role within its surrounding functional block (e.g., a toolbar or navigation panel). Standard high-resolution processing often suffers from attention dispersion, failing to grasp this coarse-grained block structure as it fixates on high-frequency pixel details. Our framework resolves this via the Global Proposal Module, which utilizes the low-resolution prior to aggregate visual information into coherent semantic regions. Subsequently, the Recursive Pairwise Verification in Stage II discriminatively refines these regional proposals, ensuring the final localization aligns with the intended functional module.

To ensure that our optimization for high-density professional GUIs does not compromise performance in standard scenarios, we further evaluated FastFocus on the general ScreenSpot V2 benchmark (Table 2). On this relatively simpler dataset, where the UI-Venus baseline already approaches performance saturation (94.1%), our method maintains robust accuracy (94.6%). Notably, we observe a consistent trend where Icon grounding continues to improve (+1.3% to +1.6%), validating that our Global Proposal Module effectively generalizes across varying levels of layout complexity without introducing regression in simpler tasks.

4.3 Analysis of Resolution Priors

To quantitatively validate our qualitative observation regarding the conflict between resolution and attention in Section 1, we evaluated the Proposal Recall@4 of the UI-VenusGround-7B model across input image edge scales ranging from 224 to 4096. As illustrated in Figure 3, the results exhibit a distinct Inverted-U trajectory, peaking at 448. This empirical evidence strongly corroborates the inherent conflict between resolution and attention cohesion:

• Validation of Semantic Fragmentation: Performance degrades significantly at native high resolutions. This confirms that while high resolution is necessary for details, excessive visual token density leads to attention dispersion, preventing the model from effectively locking onto the object's center during the proposal phase.
• Optimal Structural Abstraction: The peak at 448 validates that moderate downsampling serves as an effective Structural Abstractor. It suppresses high-frequency noise while preserving the global layout, verifying our observation that lower resolutions yield more cohesive attention patterns for initial proposal generation.

4.4 Ablation Analysis

To validate the rationale behind our architectural decisions, we conducted a component-wise ablation study on the ScreenSpot-Pro benchmark, as detailed in Table 3.

Impact of Proposal Strategy (Rows a-c). We first investigate the optimal strategy for generating object proposals.

• Intrinsic Prior vs. External Detectors: Utilizing an external object detector (Grounding DINO, Row a) yields suboptimal performance (41.2%). This confirms the existence of a Semantic Gap: generic detectors are agnostic to user intent, generating geometrically valid but semantically irrelevant proposals. In contrast, our VLM-based intrinsic attention naturally aligns visual features with linguistic instructions.
• Explicit Attention vs. Direct Regression: While standard Direct Regression (Row b) and Native Resolution processing (Row c) achieve moderate baselines (∼54%), they hit a performance ceiling. We attribute this to the Resolution-Attention Conflict discussed in Sec. 1: native resolution processing forces the model to handle excessive token density, leading to attention dispersion. Our Coarse-to-Fine approach (Row e, 61.8%) outperforms these single-stage baselines by decoupling structural localization from fine-grained identification.

Necessity of Coarse-to-Fine Synergy (Rows d-e). The most critical insight stems from the interaction between Stage I and Stage II.

• The Power of Verification: Using the Global Proposal Module alone (Row d, w/o Verification) achieves 48.5% accuracy. This configuration corresponds to setting K = 1, where the model directly performs coordinate regression on the top-ranked proposal derived from the intrinsic attention map, bypassing the pairwise comparison entirely. While this is lower than the full model, it serves as a high-recall Structural Abstractor, effectively narrowing the search space from millions of pixels

to a few salient regions. Integrating the Parallel Verification Module (Row e) yields a massive +13.3% performance jump (48.5%→61.8%). This empirical evidence validates our core hypothesis: low-resolution priors are necessary to capture global layout semantics, but insufficient for precise localization. The Recursive Pairwise Verification in Stage II is indispensable for resolving the visual ambiguity among the candidate crops, providing the discriminative power needed to distinguish the target from subtle distractors.

5 Conclusion

In this work, we critically examine the efficacy of native resolution processing in GUI grounding, uncovering the Resolution-Attention Conflict where excessive visual density leads to Semantic Fragmentation. We propose FastFocus, a biomimetic perception framework that resolves this dilemma by decoupling grounding into coarse-grained structural abstraction and fine-grained recursive verification.

Our empirical findings lead to three key conclusions: (1) Structure Precedes Detail: Low-resolution inputs act as superior Structural Abstractors, effectively aggregating semantic layout information where native high-resolution baselines often succumb to attention dispersion. (2) Discrimination over Regression: The success of our Recursive Pairwise Verification demonstrates that discriminative tournament selection offers significantly higher robustness against visual ambiguity compared to standard coordinate regression, particularly for abstract icons. (3) Consistency Across Evaluated Backbones: Validated across diverse SFT and RL baselines, FastFocus proves that a cognitive-inspired "System 2" perception module can consistently improve grounding across the evaluated backbones without backbone retraining.

By establishing new state-of-the-art results on the ScreenSpot-Pro benchmark, FastFocus demonstrates that precise grounding arises not from brute-force resolution scaling, but from the intelligent organization of visual attention. We hope this work inspires future research to move toward such hierarchical perception strategies for more robust autonomous agents.

Acknowledgments

Xiaocheng Feng and Bing Qin are the co-corresponding authors of this work. We thank the anonymous reviewers for their insightful comments. This work was supported by the National Natural Science Foundation of China (NSFC) under Grants 62522603 and 62276078, the Key R&D Program of Heilongjiang under Grant 2022ZX01A32, and the Fundamental Research Funds for the Central Universities under Grant XNJKKGYDJ2024013.