TextGaze: Prompting Gaze Target Estimation with Textual Scene Cues

1 Introduction

Gaze target estimation [Recasens et al., 2017; Chong et al., 2018; Miao et al., 2023; Ryan et al., 2025; Liu et al., 2024] is a task in computer vision that aims to predict a person's gaze direction in a scene or determine if the gaze target lies within the image. As human gaze is a fundamental nonverbal cue reflecting attention allocation, intent, and cognitive state [Eckstein et al., 2017], it is essential for human behavior understanding [Li et al., 2023b; Li et al., 2025; Li et al., 2026; Wang et al., 2024b], with broad applications in human-computer interaction [Katsini et al., 2020; Wang et al., 2026; Chen et al., 2025], extended reality [Sitzmann et al., 2018], and education [Wang et al., 2025].

∗Corresponding authors

In multi-branch methods, the classical architecture is the dual-stream appearance-based design [Chong et al., 2018; Lian et al., 2018; Chong et al., 2020; Wang et al., 2024a]. In these designs, the head stream localizes the subject, and the scene stream encodes contextual visual information. Because purely visual information is insufficient to enhance effects, this paradigm has evolved into multi-branch frameworks that incorporate auxiliary cues. These include head features [Chen et al., 2021], scene context [Saran et al., 2018], depth [Bao et al., 2022], and pose [Gupta et al., 2022].

Although these designs enrich visual representations, they introduced the slow convergence problem [Ryan et al., 2025] as adding an extra branch requires additional task-specific supervision beyond gaze annotations. As the number of branches increases, it not only increases the annotation burden and subjectivity but also complicates deployment and hinders domain transfer. More fundamentally, these systems function as post-processing geometric filters. They are capable of eliminating illogical regions but cannot reason about which objects constitute valid gaze targets. Consequently, the problem of semantic under-specification remains unresolved.

To simplify such complex architectures, recent work has introduced prompt-tuning methods based on foundation models. Gazelle [Ryan et al., 2025] replaces person-specific scene re-encoding and handcrafted fusion with a unified decoder conditioned on a head position prompt, thereby addressing the need to integrate head information with global scene cues while avoiding multi-branch designs. This strategy substantially reduces architectural complexity and improves convergence behavior. However, the performance of such methods is heavily influenced by the base scene encoder and remains constrained by geometric conditions. Head position prompts also fail to provide effective information regarding task semantics. Therefore, the reasoning process is still largely influenced by low-level appearance heuristics (such as spatial proximity and visual salience), leading to semantic defocus in Figure 1, where predicted attention contradicts the subject's actual gaze intention.

Gaze following is inherently an intent-level reasoning problem rather than a purely geometric regression task. Cognitive studies [Mayrand et al., 2024] show that human gaze integrates spatial cues with semantic judgments of meaningful targets and mental-state reasoning about target visibility. Yet existing models emphasize geometric signals and largely neglect task-level semantics, highlighting the need for accessible semantic priors that can guide gaze inference without reintroducing multi-branch complexity.

With the rise of large language models and LVLMs [Bai et al., 2025; Wang et al., 2024c], the capabilities of LVLMs are being recognized across various fields. LVLMs offer robust zero-shot capabilities for extracting contextual visual cues [Gupta et al., 2024] and alleviate vision-dominated semantic underspecification by framing gaze estimation as a top-down, semantics-guided attention task. Compared to specialized, multi-branch architectures, LVLMs offer a more scalable and streamlined approach to acquiring auxiliary visual information. Their text tokens encode abstract concepts, such as objects and actions, serving as differentiable, highlevel priors [Bi et al., 2025]. These priors are guided by carefully designed visual prompts that specify task objectives. They constrain the solution space, improve adaptation to complex scenes, and eliminate the need for redundant branches and extra annotations. This mitigates the imbalance in the multi-branch semantic compensation cost.

Building on these insights, we introduce TextGaze, a unified trainable decoder that integrates LVLMs, scene encoding, and visual-text fusion. The model adopts a dual-stream design, shifting gaze estimation from direct geometric regression (pixels to gaze coordinates) to an intent-recognition process via a semantic layer (pixels to semantic intent to gaze target). One stream uses an LVLM with a BERT encoder to generate textual descriptions of subject behavior and plausible targets, yielding semantically enriched visual-text embeddings; the other extracts fine-grained scene cues via a frozen scene encoder and utilizes positional head prompting to enhance character positioning and facial cues. These complementary streams are fused by a Visual-Text Fusion Transformer for context-aware, cross-modal gaze estimation. To enhance model capability and robustness, we devise multiple lightweight and effective optimization strategies. Based on our design implementations, TextGaze strikes a balance between multi-branch methods and streamlined designs. And TextGaze achieves the best performance on the majority of evaluation metrics, validating the effectiveness of our fusion strategy.

In summary, the main contributions of this paper are:

• We introduce textual scene cues into gaze target estimation, providing auxiliary information to mitigate the issue of semantic defocus in the gaze decoder.
• We strengthen the GazeFollow and VideoAttentionTarget benchmarks by integrating scene-target-person attention fields with LVLM-oriented prompts; this approach generates robust contextual cues that effectively improve the accuracy of text-aided gaze estimation.
• We design a unified dual-stream cross-modal fusion framework that combines textual embeddings with finegrained visual representations, improving spatial reasoning, scene adaptability, and mimicking human top-down attention.

2 RelatedWork

2.1 Gaze Target Estimation

Gaze target estimation [Recasens et al., 2015; Recasens et al., 2017; Lin et al., 2025; Liu et al., 2024; Ryan et al., 2025] aims to predict the location a person is looking at in a scene image [Recasens et al., 2015] or video frame [Recasens et al., 2017]. The straightforward pipeline is the dualstream approach, where one stream focuses on scene understanding and the other learns head features from cropped regions via off-the-shelf detectors [Lin et al., 2025]. This design addresses the core requirements of locating individuals and learning visual scene features, but falls short of outstanding performance due to its sole reliance on visual appearance. Subsequently, multi-branch architectures are proposed to capture diverse auxiliary cues (e.g., facial or head features [Chen et al., 2021], scene context [Saran et al., 2018], depth structures [Bao et al., 2022], pose information [Gupta et al., 2022], and eye [Fang et al., 2021]), with multi-modal fusion becoming key for performance gains. However, complex multibranch designs suffer from diminishing returns and high data acquisition costs, driving efforts to streamline architectures. Recently, the prompt-tuning method Gaze-LLE [Ryan et al., 2025] proposed to leverage visual foundation models for gaze prediction via positional head prompt. However, it still faces challenges in explicit visual modeling and task-relevant semantic focus. To address these issues, we propose TextGaze, a framework that leverages accessible, interpretable auxiliary semantic cues, enabling high-performance gaze target estimation through effective cross-modal fusion.

2.2 Large Vision Language Models

Large Vision Language Models (LVLMs) [Radford et al., 2021; Li et al., 2023a; Yang et al., 2023] have become a central paradigm for multimodal visual understanding [Li et al., 2023c; Li et al., 2023d]. Models such as CLIP [Radford et al., 2021], Claude [Priyanshu et al., 2024], and GPT-4V [Yang et al., 2023] exhibit strong multimodal reasoning capabilities and achieve competitive or superior performance to classical vision models in zero-shot classification. Their architectures increasingly rely on pre-trained large language models to align visual and textual representations [Grattafiori et al., 2024], supported by components such as vision encoders, text encoders, decoders, and cross-attention modules. Recent advances further strengthen cross-modal fusion by unifying modalities into tokenized streams [Wang et al., 2024c] or employing transfusive mechanisms [Zhou et al., 2024]. In our work, LVLMs are used not as task solvers but as generators of structured visual auxiliary information for gaze target estimation. The LVLM-derived descriptions are embedded together with images into a joint representation space. A fused transformer with hierarchical fine-grained supervision is then applied to integrate these signals. This design significantly enhances cross-modal fusion and yields strong performance on gaze target estimation benchmarks.

3 Methodology

3.1 ProblemFormulation and Notation

Gaze target estimation [Recasens et al., 2017; Ryan et al., 2025] aims to predict a heatmap that indicates the per-pixel probability of the gaze target. Formally, given an RGB image Iimg ∈ RHin×Win×3 and the bounding box of the target person's head xbbox, represented by length-4 coordinates, the model predicts a 2D gaze heatmap Iheat ∈ [0, 1]H×W . If an in-/out-of-frame estimation task (e.g., VideoAttentionTarget and ChildPlay benchmarks) is included, the model should predict a scalar probability py ∈ [0, 1].

3.2 Model Architecture

An architecture overview of the proposed method is shown in Figure 2. It consists of three core components: Textual Scene Cues (§3.3), which extracts attention-aware textual cues from a frozen LVLM. Feature token sequence processing (§3.4), responsible for enhancing visual features, embedding positional information, and constructing token sequences for fusion. Cross-modal fusion and prediction (§3.5), which integrates the visual and textual sequences through fusion transformer blocks and produces a heatmap and in/out predictions. Below, we detail each component.

3.3 LVLM-based Textual Scene Cues Module

Obtaining semantically rich textual cues that align with visual focus is a key component of our design. We incorporate a frozen LVLM L into our design to accomplish this task. Specifically, we adopt Qwen2.5-VL-7B-Instruct in our experiments, while the framework is backbone-agnostic and compatible with any multimodal model supporting joint imagetext prompting. Leveraging the zero-shot contextual reasoning and visual grounding ability of modern LVLMs, we design task-specific prompts to guide L in producing structured descriptions of the subject's appearance, behavior, and coarse gaze orientation. Instead of directly localizing gaze targets, which is unreliable in out-of-frame cases, this design generates context-aware auxiliary cues that remain effective across diverse scenarios.

The natural language outputs of L are encoded by a frozen text encoder, BERT-base-uncased in our implementation, into dense embeddings St ∈ RTt×dt , where Tt denotes the sequence length and dt the embedding dimension. These embeddings bridge high-level linguistic context and low-level visual features, enabling effective cross-modal fusion. Overall, the prompt design and LVLM-based cues module provide a flexible and robust solution for gaze target estimation in complex settings.

3.4 Feature Token Sequence ProcessingModule

This module prepares text and visual features for bimodal fusion. For text processing, we augment the text feature sequence Ft ∈ RLt×d with learnable positional embeddings PEtext ∈ Rd , obtaining Zt = Ft +PEtext. For visual feature processing, we use a frozen DINOv2 encoder to extract a visual feature map Fv ∈ RH×W×d . To enhance sensitivity to the target person's head, we incorporate head-region semantic cues using a learnable head token embedding. Given the head bounding box, we generate a downsampled binary mask M ∈ {0, 1}H×W for the head region. We then add a learnable head embedding Ehead to the visual features via masked modulation: Zv = Fv +M · Ehead, where · denotes broadcasted element-wise multiplication. The resulting feature map is flattened into a sequence Sv, and absolute 2D sinusoidal positional embeddings PEv (fixed for the visual feature size) are added to form Sve = Sv + PEv.

For fusion, we concatenate the text and visual sequences to construct the bimodal fusion sequence Sf = Concat(Ste,Sve). The concatenation is operated along the token dimension. We record index boundaries to extract the visual segment for subsequent prediction. For the in-/out-offrame classification task, a learnable token zin/out ∈ Rd is prepended, yielding Zjoint ∈ R(1+HW+Lt)×d :

3.5 Cross-Modal Fusion and PredictionModule

During the design of cross-modal fusion modules, we observed inherent challenges in vision-text fusion based on transformers. Such models lack explicit hierarchical constraints, leading to an excessive focus on global token interactions, resulting in representation drift across layers [Clark et al., 2019; Raganato and Tiedemann, 2018]. In visionlanguage settings, this manifests as token-level misalignment between modalities [Lu et al., 2019]. During deep fusion, gaze-related textual semantics are gradually diluted. In the absence of intermediate supervision, this discrepancy arising from visual gaze cues and the a priori textual intent accumulates layer by layer, ultimately leading to misalignment.

To address these issues, we design a three-layer stacked fusion transformer with post-normalization. It is tailored for unified attention construction, global cross-modal interaction, and hierarchical text semantic supervision, enabling precise vision-text alignment while retaining task-critical text semantics.

The query Q and key K are derived by adding positional embeddings to the input sequence, while the value V retains the original features:

Multi-head self-attention (MHSA) follows the standard scaled dot-product formulation. Only padding masks are applied to remove invalid tokens, allowing unrestricted interaction between text and visual tokens.

Each fusion block contains two sublayers: MHSA and a feed-forward network (FFN), both equipped with residual connections and layer normalization. The MHSA sublayer output is:

followed by the FFN sublayer:

To safeguard task-relevant textual semantics from degradation during cross-modal fusion, we propose a hierarchical text feature consistency supervision mechanism that enforces semantic alignment across all transformer layers. For each layer l ∈ {1, 2, . . . ,L}, the text segment of Xl is extracted and aligned with pre-computed text encoder features.

Let Xl text ∈ RLt×d denote the text token subset, where Lt is the number of text tokens. After permuting to Lt × d, it is projected to the text hidden dimension dtext via a layerspecific linear projection (weight matrixWl ∈ Rdtext×d , bias term bl ∈ Rdtext ):

where ˆ Sl t ∈ RLt×dtext are the reconstructed text features.

The layer-wise text supervision loss is the mean squared error between ˆ Sl t and target text features St:

where N = Lt · dtext is the total number of elements after vectorizing the text features. The hierarchical text loss is:

Following the fusion module, the visual token segment is reshaped into F ∈ RH×W×d , where H and W denote spatial dimensions. A two-layer convolutional decoder outputs the gaze probability heatmap Iheat, supervised by pixel-wise BCE loss with Gaussian targets (σ=3). For in-/out-of-frame classification, the task token zin/out is fed to a two-layer MLP Din/out to predict a Bernoulli probability.

The overall objective is:

where λ1 and λ2 balance the auxiliary losses.

4 Experiments

4.1 Experimental Setup

Datasets. We evaluate our method on four benchmark datasets: GazeFollow [Recasens et al., 2015], VideoAttentionTarget [Chong et al., 2020], ChildPlay [Tafasca et al., 2023], and GOO-Real [Tomas et al., 2021]. GazeFollow contains 117,727 training images with 124,095 annotated target persons and 4,782 test images. VideoAttentionTarget consists of 58,507 training frames with 132,563 target persons and 13,127 test frames with 31,978 target persons. ChildPlay and GOO-Real are used to assess generalization without finetuning, focusing on children's gaze behaviors and everyday shopping scenarios, respectively. To construct visual descriptive cues, we employ prompt engineering to guide Qwen2.5VL-7B [Bai et al., 2025] to generate appearance attributes for each image. Specifically, these attributes include (1) appearance descriptions of the target person for multi-person disambiguation; (2) action descriptions to better capture gaze behavior; and (3) gaze direction estimates without naming the target object, avoiding hallucination when the target is out of frame. Detailed implementation is given in the appendix of the supplementary material.

Evaluation Protocols. Following existing gaze target estimation settings [Chong et al., 2018; Tonini et al., 2022; Miao et al., 2023; Ryan et al., 2025], we use heatmap AUC and pixel-wise L2 distance as the primary metrics. For heatmap AUC, each pixel in the predicted heatmap is treated as a confidence score to compute the ROC curve. The pixelwise L2 distance is defined as the Euclidean distance between the argmax location of the predicted heatmap and the groundtruth gaze target. For the GazeFollow dataset, as it has multiple annotation points, we compute the distance to the mean of annotations (Avg L2) and the distance to the nearest annotation (Min L2). For the VideoAttentionTarget and ChildPlay datasets, we also evaluate the Average Precision (AP) for the in-/out-of-frame prediction task to assess whether the gaze target lies within the visual field.

Implementation Details. The input image is resized to 448×448, while the predicted gaze heatmap has a resolution of 64×64. The scene encoder uses frozen DINOv2 [Oquab et al., 2023] optimized by AdamW [Loshchilov and Hutter, 2017] optimizer with the initial learning rate of 1e-4 and weight decay of 1e-2, including ViT-B and ViT-L versions. The text encoder utilizes a frozen BERT-base-uncased model. Default loss weights are λ1 = 1 and λ2 = 0.1.

4.2 Performance Comparison

Comparison to State-of-the-Art. Our method's performance on the GazeFollow and VideoAttentionTarget datasets compared with existing leading approaches is summarized in Table 1. It is easy to notice that our model differs frommost prior methods, which rely on additional modalities such as depth or pose; instead, we take image and text as inputs. This design originates from our thinking on balancing the acquisition of auxiliary cues and architectural complexity.

On GazeFollow, our method achieves competitive results across all metrics. With the DINOv2 ViT-L backbone, we obtain the lowest average L2 error and match the best minimum L2, while delivering a high AUC that is comparable to the state-of-the-art. Even with the lighter ViT-B backbone, our model maintains top-tier performance on key metrics. These results validate our hypothesis that the text-guided semantic priors provided by LVLMs effectively supply auxiliary cues, thereby enhancing the performance of gaze target estimation models. On VideoAttentionTarget, our DINOv2 ViT-L variant outperforms existing approaches across all three evaluation metrics (AUC, L2 error, and APin/out), demonstrating the value of integrating text cues for reliable gaze estimation. Our method particularly excels in the in-/out-of-frame task, highlighting its ability to determine whether a target lies within the field of view. This conclusion is further validated by crossdataset comparison results.

It should be noted that fair comparisons with the state-ofthe-art method by Ryan et al. [Ryan et al., 2025] are conducted under matched experimental settings using DINOv2 ViT-B and DINOv2 ViT-L backbones, with their results reproduced from official code. Qualitative examples under the DINOv2 ViT-B configuration are provided in Section 4.4 to further illustrate our approach's effectiveness.

Cross-dataset Results. To evaluate the generalization capability of our approach, we conduct cross-dataset evaluation on the GOO-Real and ChildPlay datasets without modifying LVLM prompts or performing training, fine-tuning, or data augmentation. ❶ The test scenarios in the GOO-Real dataset involve gaze targets within the field of view. Table 2 compares our results with existing methods, where our approach achieves state-of-the-art performance on both AUC and L2 metrics, demonstrating strong generalization in real-world scenarios. ❷ The ChildPlay dataset includes predictions of gaze target presence within the visual field of view. Table 3 compares our results with existing methods: although our approach slightly lags behind the state of the art in AUC and L2, it outperforms all baselines on the AP metric. Combined with results from the VideoAttentionTarget, this further highlights the contribution of LVLM-derived visual descriptive cues to AP (in/out) classification. Figure 3 presents qualitative results across four datasets: GazeFollow (trained with DINOv2 ViT- B), VideoAttentionTarget (fine-tuned), and ChildPlay/GOOReal (without fine-tuning). The left panel shows raw inputs, while the right overlays predicted heatmaps.

4.3 Ablation Study

Ablation Results of Bimodal Fusion. We explore multiple representative fusion approaches that align with our design principles to formulate our fusion strategy: additive fusion, cross-attention fusion, and the transformer-based fusion adopted in our model. Their schematic structures are illustrated in Fig. 4. Additive fusion enforces element-wise alignment between modalities and thus struggles to handle heterogeneous visual-textual representations. By comparison, cross-attention mechanisms enable directional interactions, yet remain constrained by asymmetric information flow and lack holistic cross-modal reasoning capabilities. The transformer-based fusion performs unified attention across all visual and textual tokens, allows globally symmetric interactions, and progressively optimizes gaze-related semantic information. The relevant experimental results in Table 4 demonstrate the effectiveness of this fusion strategy within our design. The ablation studies in this section are all conducted using the GazeFollow dataset. All tested models use the same DINOv2 ViT-B scene encoder and textual cues data. Ablation Results of Key Components. Building upon the established fusion strategy, we design key components to enhance models' performance in gaze target estimation tasks. We first focus on the critical semantic guidance capability of textual cues within tasks, noting that gaze-related textual

(a) Additive

(b) Cross-Attn FFN

Add & Norm

(c) Transformer-based

Figure 4: Schematic of fusion strategy under ablation experiments. The grey and yellow blocks represent visual tokens and textual tokens, respectively.

0.056

0.055

0.047

cues may undergo semantic drift during multi-layer fusion processes. Consequently, we design an inter-layer text supervision mechanism as the foundational component and test its efficacy. We also wish to achieve spatial fidelity preservation concisely. So, we introduce pre-prediction visual residual connections to preserve the fine-scale spatial details crucial for gaze localization. Regarding positional embedding strategies for feature sequences, we attempt to incorporate explicit spatial inductive biases to model gaze-specific localization patterns. We test two approaches: modality-specific positional embedding and learnable full-sequence positional embedding. The former utilizes 2D sinusoidal positional embedding for visual features and learnable positional embedding for textual features; the latter applies learnable positional embedding to all features. Each component we designed effectively enhances the overall performance of the architecture, with relevant results shown in Table 5. Stepwise optimization experiments further validate the rationality and effectiveness of our design approach. The results further indicate that simultaneously activating these three components yields optimal performance, confirming their complementary roles in our framework design for the gaze target estimation task. We ultimately adopted the simultaneous activation scheme for our architectural implementation.

4.4 Qualitative Results

Figure 5 presents three representative scenarios from GazeFollow that illustrate the qualitative advantages of TextGaze. In the scene depicted in the first row of the image, multiple targets and ambiguous candidate regions are present. Purely vision-guided methods are disrupted by visually salient areas due to the absence of explicit task-oriented semantics. By incorporating textual cues from LVLM to assign subject identity and gaze intent, our model is able to avoid misjudgments. In the second typical scenario, facial cues are rendered unreliable by occlusion or low resolution. Consequently, low-level visual cues are insufficient to support the precise predictions of purely vision-guided methods. However, our approach remains reliable through LVLM-based behavioural descriptions that compensate for missing details with high-level semantic priors. Finally, the target was misclassified as being within the bounding box, a frequent error arising from its reliance on spatial proximity. Our approach prevents such errors by integrating scene context with semantic constraints. These cases validate the efficacy of our approach. Implementations that rely excessively on purely bottom-up visual understanding lack task-oriented semantics, resulting in semantic defocusing. By integrating prior knowledge guided by LVLM, we are able to shift gaze estimation from geometric regression towards a semantically driven decision process. This enables us to achieve accurate localization in complex scenes.

5 Conclusion

We address two issues in gaze target estimation: multibranch designs require extensive supervision and annotations, causing cost imbalance and slow convergence; streamlined designs over-rely on low-level visual saliency rather than true gaze intent, leading to semantic defocus. We propose TextGaze, a unified cross-modal framework guided by an LVLM. Exploiting zero-shot capability, we produce text prompts and fuse them with frozen visual features through a dual-stream structure, enabling effective fusion while keeping the pipeline simple. We augment four datasets with text cues for future research. Experiments show strong performance and generalization, verifying our design and demonstrating LVLMs' ability to connect vision with high-level intentions.

Acknowledgements

This work was supported by the Key Science & Technology Project of Anhui Province (202304a05020068), the Yangtze River Delta Science and Technology Innovation Community Joint Research (Basic Research) Project under Grant (2025CSJZN01600), and the New Generation of Artificial Intelligence National Science and Technology Major Project (2025ZD0123303).