Denoising Diffusion Models are Good General Gaze Feature Learners

1 Introduction

Gaze estimation is the task of measuring the gaze direction of a human in an image. It has been widely applied in human-computer interaction [Zhang et al., 2017a; Sugano et al., 2016; Park et al., 2021], augmented reality [Padmanaban et al., 2017] and driver monitoring system [Mavely et al., 2017]. With the rapid development of deep learning technologies, appearance-based gaze estimations have achieved remarkable breakthroughs and promising performances.

Although deep learning methods have shown promising results for gaze estimation. Training a well-performed model needs sufficiently large and diverse labeled data, covering a wide range of gaze directions, appearances, and head poses, which is very laborious and time-consuming. The limited labeled data hinders the development of gaze estimation methods. When trained on only a small amount of annotated samples, supervised learning methods are easy to overfit the training data, and their performance significantly degrades when encounter a new scenario with different data distribution. Therefore, methods which facilitate training with limited gaze annotations are highly desirable.

Self-supervised learning (SSL) has proven successful at learning generalizable features by leveraging large-scale available unlabeled data. Some researchers use the SSL techniques (e.g. contrastive learning [Du et al., 2023; Jindal and Manduchi, 2023] and cross-encoder disentangling [Sun et al., 2021]) to tackle the issue of data scarcity in the field of gaze estimation, and have achieved promising results. However, most of them are not generative pre-training methods. Recently, we have witnessed remarkable progress at the generative domain, especially with diffusion modeling proving to be a powerful technique which can create vivid imagery of astonishing realism. Following the idea, which considers the ability to create is among the highest manifestations of learning, more recently, a few researchers [Chen et al., 2024; Xiang et al., 2023] start to explore the representational capacity of diffusion models, and show promising results. However, its representational capacity for gaze estimation is still unexplored.

To verify whether denoising diffusion models are good general gaze feature learners, in this paper, we propose an unsupervised gaze representation learning framework based on diffusion modeling as shown in Figure 1. Our intuition is that if the network learns how to generate target images with gaze direction corresponding to the reference images, then it has learned to extract gaze-related features from given reference images. Specifically, we utilize a conditional diffusion model to generate target image with gaze direction specified by the reference image as the pre-training task. The reference image goes through a feature extractor and gets a latent feature as the condition. If we utilize the target image as the reference image, then it behaves more like an autoencoder, without focusing on extracting gaze-related features, which goes against our original intention. To make the extractor learn meaningful features for gaze direction control, we propose a disentangling feature learning strategy. Considering gaze direction is mainly influenced by head pose and eye direction (as illustrated in supplementary material Figure 1), a image with same identity as the target image but with different gaze directions is used as input to extract the appearance feature. The eye region of the target image is used as input to extract the eye direction feature, and the target image with Gaussian blur augmentation is used as the input to extract the head pose feature. Finally these features are combined as the conditional features to guide the target image generation.

In summary, the contributions of our work are as follows:

• We are the first to propose a self-supervised learning framework based on diffusion modeling for gaze estimation. It leverages the powerful generative ability of diffusion models to enhance the gaze representation learning.
• To facilitate the diffusion model to learn gaze related features as condition to control the gaze direction of generated image accurately, we design a disentangling feature learning strategy. It can learn appearance feature, head pose feature, and eye direction feature respectively, and combine them as the conditional features.
• Our method achieves the state-of-the-art performance in extensive evaluation settings, which demonstrates that, denoising diffusion models are also good general gaze feature learners.

2 RelatedWork

Appearance-based Gaze Estimation

Result from the availability of large-scale datasets [Zhang et al., 2020; Park et al., 2020; Krafka et al., 2016] and dramatic improvements in computing power, appearance-based gaze estimation methods have received widespread attention and achieved remarkable performance. However, these methods still suffer from obvious performance deterioration on cross-dataset evaluation, owing to the data distribution shifts. The common approach is to learn a more generalizable gaze representation. Such as, PureGaze [Cheng and Bao, 2022] and [Xu et al., 2023] improve robustness to unfavorable interference through adversarial learning. CLIP-Gaze [Yin et al., 2024c] and LG-Gaze [Yin et al., 2025] exploit visual language models to strengthen feature representation. GLA [Zeng et al., 2025] further eliminates label distribution shifts. And [Wang and Yin, 2025; Yin et al., 2024b] aligns feature distribution at test time to improve target domain performance. But these methods are still limited by the diversity of training datasets. In addition, [Ruzzi et al., 2023; Yin et al., 2024a] generates data with gaze annotations to improve sample diversity, but the realness of generated data cannot be guaranteed. There are still certain risk in transfering to unseed domains. So that, fully utilizing massive amounts of unlabeled data for self-supervised learning has gradually become popular.

Gaze Self-supervised Learning

Contrastive learning is widely popular due to its superior performance in self-supervised learning. It maximizes mutual information between latent representations to promote the discrimination ability of feature extractor [Bachman et al., 2019]. Inspired by this idea, GazeCLR [Jindal and Manduchi, 2023] and ConGaze [Du et al., 2023] construct various contrastive tasks to learn effective gaze representation. On the other hand, [Yu and Odobez, 2020] learns low-dimensional gaze representation by gaze redirection, [Sun et al., 2021] proposes latent-code-swapping mechanism to decouple eye features and gaze features. Furthermore, MV-DE [Bao and Lu, 2024] draws upon multi-view constraints and designs a multi-view gaze representation swapping strategy to derive gaze-related feature. Nevertheless, most of them are not generative pre-training methods. Following the idea, which considers the ability to create is among the highest manifestations of learning. A few researchers start to explore the representational capacity of diffusion models, and show promising results. However, its representational capacity for gaze estimation is still unexplored.

Denoising Diffusion Models as Feature Learners

Since the advent of denoising diffusion model, marked breakthroughs have been made in numerous tasks and modalities. Based on its powerful generative capacity, some works [Clark and Jaini, 2024] explored the representation ability of diffusion model. l-DAE [Chen et al., 2024], DDAE [Xiang et al., 2023] and DiffMAE [Wei et al., 2023] auto encode the same image to derive visual representation capability, the final feature learner is detached from a part of the diffusion model. Meanwhile, SODA [Hudson et al., 2024] is proposed to generate novel images from another image and consists of an encoder and denoiser. It considers images as different views that relate visually or semantically, which significantly enhances the encoder's representation skills. Inspired by its success, we explore the potential of diffusion-based representation learning for gaze estimation, and propose an efficient self-supervised framework to learn general gaze representations.

3 Prilimilary

Our research is based on Denoising Diffusion Model (DDM) [Sohl-Dickstein et al., 2015; Ho et al., 2020]. DDM can be briefly considered to contain a pair of forward and backward Markov chains. The forward process starts from a clean data x0 and sequentially adds noise to it. At a specific time step t, the noised data xt can be formulated by:

where ¯αt defines the scaling factors of the signal and noise [Ho et al., 2020], ϵ ∈ N(0, I) is the sampled Gaussian noise.

The backward process performs image denoising in order to recover the clean data. Specifically, DDM employs trainable network to estimate ϵt by minimizing:

where ϵθ(·) is a function approximator intended to predict ϵ from xt. Therefore, DDM can generate realistic images from noise signal as ϵθ(·) be iteratively applied. Similar to other types of generative models [Mirza and Osindero, 2014], DDM also are capable of modeling conditional distributions of the form p(x|y) [Rombach et al., 2022]. This can be implemented by adding condition c into ϵθ(xt, t, c). Based on the additional condition c, DDM can generate the images we need instead of the images with random content.

4 Method

4.1 Self-supervised Gaze Representation Learning Framework

GazeDiff is a self-supervised representation learning framework for gaze feature extractor training. As shown in Figure 1, it consists of two parts, a model agnostic gaze feature extractor G and a conditional diffusion model C. Specifically, G includes a general network and a Fully Connected (FC) layer, the FC layer maps image features to a specified dimension. Such an architecture has been widely used in gaze estimation [Cheng and Bao, 2022; Wang et al., 2022; Bao and Lu, 2023]. As a result, different gaze estimation models can be easily inserted into the GazeDiff framework. Additionally, in order to reduce computational complexity and accelerate training speed, we use the popular latent diffusion model [Rombach et al., 2022] as C.

Since gaze direction is mainly influenced by head pose and eye direction, we hope gaze feature extractor can pay attention to the head pose and eye areas information from the whole image. Thus, we extract multiple images features with G, and explicitly disentangle the image representations into appearance representation, head pose representation and eye direction representation. After that, we manipulate the disentangling representations to construct a gaze-specific representation and send it to C as a condition to generate a target image with specific appearance, head pose and eye direction. The operational details will be explained later.

4.2 Conditional DiffusionModel

We generate gaze images to enhance the capability of gaze representations extracted by G. In detail, we freeze the encoder and encoder, and fine-tune the denoising U-Net only as described in Figure 1. First, the encoder E encodes face image I t into a latent representation z = E(I t ). After the diffusion process, we got a noised latent representation zt. Second, we condition the denoising process through crossattention (similar to txt-to-image diffusion models [Rombach et al., 2022]). To be more specific, we project the conditional recombined feature ˜ f t to an intermediate representation τθ( ˜ f t ) with a gaze specific encoder τθ. Then, we map it to the intermediate layers of U-Net via Attention(Q, K, V ) = softmax( QKT √d ) · V :

Here, φi (zt) indicates a flattened intermediate representation of U-Net, W (i) Q , W (i) K and W (i) V are trainable projection matrices. Next, the U-Net gradually denoises zn to restore the target latent representation ˜ z. Finally, the decoder D decodes ˜ z to a realistic face image ˜ I t = D(˜ z). We fine-tune C by minimizing the loss function:

4.3 Disentangling Feature Learning Strategy

In order to control the appearance, head pose and eye direction of generated image, we need to ensure that the condition of C contains these corresponding information. To achieve this goal, we input various data pairs and introduce a disentangling feature learning strategy.

A General Manner

In addition to inputting a single face image (we call it base input data), just like Figure 2 - (a). Easy-to-obtain unlabeled facial datasets can usually construct such a hybrid data pair (I r , I t ), which consists of two images from the same identity with different head poses and eye directions, as shown in Figure 2 - (b). Wherein, I r is a reference image, and I t is the target image that is the generate target of conditional diffusion model C. We respectively perform a Gaussian blur augmentation Ab and eye region cropping augmentation Ac on I t , then we obtained two augmented images I tb and I tc . Ab removes appearance and eye direction information, while retaining the head contour information representing the head pose. And Ac only retains the eye region image including the information of eye direction, and losing appearance information and head pose information.

As shown in Figure 1, the hybrid data pair is inputted to GazeDiff. G extracts image features from I r , I tb and I tc , and combines the appearance feature f ra from I r , the head pose feature f tb h from I tb and the eye direction feature f tc e from I tc into a new image features ˜ f t . Since I r and I t have the same identity, the generated image based on feature ˜ f t should have the same appearance, head pose and eye direction as I t . Finally, for most facial datasets, We feed base and hybrid data pairs together into GazeDiff for self-supervised training.

Enhanced Manners

For some special datasets, we can construct enhanced data pairs to further promote the extraction and disentanglement of gaze features. Specifically, some facial datasets are collected synchronously by multiple cameras, such as ETH-XGaze [Zhang et al., 2020], EVE [Park et al., 2020] and ColumbiaGaze [Smith et al., 2013]. At the same time, face images captured by different cameras have different head pose and the same eye direction. So we can also use the images from different cameras and a same timestamp to construct additional data pair to further enhance the ability to extract head pose information, as shown in Figure 2 - (c). We call this pair as pose, because pose is used to enhance the disentangling of head pose. Moreover, there are some datasets collected with fixed head pose, such as ETH-XGaze and ColumbiaGaze. Multiple images captured by the same camera have different eye directions and similar head poses. So we construct a new data pair as shown in Figure 2 - (d) and call this pair as eye.

4.4 Supervised Gaze Estimation

After the self-supervised gaze representation learning, the gaze feature learner G pre-trained by GazeDiff can be used for the task of gaze estimation. When fine-tuning on labeled gaze datasets, we add a Multi Layer Perceptrons (MLP) as gaze regressor M to predict 3D gaze direction. We train M by the loss function:

where ˆg is model predicted gaze direction and g is the ground truth label.

5 Experiments

5.1 Dataset

GazeDiff is a general self-supervised representation learning framework. Therefore, we use the following datasets in our experiments as previous methods do: ETH-XGaze [Zhang et al., 2020], EVE [Park et al., 2020], Gaze360 [Kellnhofer et al., 2019], GazeCapture [Krafka et al., 2016], ColumbiaGaze [Smith et al., 2013], MPIIFaceGaze [Zhang et al., 2017b], EyeDiap [Funes Mora et al., 2014] and VGG-Face2 [Cao et al., 2018]. More details about above datasets can be found in the supplementary material.

Respectively, we denote them as DX (ETH-XGaze), DE (EVE), DG (Gaze360), DC (GazeCapture), DO (ColumbiaGaze), DM (MPIIFaceGaze), DD (EyeDiap) and DV (VGGFace2).

5.2 Baseline Methods

We compare our approach with six following baselines: (i) SimCLR [Chen et al., 2020], (ii) MoCo v3 [Chen et al., 2021], (iii) DINO v2 [Oquab et al., 2023], (iv) MAE [He et al., 2022], (v) DDAE [Xiang et al., 2023], (vi) GazeDiff-, (vii) GazeDiff and (viii) Supervised. All methods except GazeDiff- load the released pre-trained weights on ImageNet, which means only the parameters of GazeDiff- are initialized randomly. Supervised method is trained on the annotated dataset and serves as the possible performance upper bound of the self-supervised gaze representation learning, the other methods are trained according to the self-supervised paradigm without labels. More details can be found in the supplementary materials.

5.3 Linear Probe Analysis

We perform linear probe analysis through within-dataset evaluation and cross-dataset evaluation, respectively.

Within-dataset Evaluation For the self-supervised methods, we pre-train in DX, then freeze the pre-trained learner G and add a FC layer as gaze regressor M for linear-probe analysis following [Chen et al., 2020]. We use different proportions of annotated training data to adapt M and evaluate on the separate validation data. As for the Supervised method, we finetune the whole gaze model on different proportions of annotated data. We report the experiments results in Figure 3 - (a). As we can see, regardless of whether the pre-trained weights are loaded, our proposed method GazeDiff outperforms other pre-training methods at different percentage of fine-tuning data. And when fine-tuning on a small subset of labeled data (such as 0.1% or 10% labels), GazeDiff achieves the similar performance to Supervised method, which suggesting that the learned representations are very effective for gaze estimation task.

Cross-dataset Evaluation We perform the cross-dataset evaluation following [Jindal and Manduchi, 2023]. The gaze feature learner G is pre-trained in DX, then we finetune the gaze regressor M on DM and DO separately. In order to verify the generality of G, we sample a few samples from each test subject for fine-tuning and evaluate on the remaining samples of the same subject. We conduct 10 times for each subject on both datasets and report the mean angular errors in Figure 3 - (b) and (c).

We can observe that our GazeDiff is still obviously superior to other self-supervised pre-training methods. The angular errors decrease as the number of fine-tuning samples increasing for all methods. Meanwhile, the performance gap between GazeDiff and Supervised reduces.

5.4 Comparsion with SOTAGaze Representation Learning Methods

We compare GazeDiff with existing SOTA supervised [Cheng and Bao, 2022; Lee et al., 2022; Wang et al., 2022; Xu et al., 2023; Bao and Lu, 2023; Yin et al., 2024c; Yin et al., 2025; Zeng et al., 2025] and unsupervised [Yu and Odobez, 2020; Sun et al., 2021; Du et al., 2023; Jindal and Manduchi, 2023] gaze representation learning methods. Since [Bao and Lu, 2024] is limited to multi-view datasets and its results are not available in DM and DO, we exclude their method in comparison.

Comparison with Supervised Methods

To compare the quality of learned representations with gaze labels, we fine-tune the entire gaze estimation network in a supervised manner based on unsupervised pre-training weights of GazeDiff. Besides, we add two baselines with random initialized and ImageNet pre-trained weights as a reference. Other SOTA methods trained on Dx, then evaluated on DM and DD, the experimental results on are shown in Table 1. We can observe that the baseline with ImageNet pre-training weights shows a performance improvement compared to the baseline with random initialized weights. When we load the GazeDiff pre-trained weights, the cross-dataset errors further reduce significantly. Compared with other SOTA methods, GazeDiff achieves the best performance on DD and gains the second-best performance on DM. In summary, GazeDiff derives the best average performance on two evaluation datasets. This indicates that GazeDiff provides a excellent initialization parameters for gaze estimation task.

Comparison with Unsupervised Methods

For a fair comparison, we follow the experimental settings of [Yu and Odobez, 2020; Sun et al., 2021]. Specifically, we use the leave-one-out (15-fold) and 5-fold cross-validation for DD and DO. In each fold, we randomly select 50 samples with labels to fine-tune M and repeat each experiment for 10 times. Finally, we report the average error in Table 2.

Note that Yu et al. [Yu and Odobez, 2020], Sun et al. [Sun et al., 2021] and ConGaze [Du et al., 2023] pre-trained on the evaluation dataset, [Jindal and Manduchi, 2023] uses DE as pre-training dataset, so we provide results with the same experimental setup for each method. The results show that regardless of which dataset is selected for pre-training, GazeDiff always achieves the best performance.

5.5 Ablation Study

Next, we conduct detailed ablation studies to explore the effectiveness of the proposed disentangling feature learning strategy and the stable performance improvement of GazeDiff in different setting.

Ablation Study on Disentangling Feature Learning Strategy

We proposed disentangling feature learning strategy to focus on extracting gaze-related features, and mentioned earlier that datasets with different collection settings can construct different input data pairs. For example, we can construct the four input pairs in Figure 2 on DX. So we pre-train GazeDiff on DX, and fine-tune gaze model in a supervised manner to analyze the contribution of our learning strategy. Hence, we report the cross-dataset performance of different pre-training input manners. Additionally, we also provide the results of Supervised baseline with ImageNet pre-training weights as a reference. The experimental results are reported in Table 3. Compared with the Supervised baseline, we can see that even with the powerful generation capability of diffusion model, GazeDiff with base input data (equivalent to autoencoder pretraining) only achieve a slight improvement of 0.11◦ on the average error. With regard to our learning strategy, after adding the hybrid data pair as input, the model gains an obvious improvement of 0.58◦ compared to the GazeDiff with base input data. Next, we add pose and eye pairs as input, the cross-dataset performances improve again. The absolute performance gains brought by pose and eye pairs are 0.25◦ and 0.11◦ . Overall, the hybrid data pair contributes the most to performance enhancement, and this data pair can be constructed from a vast majority of facial datasets. Above results prove that our GazeDiff is a general and effective framework.

Ablation Study on Disentangling Augmentation

We apply Gaussian blur augmentation and eye region cropping to reduce the impact of undesirable information in the face image. Nevertheless, information remains in blurred and cropped images, for example, there may be residual head posture information in the eye region cropped image. We train models on DX to estimate head pose and eye gaze respectively, then test on DM. The results are reported in Table 4.Ab indeed discards most of eye direction information, and Ac removes most of head pose information. While it cannot be entirely eliminated, the network is forced to extract more accurate poses from blurred images and eye directions from eye images in order to generate high-precision images.

Ablation Study on Pre-training Datasets

In the previous experiments, in order to maintain consistent experimental settings with other methods, we only reported the results of pre-training on the gaze dataset. In this section, we pre-train on DV, which is a large facial recognition dataset without gaze annotation, then fine-tune the model on the gaze dataset to demonstrate the performance in actual applications. The results are reported in Table 5. We can observe that, the models with pre-training weights perform consistent better than the models without pre-training weights on all training datasets. Then, the models pre-trained on DV have a better performance than the models pre-trained on DX. This demonstrates that a more diversity and larger pre-training dataset can further improve the performance of GazeDiff. It is very practical and valuable to perform unsupervised gaze pre-training on a face dataset even without gaze labels.

For more ablation studies on training datasets and network architectures, please refer to the supplementary materials.

5.6 Visualization Analysis

Visualization of learned feature distributions

Following [Du et al., 2023], We use t-SNE [Van der Maaten and Hinton, 2008] to visualize the distributions of gaze representation on DM and DD. Models are pre-trained on DX. Different colors indicate different subjects, and each point corresponding to one image. We visualize the feature distributions of appearance, head pose and eye direction separately. The visualization results are shown in Figure 4. For the feature distributions of GazeDiff on evaluation datasets, it can be seen that features of the same identity clustered together and far away from the features of other people. As for head pose features and eye direction features, they are continuously distributed and invariant to subject identity. These feature distributions indicate the image features have been successfully disentangled into appearance, head pose and eye direction. Overall, these results show that the representations learned by GazeDiff are disentangled and can be easily transferred to unseen datasets.

Latent Interpolations

As shown in Figure 5, we explore G's feature space and discover directions that corresponding to semantic attributes of appearance, head pose and eye direction. By traversing its feature space, we can interpolate between images, morphing from one head pose to another head pose smoothly, and the same goes for appearance and eye direction contributes. For example, we disentangle the image features into appearance feature, head pose feature and eye direction feature. In order to control the head pose, we linearly interpolate from one head pose feature fh1 to another fh2 , and keep the appearance feature and eye feature unchanged. The visualization demonstrates that, GazeDiff is trained fully unsupervised over facial images only, but successfully disentangles the image features into different parts of gaze-related representations, which proves that the effectiveness of our framework and proposed disentangling feature learning strategy. more visualization analysis can be found in the supplementary material.

6 Conclusion

We propose a diffusion-based framework (GazeDiff) for selfsupervised gaze representation learning. GazeDiff exploits the generative power of diffusion models and enhances the quality, informativeness and interpretability of gaze representations. In order to guarantee the important head pose and eye information are learned from face images, we design a disentangling feature learning strategy. Specifically, we use Gaussian blur augmentation to keep head pose information and extract head pose feature. Then, we crop eyes region to extract eye direction feature. Finally, we take appearance

Text Text feature from a reference image and combine these features as a condition to guide the generation. Our method achieves the SOTA performance in extensive evaluation settings, and the detailed ablation study and visualization analysis have demonstrate the generality and effectiveness of our work.

Acknowledgements

This work was sponsored by National Natural Science Foundation of China (62476173), Shenzhen Fundamental Research Fundation(JCYJ20240813142610014) and National Key R&D Program of China (2023YFE0204200).

Contribution Statement

This work was a collaborative effort by all contributing authors. Guanzhong Zeng and Jingjing Wang made equal contributions to this study and are designated as co-first authors. Mingyang Zhou serving as the corresponding author, is responsible for all communications related to this manuscript.