GASP: Gated Attention for Saliency Prediction

1 Introduction

Attending to regions or objects in our perceptual field implies an interest in acting towards them. Humans convey their attention by fixating their eyes upon those regions. By modeling fixation, we gain an understanding of the events that attract attention. These attractors are represented in the form of a Fixation Density Map (FDM), displaying blurred peaks on a two-dimensional map, centered on the eye Fixation Point (FP) of each individual viewing a frame. The FDM is a visual representation of saliency, a useful indicator of what attracts human attention.

∗Contact Author

1 Code: http://software.knowledge-technology.info#gasp

Early computational research focused on bottom-up saliency, by which the conspicuity of regions in the visual field was purely dependent on the stimuli [Itti and Koch, 2001; Bruce and Tsotsos, 2009]. On the other hand, taskdriven approaches are top-down models utilizing supervised learning for performing tasks and allocating attention to regions or objects of interest. Combining face detections with low-level features has been shown to outperform bottom-up saliency models agnostic to social entities in a scene. Birmingham et al. [2009] corroborate the advantage of facial features in modeling saliency. They establish that when social stimuli are present, humans tend to fixate on facial features, a phenomenon weakly portrayed by bottom-up saliency detectors. Moreover, studies on human eye movements indicate that bottom-up guidance is not strongly correlated with fixation, which is rather influenced by the task [Foulsham et al., 2011]. The existence of social stimuli in a scene alters fixation patterns, supporting the notion that even with the lack of an explicit task, we form intrinsic goals for guiding our gaze.

Although facial features attract attention, studies show that humans tend to follow the gaze of observed individuals [Bylinskii et al., 2016]. Additionally, psychological studies [Pritsch et al., 2017] indicate a preference in attending towards emotionally salient stimuli over neutral expressions, a phenomenon described as affect-biased attention. By augmenting saliency maps with emotion intensities, affect-biased saliency models show significant improvement over affectagnostic models [Fan et al., 2018; Cordel et al., 2019]. These approaches, although exclusive to static saliency models, are not limited to facial expressions, allowing for a greater domain coverage irrespective of the presence of social entities in a scene.

In light of the social stimuli relevance to modeling attention, we design a model to predict the FDM of multiple human observers watching social videos as shown in Figure 1. Such models employ top-down and bottom-up strategies operating on a sequence of images, a task referred to as dynamic saliency prediction [Bak et al., 2017; Borji, 2019]. Our model utilizes multiple social cue detectors, namely gaze following and direction estimation, as well as facial expression recognition. We integrate the eye gaze and affective social cues, each with its spatiotemporal representation, as input to our saliency prediction model. We describe the resulting output from each social cue detector as a feature map (FM). We also introduce a novel FM weighting module, assigning different intensities to each FM in a competitive manner representing its priority. Each representation is best described as a priority map (PM), combining top-down and bottom-up features to prioritize regions that are most likely to be attended. We refer to the final model output as the Predicted FDM (PFDM).

Our work is motivated by the following findings: 1) Taskdriven strategies are pertinent to predicting saliency [Foulsham et al., 2011]; 2) Changes in motion contribute to the relevance of an object, underlining the importance of spatiotemporal features for predicting saliency [Min et al., 2020]; 3) Psychological studies indicate that attention is driven by social stimuli [Salley and Colombo, 2016]. To address the first finding, we state that our approach is task-driven by virtue of supervision since the objective is predicated on modeling multiple observer fixations. Although the datasets employed in this study were collected under a free-viewing condition, the top-down property is arguably maintained due to the intrinsic goals of the observer. These goals are driven by socially relevant stimuli addressed in our model through its reliance on multiple social cue modalities and facial information. We detect the social and facial features in a separate stage, hereafter described as the Social Cue Detection (SCD) stage.

To address the second finding, our model learns temporal features in two stages. Sequential learning in SCD is not a necessity but a result of the models employed for social cue detection, e.g., recurrent models or models pre-trained on optical flow tasks. In the second stage (GASP), we integrate social cues as illustrated in Figure 2. GASP also employs sequential learning, not only registering environmental changes such as color and intensity but also features pertaining to motion.

Finally, we consider social attention by employing an audiovisual saliency prediction modality, as well as social cue detectors (also described as modalities) that specialize in per-

Algorithm 1 SCD sampling and generation

Input:

Video and audio frames sampled from ds = AVE dataset Parameters: Window sizes WSP = 15, WGE = 7, WGF = 5, WFER = 0 O/P steps T'SP = 15, T'GE = 4, T'GF = 0, T'FER = 0 Output: Modality windows mdlwin O/P buffers bufmdl 1: for vid in ds do 2: Let t = 0 3: for frm in vid do 4: Let fcs = face crops & bounding boxes in frm 5: for mdl in ∈ {SP, GE, GF, FER} do 6: if Wmdl > t then 7: Let ∆ = Wmdl - t 8: for δ in ∈ {∆, ..., Wmdl} do 9: Let mdlwin[δ] = <frm, fcs> 10: end for 11: else 12: Shift mdlwin by 1 to the left 13: Let mdlwin[Wmdl] = <frm, fcs> 14: end if 15: Execute bufmdl[t] = mdl(mdlwin)[T'mdl] 16: end for 17: Propagate buf[t] to GASP 18: Let t = t + 1 19: end for 20: end for

forming distinct tasks. Each of these tasks is highly relevant to visual attention, both from a behavioral and a computational perspective. We aim to explore feature integration approaches for combining social cues. We present gated attention variants and introduce a novel approach for directing attention to all modalities. To the best of our knowledge, our model is the first to consider affect-biased attention by using facial expression representations for dynamic saliency prediction based on deep neural maps.

2 Social Cue Detection

In the first stage (SCD), we extract high-level features from three social cue detectors and an audiovisual saliency predictor. We utilize the S 3 FD face detector [Zhang et al., 2017] for acquiring the face locations of actors in an image. The cropped face images are passed to the social cue detectors as input. The window size W, i.e., the number of frames fed simultaneously as input to each model, varies according to the requirements of each model. We generate modality representations at output timestep T ′ for each social video in AVE [Tavakoli et al., 2020] as shown in Algorithm 1.

Our motivation behind selecting social cue detectors lies in their ability to generalize to various "in-the-wild" settings, regardless of the surrounding environment or lighting conditions. All chosen models were trained on datasets containing social entities captured from different angles and distances.

2.1 Gaze FollowingModel (GF)

VideoGaze [Recasens et al., 2017] receives the source image frame containing the gazer, the target frame into which the gazer looks, and a face crop of the gazer in the source frame along with the head and eye positions as input to its pathways. All frames are resized to 227 × 227 pixels. The model acquires five consecutive frames (WGF ) at timestep T ′GF and returns a fixation heatmap of the most probable target frame for every detected face in a source frame.

Representation. The mean fixation heatmaps resulting from each face in the source frame are overlaid on a single feature map in the corresponding target frame timestep. We transform the fixation heatmaps using a jet colormap.

2.2 Gaze EstimationModel (GE)

Gaze360 [Kellnhofer et al., 2019] estimates the 3D gaze direction of an actor. The model receives face crops of the same actor over a predefined period, covering seven frames (WGE) centered around timestep T ′GE . Each crop is resized to 224 × 224 pixels. The model predicts the azimuth and pitch of the eyes and head along with a confidence score.

Representation. We generate cones and position their tips on detected face centroids. The cones are placed upon a zerovalued map with identical dimensions to the input image. The cone base is rotated towards the direction of gaze. The apex angle of the cone is set to 60◦ . The face furthest from the lens is projected first with an opacity of 0.5, followed by the remaining faces ordered by their distances to the lens. A jet colormap is then applied to the cone map.

2.3 Facial Expression RecognitionModel (FER)

We employ the facial expression recognition model developed by Siqueira et al. [2020]. The model is composed of convolutional layers shared across 9 ensembles. The model receives all face crops in a frame as input, each resized to 96 × 96 pixels, and recognizes facial expressions from 8 categories. Since the model operates on static images, we set the window size WFER and output timestep T ′FER to 0.

Representation. Grad-CAM [Selvaraju et al., 2017] features are extracted from all 9 ensembles. We take the mean of the features for all faces in the image and apply a jet colormap transformation on them. A 2D Hanning filter is applied to the features to mitigate artifacts resulting from the edges of the cropped Grad-CAM representations. We center the filtered representations on the face positions upon a zero-valued map with dimensions identical to the input image.

2.4 Audiovisual Saliency PredictionModel (SP)

In the SCD stage, we utilize DAVE [Tavakoli et al., 2020] for predicting saliency based on visual and auditory stimuli. Separate streams for encoding the two modalities are built using a 3D-ResNet with 18 layers. The visual stream acquires 16 images (WSP ), each resized to 256 × 320 pixels. The auditory stream acquires log Mel-spectrograms of the video-corresponding audio frames, re-sampled to 16kHz. The model produces an FDM at the final output timestep T ′SP considering all preceding frames within the window WSP .

Representation. We transform the resulting fixation density map from DAVE using a jet colormap.

3 Gated Attention for Saliency Prediction

We standardize all SCD features to a mean of 0 and a standard deviation of 1. The input image (IMG) and FMs are resized to 120 × 120 pixels before propagation to GASP.

3.1 Directed AttentionModule (DAM)

The Squeeze-and-Excitation (SE) [Hu et al., 2018] layer extracts channel-wise interactions, applying a gating mechanism for weighting convolutional channels according to their informative features. The SE model, however, emphasizes modality representations having the most significant gain, mitigating channels with lower information content. For our purpose, it is reasonable to postulate that the most influential FM channels are those belonging to the SP since it would result in the least erroneous representation in comparison to the ground-truth FDM. However, this causes the social cue modalities to have a minimal effect, mainly due to their low correlation with the FDM as opposed to the SP.

To counter bias towards the SP, we intensify non-salient regions such that the model learns to assign greater weights to modalities contributing least to the prediction. Alpay et al. [2019] propose a language model for skipping and preserving activations according to how surprising a word is, given its context in a sequence. In this work, we assimilate surprising words to channel regions with an unexpected contribution to the saliency model.

We construct a model for emphasizing unexpected features using two streams as shown in Figure 3: 1) The inverted stream with output heads; 2) The direct stream attached to the modality encoders of our GASP model. The inverted stream is composed of an SE layer followed by a 2D convolutional layer with a kernel size of 3 × 3, a padding of 1, and 32 channels. A max pooling layer with a window size of 2 × 2 reduces the feature map dimensions by half. Finally, a 1 × 1 convolution is applied to the pooled features, reducing the feature maps to a single channel. To emphasize weak features, we invert the input channels:

where uc′ represents the individual channels of all modalities. The spatially inverted channels u-1 c′ are standardized and propagated as input features to the inverted stream. The direct stream is an SE layer with its parameters tied to the inverted stream and receives the standardized FM channels uc′ as input. Finally, the direct stream propagates the channel parameters multiplied with each FM to the modality encoders of GASP. The resulting weighted map is the priority map (PM).

3.2 Modality Encoder (Enc.)

The modality encoder is a convolutional model used for extracting visual features from the priority maps. The first two layers of the encoder have 32 and 64 channels respectively. A maximum pooling layer reduces the input feature map to half its size. The pooled layer is followed by two layers with 128 channels each. Finally, the representations are decoded by applying transposed convolutions with 128, 64, and 32 channels. The last layer has a number of channels equivalent to the input channels. All convolutional kernels have a size of 3 × 3, with a padding of 1. For GASP model variants operating on single frames (static integration variants), all modalities share the same encoder. For sequential integration variants, each modality has a separate encoder shared across timesteps.

3.3 Recurrent GatedMultimodal Unit (RGMU)

Concatenating the modality representations could lead to successful integration. Such a form of integration is commonly used in multimodal neural models, including audiovisual saliency predictors. We describe such approaches as nonfusion models, whereby the contribution of each modality is unknown. To account for all modalities, we employ the Gated Multimodal Unit (GMU) [Arevalo et al., 2020]. The GMU learns to weigh the input features based on a gating mechanism. To preserve the spatial features of the input, the authors introduce a convolutional variant of the GMU. This model, however, disregards the previous context since it does not integrate features sequentially. Therefore, we extend the convolutional GMU with recurrent units and express it as follows:

where h(m) t is the hidden representation of modality m at timestep t. Similarly, z(m) t indicates the gated representation. The total number of modalities is represented by M. The parameters of the Recurrent Gated Multimodal Unit (RGMU) are denoted by W(m) x , W(m) z , U(m) h , and U(m) z . The modality inputs x(m) at timestep t are concatenated channel-wise as indicated by the [·,·] operator and convolved with W(m) z . The z(m) t representation is acquired by summing the current and previous timestep representations, along with the bias term b(m) z . A sigmoid activation function denoted by σ is applied to the recurrent representations zt. The final feature map ht is the Hadamard-product between z(m) t and h(m) t summed over all modalities.

The aforementioned recurrent approach suffers from vanishing gradients as the context becomes longer. To remedy this effect, we propose the integration of GMU with the convolutional Attentive Long Short-Term Memory (ALSTM) [Cornia et al., 2018]. ALSTM applies soft-attention to single timestep input features over multiple iterations. We utilize ALSTM for our static GASP integration variants. For sequential variants, we modify ALSTM to acquire frames at all timesteps instead of attending to a single frame multiple times:

where zt-1 represents the pre-attentive output of the previous timestep. We adapt the sequential ALSTM to operate in conjunction with the GMU by performing the gated fusion per timestep. We refer to this model as the Attentive Recurrent Gated Multimodal Unit (ARGMU). Alternatively, we perform the gated integration after concatenating the input channels and propagating them to the sequential ALSTM. Since the modality representations are no longer separable, we describe this variant as the Late ARGMU (LARGMU). We refer to the total number of timesteps as the context size. Analogous to the sequential variants, we create similar gating mechanisms for static integration approaches. Replacing the sequential ALSTM with the ALSTM by Cornia et al. [2018], we present the non-sequential Attentive Gated Multimodal Unit (AGMU), as well as the Late AGMU (LAGMU).

4 Experimental Setup

We train our GASP model on the social event subset of AVE [Tavakoli et al., 2020]. AVE is a composition of three datasets: DIEM [Mital et al., 2011], Coutrot Databases 1 [Coutrot and Guyader, 2014] and 2 [Coutrot and Guyader, 2015]. To train the model, we employ the loss functions introduced by Tsiami et al. [2020], assigning the loss weights λ1 = 0.1, λ2 = 2, and λ3 = 1 to cross-entropy, CC, and NSS losses respectively. The loss functions LPFDM are weighted, summed, and applied to the final layer for optimizing the modality encoder and integration model parameters. The model is trained using the Adam optimizer, having a learning rate of 0.001, with β1 = 0.9 and β2 = 0.999. All models are trained for ∼10k iterations with a batch size of 4. We conduct five trials acquiring the mean results for evaluation.

The models are evaluated on the test subset of social event videos in AVE. We employ five commonly used metrics in dynamic saliency prediction [Tavakoli et al., 2020; Tsiami et al., 2020]: Normalized scanpath saliency (NSS); Linear correlation coefficient (CC); Similarity metric (SIM); Area under the ROC curve (AUC-J); Shuffled AUC (sAUC). The negative fixations for the sAUC metric are sampled from all the mean eye positions in the social event subset of AVE.

The inverted stream of our DAM layer has a separate output head for each timestep. We compute the cross-entropy between the DAM prediction and the FDM. For sequential integration models, the loss is summed over all timesteps. The loss LDAM with a weight λDAM = 0.5 is computed for optimizing the inverted stream parameters. The parameters are transferred to the direct stream with frozen parameters.

An NVIDIA RTX 2080 Ti GPU with 11 GB VRAM and 128 GB RAM is used for training all static and sequential models. To extract spatiotemporal maps in the first stage (SCD), we employ an NVIDIA TITAN RTX GPU with 24 GB VRAM and 64 GB RAM to accommodate all social cue detectors simultaneously. We perform the SCD feature extraction in a preprocessing step for all AVE dataset videos.

5 Results

5.1 Static Integration

We examine integration approaches operating on a single frame in GASP. The Additive model refers to the integration variant in which the feature maps of all encoders are summed, followed by a 3 × 3 convolution with 32 channels and a padding of 1. The Concatenative variant applies a channel-wise concatenation to the feature maps, followed by the aforementioned convolutional layer. ALSTM, LAGMU, and AGMU employ the non-sequential ALSTM variant by Cornia et al. [2018]. The Squeeze-and-Excitation [Hu et al., 2018] (SE) model precedes the modality encoder. We note that all models excluding SE and DAM replace the integration model. Finally, all model variants are followed by a 1 × 1 convolution resulting in the final output feature map.

In Table 1, we show that the best NSS scores are achieved by our DAM + GMU variant. This indicates fewer false positives predicted by fusion in comparison to other non-fusion methods. We also observe that the DAM enhances fusion models but appears to have minimal effect on the GMU variant (the difference across all metrics is insignificant).

5.2 Sequential Integration

We modify our GASP integration model to have a context greater than one. All models employ batch normalization applied to the temporal axis. The integration models are followed by a 1 × 1 convolution resulting in the final output feature map. In Table 2, we experiment with context sizes ∈ {2, 4, 6, 8, 10, 12} and observe an overall improvement in performance with a context size of 4. The directed attention variant with late non-fusion gating (DAM + LARGMU) achieves the best scores on all metrics. This implies that gated integration is beneficial, even though the representations preceding the GMU are not separable.

Comparing the results of static integration in Table 1 to dynamic integration approaches in Table 2, we observe that several static approaches perform on a par with recurrent models. Nonetheless, the sequential DAM + LARGMU with context sizes of 8 and 10 outperform all integration methods. In Table 3, we observe an insignificant difference in metric scores among the best sequential models for all context sizes. Compared to the best static model, the variances of sequential model scores are lower, indicating the stabilizing influence of attentive LSTMs with the addition of context.

5.3 Modality Contribution

We evaluate the contribution of each modality to the final prediction by computing the mean activation of the gates across channels and timesteps. This evaluation is only applicable to fusion methods in either static or sequential forms of integration. We observe that the DAM does not alter the modality contribution of the static GMU. For sequential variants, introducing the DAM allows modalities to have a uniform contribution to the final output.

We examine the modalities contributing an improvement to the best non-fusion sequential model. As shown in Table 4, FER in combination with GE achieves results on par with the best model. The exclusion of GF has a minimal effect on the model due to the sparsity of its representation. Significant degradation in the model variant with social modalities ablated implies the necessity of social cues in concert.

5.4 Comparison with State-of-the-art

We compare the performance of our model with four dynamic saliency predictors. We replace DAVE [Tavakoli et al., 2020] with STAViS [Tsiami et al., 2020], TASED [Min and Corso, 2019] and UNISAL [Droste et al., 2020] in the SCD stage during the evaluation phase. Due to the overlap in datasets between DAVE and STAViS, we retrain our GASP model with STAViS as the SCD audiovisual saliency predictor. We evaluate and train our STAViS-based model on social event videos according to the data splits concocted by Tsiami et al. [2020] to avoid data leakage. The average scores are computed for all split test sets over five trials.

Combining our best GASP model with different saliency predictors improves their performances, as shown in Table 5. Although GASP is not retrained, it extracts information from the untuned saliency predictors pertinent to the prediction. The sequential GASP also exhibits greater resistance to central bias as shown in Figure 5 (middle row) compared to other models, where the actor closest to the center is incorrectly predicted as a fixation target. The social cue integration and sequential learning in GASP contribute to such resistance.

6 Conclusion

We introduce GASP, a gated attention model for predicting saliency by integrating knowledge from two social cues using three separate detectors. We represent each social cue as a spatiotemporal map, accounting for the gaze direction, gaze target, and facial expressions of an observed individual. We examine gated and recurrent approaches for integrating the social cues. Our gated integration variants achieve better results than non-gated approaches. We also present a module for emphasizing weak features, which is effective for static and sequential integration methods. Furthermore, we show that gaze direction and facial expression representations have a positive effect when integrated with saliency models. The latter supports the importance of considering affect-biased attention. GASP improves state-of-the-art saliency model prediction performances on multiple metrics, indicating the efficacy of social cue integration in our architecture.

Acknowledgements

The authors gratefully acknowledge partial support from the German Research Foundation DFG under project CML (TRR 169).