Counterfactual Thinking Driven Emotion Regulation for Image Sentiment Recognition

1 Introduction

Emotion is a state that individuals constantly evoke and experience [Cui et al., 2023]. With the continuous development of social communication channels, vast amounts of visual information, particularly images, now flood social media platforms. Image Sentiment Recognition (ISR) focuses on detecting and classifying the emotions conveyed by images through visual analysis. On a personal level, ISR can play a crucial role in identifying early signs of mental health issues, such as depression and anxiety, enabling timely intervention [She et al., 2019]. On a commercial level, ISR can analyze users' emotional states and their fluctuations on social media, helping platforms enhance user experience, identify social trends, and track emerging hot topics [Wang et al., 2020]. As a result, ISR has found widespread applications across diverse fields, including education [Tan et al., 2023] and opinion mining [Li et al., 2019].

∗Corresponding Author

Traditional ISR methods rely on handcrafted features to identify the emotions conveyed by images. With the development of the ISR field, these methods can be broadly categorized into image-level and region-based approaches. Imagelevel methods employ convolutional neural networks to analyze an entire image and determine its predominant emotion [You et al., 2015]. In contrast, region-based methods focus on specific affective regions within an image, which are critical for evoking emotional responses [Zhang et al., 2022b; She et al., 2019; Zhang et al., 2024]. As a result, region-based ISR has emerged as a primary area of research interest. However, these methods face two significant challenges:

First, current ISR methods rely on correlation-based learning [You et al., 2016; Zhang et al., 2023], focusing on statistical associations rather than causal relationships. This reliance often results in overfitting to dataset biases or spurious correlations, ultimately limiting the model's ability to accurately identify regions that genuinely evoke emotions [Rao et al., 2021]. To address these limitations, some researchers have attempted to model the innate causality mechanisms of humans [Zhang et al., 2024; Yang et al., 2023a]. These causality mechanisms enable humans to naturally mitigate emotional content biases, allowing fair emotional judgments despite visual biases [Yang et al., 2024]. For example, flowers are often perceived as positive imagery [You et al., 2016], yet this perception is not solely based on their visual features but also influenced by causal reasoning rooted in personal experiences and cultural context. Despite cultural differences, humans consistently interpret emotions in visual content by leveraging these experiences and contexts, resonating with familiar symbols and associations through well-developed causal reasoning processes [You et al., 2015; Sun et al., 2022]. Secondly, the lack of effective tools to evaluate affective regions often causes models to focus on irrelevant attributes. For instance, if "joy" samples frequently include a table, the model might associate the table with joy. Region-based methods address this using additional annotations (e.g., bounding boxes) [Yang et al., 2018b], yet this approach is labor-intensive and difficult to scale.

Therefore, we integrate the innate counterfactual thinking ability of humans to adjust predictions more effectively. This approach mimics the natural human emotion judgment process, driven by causality, within an ISR model. According to Gross's Emotion Regulation Theory, emotion regulation involves processes that influence the occurrence, experience, and expression of emotions [Gross, 2015]. These processes are categorized into five stages: situation selection, situation modification, attentional deployment, cognitive change, and response modulation [Gross, 2002]. Inspired by this, we build a Counterfactual Thinking driven Emotion Regulation Network (CTERNet) for the ISR task, which aims to simulate the Emotion Regulation process of humans by re-evaluating the scenarios of emotional stimuli to change the meaning of the outcomes. The CTERNet is employed with a responsefocused strategy [Gross, 2002] in Emotion Regulation, where the adjustment occurs after the emotion has already been generated and the emotional response has been activated. The response-focused strategy aligns well with how humans experience emotions when observing an image. For instance, as shown in Figure 1(a), when people notice the rose (Area 1) in a picture and interpret the emotion conveyed as "joy", they might then reflect due to the nature of cognitive load and attention [Sweller, 2020]: What if I do not notice the Area 1 in the image? That is, after obtaining a prediction, humans will reflect through counterfactual thinking [Rao et al., 2021], which can re-examine the problem to ensure that no important information has been overlooked and to obtain the correct emotion prediction.

Specifically, we first utilize a multi-scale perception network to extract features from images, simulating various situation selections. Next, we integrate counterfactual thinking to combine situation modification, attentional deployment, and cognitive change into a counterfactual-thinking-based cognitive reappraisal module (C2RM). To clarify the causal relationships between affective regions and emotional predictions within the counterfactual framework, we build a structural causal graph In C2RM, we simulate human emotional arousal under different situations by learning affective regions (factual) and exploring alternative potential regions (counterfactual). During the response modulation stage, emotional predictions are obtained by comparing the impacts of factual and counterfactual regions. This approach enables context-aware predictions and evaluates the quality of identified affective regions. To focus on emotionally representative regions and minimize biased sentimental cues, we apply the total effect (TE) [Pearl, 2014; Yang et al., 2024].

The main contributions are summarized as follows:

• We propose CTERNet, a novel method based on Emo-
• tion Regulation Theory, for ISR through total effect analysis. This method equips machines with the ability to compare factual and counterfactual outcomes based on causality.
• We construct counterfactual thinking based cognitive reappraisal module (C2RM) as the core component of CTERNet, simulating human emotional arousal across various situations.
• We utilize structural causal graphs to reformulate CTERNet, revealing the ISR process from image input to prediction, identifying irrelevant attributes, and intuitively providing counterfactual thinking interventions.

2 RelatedWork

2.1 Region-based Image Sentiment Recognition

Region-based ISR methods aim to identify specific areas within an image that strongly convey emotions [Yang et al., 2018a]. Existing approaches mainly follow two strategies. The first is precise annotation-based methods (e.g., bounding boxes and segmentation masks) [Zhang et al., 2022b], which require additional prior knowledge, such as generating proposals to locate bounding boxes. However, these methods are time-consuming and labor-intensive compared to image-level annotations [Zhang et al., 2023]. Additionally, they often retain proposals focused on foreground objects after processing through a localization regressor, potentially leading to information loss during emotional analysis [She et al., 2019]. To address these issues, a mainstream weakly-supervised strategy has emerged, offering soft proposals for evoking emotions. This strategy identifies attention-grabbing regions related to human visual attention, leveraging visual saliency in weakly-supervised ISR [You et al., 2017; Fan et al., 2017; He et al., 2019; Zhang et al., 2024].

2.2 Counterfactual in Emotion Regulation

In psychology, Emotion Regulation involves managing emotions through planned efforts. Despite varying definitions [Thompson, 1994; Feldner et al., 2003; Gross, 2015; Cludius et al., 2020], Gross's process model is widely accepted [Gross, 2002; Gross and Feldman Barrett, 2011], distinguishing between antecedent-focused and response-focused strategies. Cognitive reappraisal, a form of cognitive change, alters perceptions of emotional events by hypothesizing different outcomes [McRae et al., 2012; Theodorou et al., 2023], essentially using counterfactual thinking [Rye et al., 2008; Sirois et al., 2010; Parikh et al., 2022]. This involves evaluating past events to predict, reason, and attribute causality. Some studies suggest a causal relationship between emotion and counterfactual thinking during cognitive reappraisal, aiding adaptation to social and environmental needs.

3 Methodology

Emotion Regulation Theory, as illustrated in Figure 1(a), delves into the traditional understanding of emotions or sentiments, suggesting that our emotional experiences are part of a broader causal chain [Zhang et al., 2024; Co¨egnarts and Kravanja, 2016; Gross, 2015], which lay the foundation for SMU Classification: Restricted the proposed method in this proposal. According to Emotion Regulation [Gross, 2002; Pearl and Mackenzie, 2018; Rao et al., 2021; Pearl, 2022], as shown in Figure 1(a), there are five stages including situation selection (Figure 1(a0)), situation modification and attentional deployment (Figure 1(a1)), cognitive change (Figure 1(a2)), and response modulation (Figure 1(a3)). To better simulate the Emotion Regulation process (illustrated in Figure 1(a)) within Emotion Regulation Theory, we propose a novel counterfactual thinking driven emotion regulation network (CTERNet) for identifying emotional or sentiment categories, such as positive, negative, and their respective subcategories. The outputs (binary or multiclass) depend on the corresponding datasets available and the requirements.

The proposed CTERNet simulates the five stages of Emotion Regulation Theory. Feature extractor, as shown in block (b0) in Figure 1(b) simulates situation selection as shown in Figure 1(a0). The pairs: SMAD-F (Figure 1 b1)) and SMAD-C (Figure 1(b2)) are the core components of the CTERNet, which simulate various emotional arousal processes, such as SMAD factual thinking (a1) and SMAD counterfactual thinking (a2) in Emotion Regulation Theory. The effects of SMAD-F (factual thinking), depicted in block

(b1), are compared with those of SMAD-C (counterfactual thinking), illustrated in block (Figure 1(b2)), before conducting response modulation (Figure 1(a3)). The final output of the CTERNet, YTE, can be obtained through response modulation (a3), which involves adding the factual prediction (from SMAD-F) and its counterfactual prediction (from SMAD-C).

We built this Structural Causal Graph [Pearl and others, 2000; Pearl, 2014], which corresponds to the proposed CTERNet (Figure 1(c)). By introducing such a structural causal graph, we aim to simplify the proposed method to enable the novice to understand the relationship between different variables, I, X, S, S∗ , Ys,x(I) and Y ′ s∗ ,x (I). I represent input images, X represents the extracted features, S represents pseudo sentiment maps obtained by module SMAD-F (Figure 1(b1)) (also illustrated as (Figure 1(c1)), S∗ represents pseudo sentiment maps obtained by module SMAD-C (Figure 1(b2)) (also illustrated as Figure 1(c2)), Ys,x(I) represents the final emotion response prediction output by module SMAD-F (Figure 1(b1)) (also illustrated as Figure 1(c1)), and Y ′ s∗ ,x represents the final emotion response prediction output by module SMAD-C (Figure 1(b2)) (also illustrated as Figure 1(c2)). The link I → X → S represents the extraction of features X from I through the corresponding encoder in the ISR model, followed by the generation of pseudo sentiment maps S. The link I → (X, S) → Y signifies the final emotion classification prediction obtained by integrating features X and sentiment regions S, capturing the causal effect of X on the model's prediction Y .

3.1 Feature Extractor (b0)

Feature extractor (Figure 1(b0)) simulates the situation selection (Figure 1(a0)) of the Emotion Regulation Theory. By processing and analyzing the visual information, individuals can assess the emotional impact of the current context and make appropriate choices [Gross and Feldman Barrett, 2011; Gross, 2015]. In Emotion Regulation Theory, situation selection (Figure 1(a0)) refers to the process of regulating emotions by choosing to engage with or avoid certain emotional situations (such as Area 1, Area 2, in Figure 1(a)). Therefore, I → X as a feature extraction process (Figure 1(b0)) naturally aligns with the situation selection stage in Emotion Regulation Theory. By selecting specific emotion scenarios (i.e., avoiding certain scenarios) [Gross and John, 2003], humans can achieve the goal of emotional regulation. To simulate the different emotional scenarios directed by various visual inputs, we utilize Res2Net-101 [Gao et al., 2019] as the backbone for the I →X process.

3.2 SituationModification and Attentional Deployment Factual Thinking (SMAD-F) (b1)

After selecting a scenario through situation selection (Figure 1(a0)), the corresponding module for the proposed CTERNet is Feature Extractor shown in Figure 1(b0). This scenario is then modified to obtain and modify the emotional impact, which is known as situation modification and attentional deployment (SMAD) factual thinking (Figure 1(a1)). For example, suppose we choose the emotional scenario in Area 1 from Figure 1(a). In that case, we can generate different situations (e.g., different features including low-level features and high-level features) based on the Area 1. The corresponding module for the proposed CTERNet is SMADF as shown in Figure 1(b1). As shown in block (b1) of Figure 1, this preliminary control of emotional scenarios is simulated using different multi-scale convolutional filters, similar to the way situation modification work in the human emotion perception process. The process can be represented by the following equation.

where g ∈ G represents dividing X into g groups based on the number of channels. For each group, we use convolutional filters with a kernel size (KS), KS = 2g+1 for emotion perception. Following the practice in [Zhang et al., 2022a], we set g to 4.

The situations in situation modification have different aspects [Gross, 2002], therefore, we concatenate the outputs from different emotional scenario perceptions along the channel dimension as the input of attentional deployment. Through attentional deployment, we can select the specific situation aspect to focus on. To mimic the attentional deployment stage, we reshape the feature map Sin to a size of C × (W × H). We then multiply the reshaped feature map by the transpose of Sin, and after normalization, we obtain a feature map Sout of size C × C. Next, Sout is multiplied by the transpose of Sin, and the result is reshaped back to C ×W ×H with a learnable parameter φ, which is shown in Equation below.

where Sij represents the dependency between the i-th and jth channels, defined as exp((Sin)i·(Sin)j) ∑Ci=1 exp((Sin)i·(Sin)j) . This process simulates the operation of attentional deployment, helping CTERNet focus on the most discriminative and important regions of the image.

3.3 SituationModification and Attentional Deployment Counterfactual Thinking (SMAD-C) (b2)

SMAD-C component (in Figure 1(b2)), also known as cognitive change as shown in Figure 1(a2), in Emotion Regulation Theory, involves selecting potential interpretations of the significance of emotional events. Based on the counterfactual thinking strategy in Emotion Regulation Theory [Parikh et al., 2022], we consider: if the model sees other potential affective regions (e.g., Area 2), what would the prediction be? By learning from both the affective regions identified by the SMAD-F module (factual) (see Figure 1(b1)) and other potential affective regions identified by the SMAD-C module (counterfactual) (see Figure 1(b2)), we enhance the effectiveness of the ISR, which consider the SMAD-F only rather than the pairs (both SMAD-F and SMAD-C). This approach helps extract more robust emotional features. The causal relationships in the proposed method can be expressed as following:

Ys,x(I) contains confusing emotional guidance information. By assuming different variables, counterfactual interventions can be achieved [Pearl and Mackenzie, 2018; Pearl, 2022]. By introducing a structural causal graph, we can directly manipulate the values of several variables to analyze causal relationships and observe their effects [VanderWeele, 2015; Rao et al., 2021]. In the ISR tasks, we conduct counterfactual intervention do(S = s∗ ) as potential conditions [VanderWeele, 2015] for the affective region (s∗ ) (counterfactual) to replace the affective region map (S) (factual), as illustrated in the potential generation for SMAD-C in block (b3) of Figure 1(b), and in the block (c2) of Figure 1(c). In practice, we use random attention allocation, uniform attention allocation, and reverse attention allocation as counterfactuals [Rao et al., 2021]. Subsequently, response modulation affects human emotional perception outcomes after the emotional response tendencies have already been elicited. To simulate this stage, we leverage the cross-spatial pooling strategy, dividing the input feature map's channels into multiple categories, which is shown as Equation (4). For each classification category cl, the feature maps Sout,i (i = 1, 2, . . . , l) have all pixels in each channel averaged by the global average pooling operation (GGAP ), thereby achieving dimensionality reduction and information aggregation of the input feature map to produce the final output S = fcs(Sout).

As shown in the structural causal graph, we combine the pseudo sentiment maps S and X for the final prediction P :

where t(·) represents the concatenate operation, and ffully represents the fully connected layer, used to calculate the prediction scores for different emotional categories. The effect of learned Emotion Regulation on actual emotion prediction can be represented by the addition or sum of the factual thinking prediction, Y (from SMAD-F) and its counterfactual prediction Y ′ (from SMAD-C).

3.4 Response Modulation (b3)

The total effect (TE) [Pearl, 2014; Yang et al., 2024], YTE, which is the final emotion response of the CTERNet can be obtained using the following equation:

We utilize YTE as a supervision signal to guide the generation of affective regions. In combination with the original losses in region-based ISR, the total loss function can be expressed as shown in Equation (7).

where L(·) represents cross-entropy loss, and Ylabel denotes the ground truth. The loss L(S, Ylabel) calculates the difference between the generated affective regions S and the true emotion labels.

4 Experiments

4.1 Datasets and the EvaluationMetric

The experiments were conducted on datasets of various scales, including Flickr and Instagram (FI) [You et al., 2016], Emotion6 [Peng et al., 2016], ArtPhoto [Machajdik and Hanbury, 2010; Yang et al., 2018b], and Twitter II [Borth et al., 2013; Zhang et al., 2024].

Like all other ISR works, we utilize classification accuracy for evaluation.

4.2 Implementation Details

In terms of model architecture, we utilized a Res2Net-101 [Gao et al., 2019] pre-trained on the ImageNet dataset, implemented using the PyTorch framework, to parameterize the feature extraction network. The output of the last linear layer was replaced to generate the task-specific number of neurons for ISR prediction. For dataset processing, we set the input image size to 448×448. Data augmentation on the training set included random cropping, random horizontal flipping, and image normalization. For the test set, we applied center cropping and image normalization. We used the SGD optimizer with a learning rate of 0.0001 and a momentum of 0.9. The learning rate was decayed by a factor of 0.1 every 10 epochs. All experiments were implemented on NVIDIA Geforce RTX 2080 Ti GPUs.

4.3 Comparisons onMulti-class Datasets

We compared the CTERNet with classic and state-of-the-art ISR models on multi-class datasets, as shown in Table 1. WSCNet is a classic weakly-supervised ISR method that has inspired much work in the field [She et al., 2019]. Yang's model uses a semantic embedding space to explore imageemotion relationships [Yang et al., 2023b]. MSRCA introduces a multi-level sentiment region correlation analysis module and leverages a Transformer encoder for rich emotion interaction [Zhang et al., 2022b]. CausVSR models ISR based on Emotion Regulation Theory, using a frontdoor adjustment to reduce contextual confounding [Zhang et al., 2024]. Compared to these models, CTERNet achieved the highest accuracy on the FI dataset (72.71%), outperforming MSRCA and CausVSR by 0.11% and 0.14%, respectively. On the Emotion6 dataset, it performed well (60.49%), slightly behind Yang's model (60.54%), which employs multi-task learning and a fusion strategy to better utilize limited samples in smaller datasets. In contrast, CTERNet emphasizes causal relationships through Emotion Regulation strategies and counterfactual thinking, making it more effective on complex, larger-scale datasets by leveraging more information for causal analysis.

Furthermore, we present the confusion matrices for the FI and Emotion6 multi-class datasets in Figure 2 and Figure 3 to analyze the performance of CTERNet. The matrices show that the model performs well in recognizing most emotions on both datasets. In the FI dataset, significant misclassifications occur between similar emotions, such as Anger and Disgust, and Excitement and Awe, with Disgust being the hardest to recognize. This may be due to the greater diversity in how Disgust is expressed across contexts, making it more challenging for the model. In the Emotion6 dataset, overall performance is strong, but Anger shows lower accuracy, with misclassifications primarily between Anger and Sadness. This could be due to visual similarities, as training samples for both emotions often feature black-and-white images with dark tones.

4.4 Comparisons on Binary-class Datasets

We also conducted experiments focused on sentiment polarity analysis. Besides the ArtPhoto and Twitter II datasets, which inherently include sentiment polarity classification, we converted the original labels of Emotion6 and FI into two polarities: positive and negative. As shown in Table 2, CTERNet demonstrated excellent performance across all polarity datasets, showcasing its robust capabilities in polarity sentiment classification tasks. In comparison, while other methods performed well on certain datasets, none surpassed the overall performance of CTERNet.

4.5 Ablation Studies

By systematically conducting ablation studies on components of the CTERNet model that simulate Emotion Regulation Theory, we observe the impact of each module on the overall performance. These studies are conducted on the multi-class datasets and the binary classification datasets.

Impacts of the CTERNet Framework Structure. Table 3 presents the ablation study results for each regulation process within the CTERNet framework: i) The significant performance improvement when incorporating the SMAD-F module into the vanilla model indicates that our simulation of situation modification and attention allocation provides valuable emotional semantics, effectively aiding CTERNet in continuously identifying and adjusting affective regions. ii) Adding the SMAD-C module, based on counterfactual thinking, further enhances the results. Comparing facts with counterfactual assumptions helps reduce bias in the affective region generation process. Additionally, this demonstrates that the total causal effect is essential for improving de-biasing gains.

Impacts of the Potential Generation in SMAD-C. We applied three different strategies [Rao et al., 2021], including random attention allocation, uniform attention allocation, and reverse attention allocation, to the attention allocation phase of Emotion Regulation as counterfactual scenarios. The results, shown in Table 4, indicate that random attention allocation outperforms the other strategies. However, the performance of reverse attention allocation is suboptimal, even falling below the vanilla model's performance on the ArtPhoto and TwitterII datasets. We analyzed that the random attention allocation strategy can provide diverse attention patterns in different contexts, capturing more emotional information and contextual variations. This diversity aligns better with natural human attention, helping the model to more comprehensively understand and regulate emotions. In contrast, the reverse attention allocation strategy may disrupt the organization and expression of emotional features, leading to reduced focus on key emotional characteristics and thereby weakening the model's ability to regulate emotions.

4.6 Visualization

We conducted a visual analysis on Emotion6 using different affective region generation methods. CAM model [Zhou et al., 2016] can highlight the regions of the image that the model focuses on when making classification decisions. Similarly, saliency object detection method of Chen et al. [Chen et al., 2020] can identify the most prominent areas in an image, which are the parts most likely to attract human attention first. However, the ISR task differs from ordinary image classification tasks. ISR not only focuses on objects in the image but also involves more abstract and complex features that guide its decisions. As shown in Figure 4, CTERNet exhibits more precise targeting when the image content is simple and features a single main subject (e.g., the first and third rows). In the first row, while all tested models detect relevant affective regions, CTERNet considers both the foreground toad and the background environment. Similarly, in the third row, CTERNet captures both the person and the seaside environment, unlike the saliency-based method, which focuses only on the foreground object. Additionally, CTERNet avoids the interference in the upper affective region generated by DCNet, providing more accurate delineation of affective regions. For more complex images with multiple targets or lower contrast (e.g., the second, fourth, and fifth rows), the saliencybased method struggles to identify significant regions accurately. In contrast, CTERNet not only detects clear affective regions but also identifies finer details compared to other models. This highlights CTERNet's superior ability to handle complex scenarios, making it highly effective for ISR tasks.

5 Conclusion

This paper simulates the Emotion Regulation Theory by modeling the entire process of image sentiment recognition based on human causality-driven mechanisms. Specifically, we focus on counterfactual thinking as a crucial strategy in cognitive change. By leveraging counterfactual causal relationships, we jointly model the stages of situation modification, attentional deployment, and cognitive change in Emotion Regulation Theory. This approach allows us to learn both affective regions (factual) and other potential affective regions (counterfactual), simulating humans' diverse emotional arousal processes in different situations. During the response modulation phase, we obtain emotional prediction results based on interpreting various emotional scenarios, comparing the effects of factual and counterfactual information on the final emotional prediction. This process also quantifies the quality of the affective regions. Extensive experiments on public image sentiment recognition datasets demonstrate the performance of the proposed model.

Acknowledgments

This research was supported by National Natural Science Foundation of China under Grant Nos. 61871186, 61771322.