TFCD: Towards Multi-modal Sarcasm Detection via Training-Free Counterfactual Debiasing

1 Introduction

Due to the rise of social media platforms such as X and Facebook, multi-modal sarcasm detection (MSD) has garnered substantial research interest in recent years. Formally, MSD aims to recognize the sarcastic sentiment in multimodal social posts [Cai et al., 2019], which typically refer to textual paragraphs accompanying images. Unlike the traditional sarcasm detection which primarily focuses on textual cues [Riloff et al., 2013; Poria et al., 2016; Zhang et al., 2016], the key objective of MSD is to effectively identify inter- and intra-modal inconsistencies in the expression of sentiment within a coupled image text pair.

∗ Corresponding Author

Towards this goal, a series of models have been proposed for MSD. Therein, early approaches concentrated on leveraging entire image and text features for incongruity learning [Pan et al., 2020; Xu et al., 2020]. However, these methods neglected the fact that sarcastic information might be expressed in specific segments of the text and particular regions of the image. Motivated by this, recent studies tried to employ graph neural networks to explore the local semantic relationships within textual and visual modalities [Liang et al., 2021; Liang et al., 2022]. More recently, [Qin et al., 2023] employed the powerful pre-trained CLIP model [Radford et al., 2021] to detect different sarcasm cues captured from multiple perspectives, achieving state-of-the-art results.

Despite the significant advancements achieved by existing MSD works, we argue that they remain susceptible to capturing harmful dataset biases. These biases can mislead the models, leading to inaccurate predictions. Based on our observations, we group these biases into two main types: (1) Statistical label bias. As depicted in Figure 1(a), the MSD training set is dominated by the sarcastic class, which comprises 56.4%, compared to 43.6% for the non-sarcastic class. Numerous studies [Lin et al., 2022; Pan et al., 2023] have demonstrated that models trained on such imbalanced data tend to be biased towards predicting the majority class. (2) Sarcasmless word bias, which refers to the strong associations that may develop between sarcasmless specific words and the sarcastic class. As shown in Figure 1(b), the word 'weather' is highly likely to be correlated with the sarcastic class. In this context, the conventionally-trained models tend to unfairly categorize utterances containing these specific words, relying on biased statistical information rather than intrinsic semantics [Zhu et al., 2022; Zhou et al., 2023].

To mitigate biases, various debiasing strategies have been proposed, which can be mainly categorized into two types (cf. Table 1): (1) Data-level debiasing strategies (e.g., resampling [Qian et al., 2020] and generating counterfactual samples [Wang and Culotta, 2021]), aim to balance the training set but are often limited by the additional data handling [Zhang et al., 2020] and extra training time. (2) Modellevel debiasing ones (e.g., reweighting [Zhang et al., 2020] and counterfactual reasoning [Sun et al., 2022; Chen et al., 2023]), adjust category influence during training but require careful strategy selection and retraining from scratch.

Unlike current debiasing works that rely on data balancing manipulations or modifying then retraining the model, we propose a Training-Free Counterfactual Debiasing framework termed TFCD for MSD. Concretely, we first formulate the procedure of the MSD task via a proposed causal graph. Then, we employ a masking mechanism [Qian et al., 2021] to generate counterfactual examples from the original input data to identify detrimental biases in the trained models. In this case, the biases (including statistical label bias and sarcasmless word bias) are essentially unintended confounders that mislead the models to learn the spurious correlations, and need to be eliminated. We mitigate these two extracted biases using conceptually simple yet empirically robust element-wise subtraction operations on prediction distributions, without any retraining. We comprehensively evaluate the effectiveness and superiority of the proposed TFCD on two MSD benchmarks. Extensive experiments and analyses demonstrate that TFCD can significantly and consistently improve existing baselines, achieving a new state-of-the-art.

Overall, the main contribution of this paper is three-fold:

• To our best knowledge, this is the first work to investigate debiasing in MSD. We identify that certain biases (i.e., statistical labels bias and sarcasmless word bias) act as confounders, misleading models to learn the spurious correlations for prediction.
• TFCD, a training-free counterfactual debiasing framework for MSD that mitigates statistical label bias and sarcasmless word bias, to facilitate a fair contribution of diverse samples and contexts to sarcasm detection.
• Extensive experiments verify that the proposed TFCD

can facilitate existing models to achieve unbiased predictions. More encouragingly, TFCD is model-agnostic and can be seamlessly integrated into any existing MSD approach to boost baseline performance.

2 RelatedWork

Multi-modal Sarcasm Detection. With the rapid popularization of social media platforms, multi-modal sarcasm detection (MSD) has garnered increasing research attention in recent years [Zhu et al., 2024a]. [Schifanella et al., 2016] first used both textual and visual information to tackle the MSD task. [Cai et al., 2019] created an MSD benchmark based on Twitter and proposed a hierarchical fusion model. Thereafter, [Xu et al., 2020] and [Pan et al., 2020] captured both intra-modality and inter-modality incongruities based on global textual and visual features, respectively. [Liang et al., 2022] and [Liang et al., 2021] utilized cross-modal graphbased models [Zhu et al., 2024b] for drawing incongruous relations across local multi-modal features. Most recently, [Qin et al., 2023] leveraged the power of pre-trained CLIP [Radford et al., 2021] to perform MSD and achieved cutting-edge results. Despite these advancements, unintended dataset biases have been neglected in previous MSD studies, which limits the performance of MSD models.

Debiasing Strategy. To address dataset bias, several debiasing strategies are proposed to enhance the robustness and reasoning ability of models [Xin et al., 2023], which can be roughly divided into two groups: (1) Data-level debiasing strategies include data balancing, data resampling as well as data augmentation [Qian et al., 2020; Wang and Culotta, 2021]. (2) Model-level debiasing strategies include using unbiased embeddings [Sun et al., 2022], adjusting thresholds [Kang et al., 2019], and reweighting [Zhang et al., 2020] techniques. However, the former results in additional manual costs for data manipulations. On the other hand, the latter necessitates a meticulous selection of balancing techniques and retraining. Within the multi-modal research community, various debiasing efforts have emerged on different tasks such as multi-modal fake news detection [Chen et al., 2023], and multi-modal sentiment analysis [Sun et al., 2022]. Unfortunately, their methods still necessitate retraining. There is contemporaneous work [Jia et al., 2024; Yang et al., 2024a; Yang et al., 2024b] that also proposed debiasing methods for MSA and MSD. Besides, several training-free methods have been proposed [Qian et al., 2021; Tu et al., 2023] and only perform debiasing on textualmodality data. In this work, we extend the existing trainingfree debiasing scheme to multi-modal setting and make the first attempt to achieve a training-free debiasing strategy for MSD, which has superiority over complex retraining modules employed in previous approaches.

3 Preliminaries

3.1 Task Formulation

Given a sample xi from the training set, the objective of MSD task is to determine whether the sample implies any sarcasm by learning a model f(·) using the text Ti and corresponding image vi. This conventional training procedure is represented as ˆ yi = f(Ti, vi |Θ) ∈ {0, 1}, where ˆ yi = 1 indicates the sample is sarcastic and vice versa; Θ represents the learnable model parameters. For simplicity, we temporally omit the superscript i that indexes the training samples in §3.2 and §4.1.

3.2 Biased Training

Let Ht and Hv denote the encoded representations of the textual modality (T) and visual modality (V ), respectively. As mentioned in §1, existing MSD models focus on designing sophisticated architecture or mechanisms to extract sarcastic cues from entire or local text and image features:

where M denotes the fused multi-modal feature and fm(·) denotes the fusion strategy that depends on a certain MSD model. Then, we use feedforward propagation to predict examples and backward propagation to update the learnable parameters of the model in an end-to-end fashion as shown in Figure 2(a). Following previous works, we employ the crossentropy loss to optimize the MSD task:

4 Training-Free Counterfactual Debiasing

This section elaborates on the details of our proposed TFCD framework, whose architecture is shown in Figure 2(b).

4.1 Biases Extraction

During inference, the conventionally-trained models in §3.2 make predictions via the feedforward propagation to obtain the probability distribution. However, the prediction is easily affected by unplanned confounders [Pearl and Mackenzie, 2018], which may produce statistical label bias and sarcasmless word bias. Aiming to obtain unbiased prediction, our objective is to debias only during inference by blocking the spread of biases from training. Towards this goal, we propose training-free counterfactual debiasing to extract the two biases captured by the trained models and mitigate them.

In the following, we first formulate a tailored causal graph for MSD. Then, we elaborate on how to extract the two biases from the trained model based on the constructed graph.

MSD Causal Graph. The causal graph [Pearl et al., 2016] is a probabilistic graphical model used to describe how variables interact with each other, expressed by a directed acyclic graph G = {N, E} consisting of the sets of variables N and the cause-and-effect correlations E between two nodes (variables). For instance, X→Y indicates that X is the cause of the effect Y, meaning that the value of Y is influenced by X.

As illustrated in Figure 3(a), there are four variables involved in the MSD causal graph, which are textual feature Ht, visual feature Hv, multi-modal feature M, and predictions o. As such, we can obtain the factual casual graph for §3.2. Note that our causal graph applies to a variety of MSD

methods, since it is highly general, imposing no constraints on the detailed implementations.

Statistical Label Bias. As illustrated in Figure 3(b), there exist unplanned confounders Z, which is the cause of the spurious correlations between the model input H{t,v} and predictive logits o. Such confounders may occur due to the unbalanced label distribution (i.e., 'Sarcastic' dominates the training data over "Not Sarcastic"). Therefore, the trained models tend to unfairly assign outputs to 'Sarcastic' category based on biased statistical information, which is not unreliable for MSD. To decouple the spurious correlation, we use the backdoor adjustments [Pearl et al., 2016] with do-calculus operation to calculate the corresponding intervention distribution:

where M∗ can take any form as long as they are no longer influenced by Z, effectively breaking the connection between H{t,v} and o. To extract statistical label biases, in the causal intervention operation, M∗ is intervened under no-treatment condition where T and V have not been accessible. Note that neural models cannot deal with no-treatment conditions where the inputs are void. Therefore, we assume that the model will randomly guess with equal probability (i.e., uniform distribution assumption) [Niu et al., 2021] to guarantee a safe estimation. As MSD models cannot observe any words and images after the intervention, the predictive logits o∗ only reflect the adverse impact of statistical label bias.

Sarcasmless Word Bias. We further utilize a partiallyblindfolded counterfactual input where some words are masked to distill the sarcasmless word bias from the trained model. Specifically, our goal is to retain only sarcasmless words (e.g., the spurious 'weather'-to-'sarcastic' mapping) in the utterance to measure their potentially negative influence. We first utilize pysentiment library 1 to construct a main-content words dictionary Ds. By masking maincontent words, we deliberately expose any spurious correlations between sarcastic categories and sarcasmless words:

Here, H†t denotes counterfactual word embedding where the main-content words are masked. To be specific, the mask operation process can be formulated as follows:

1 https://pypi.python.org/pypi/pysentiment

where [mask] is used to hide a single token belonging to Ds in the input sentence Ht. Meanwhile, H†v should similarly represent an unseen feature. In this work, we use all-zero vectors to initialize it [Sun et al., 2022]. In this case, the predictive logits o† reflect the pure influence of sarcasmless words to the trained biased model.

4.2 Debiased Inference

After obtaining two biases, the final objective is to leverage the direct effect from multi-modal representation Mi to predictive logits oi for each sample i to facilitate unbiased prediction. This process can be formalized using the conceptually simple yet empirically robust element-wise subtraction:

where α and β are two hyper-parameters. Note that we ascertain two adjustable scaling factors which are optimized to the model's performance on the validation set, since different biases have diverse impacts on the final prediction.

5 Experiments

5.1 Experiment Setup

Datasets. We conduct experiments on two datasets: MMSD [Cai et al., 2019] and MMSD2.0 [Qin et al., 2023]. MMSD is derived from English tweets. Thereinto, tweets with some special hashtags are positive examples and those without such hashtags are negative examples. MMSD2.0 is upgraded from MMSD. The providers removed misleading cues such as hashtags and emoji words, while it does not address the inherent label bias and word bias present in the dataset, which still influence the model's learning process and potentially lead to biased predictions. The statistics of these two datasets are shown in Table 2.

EvaluationMetrics. Following previous works [Liu et al., 2022; Qin et al., 2023], we adopt accuracy (Acc.), precision (P), recall (R) and micro-average F1 score (F1) to evaluate the model performance. Note that higher numerical values signify better performance across all metrics.

ComparisonModels. In our experiments, we choose seven representative MSD models: D&R Net [Xu et al., 2020], InCrossMGs [Liang et al., 2021], HFM [Cai et al., 2019], Att-BERT [Pan et al., 2020], CMGCN [Liang et al., 2022], HKE [Liu et al., 2022] and Multi-view CLIP [Qin et al., 2023]. In detail, to demonstrate the effectiveness of the proposed TFCD, we compare the performance of five reproducible models with and without the TFCD framework.

ImplementationDetails. The proposed TFCD and five reproducible models are implemented on PyTorch [Paszke et al., 2017]. All experiments are conducted on Nvidia Tesla V100 GPUs. We have utilized grid search to determine the optimal values for the parameters α and β on the validation set. The grid search is performed with a step size of 0.1 and a range spanning from 0 to 1. For a fair comparison, the training settings (e.g., loss function, batch size, learning rate strategy, etc) of these models are consistent with the details reported in their original papers. The results reported in our experiments are the average scores from five random runs on the test set. Please refer to the Appendix for more details.

5.2 Main Results

The main results of TFCD and baselines are shown in Table 3, from which we have the following observations:

(1) The baselines with our TFCD significantly outperform their original counterparts across all evaluation metrics on both datasets. This validates the superior generalizability ability of our framework over existing methods. On the other hand, the better results show that these biases are ignored during previous MSD studies, which further supports our claims and motivation. (2) The improvements on MMSD are much sharper than MMSD2.0. We hypothesize that this is due to the specific characteristics between datasets: MMSD2.0, by correcting erroneous labels in MMSD, has somewhat mitigated the issue of label imbalance. Nonetheless, our TFCD consistently achieves gains, which illustrates the robustness of our debiasing framework across varying datasets.

5.3 Ablation Study

We select two representative models, i.e., HFM and Multiview CLIP, to evaluate the contribution of each component in the proposed TFCD framework. The ablation studies on both datasets are reported in Table 4, where all the improvements are statistically significant, as evidenced by the paired t-tests with a p-value < 0.05. And we have the following takeaways:

(1) Removing any component results in a decrease across all metrics on both datasets, which verifies the effectiveness of the proposed label and word debiasing. This is because label debiasing introduces a global offset, while word debiasing contributes to a local one to 'move' in the predicted space, which renders the trained models 'blind' to potentially harmful biases present in the observed data, allowing them to focus solely on the core content of each sample for inference. (2) The improvements in word debiasing are more pronounced. This could be attributed to the fact that trained models typically utilize word-level information for inference, which may inevitably utilize sarcasmless words that are potential biases. Thanks to word debiasing, TFCD addresses spurious correlations introduced by these words to some extent.

Effect of Label Debiasing. One of the core contributions of our work is to achieve label debiasing, where the uniform distribution assumption is introduced under the no-treatment conditions to obtain the intervened outcomes. To verify its effectiveness, we substitute it with two candidate assumptions: Random denotes M∗ is learned without any constraint and Prior denotes that M∗ obeys the prior distribution of the training set. As shown in Table 5, we find that both Random Metric Full w/o M w/ All M w/ Rand M

84.17

80.95

88.89

87.41

and Prior even perform worse than the baseline counterparts. We attribute this to the fact that the uniform distribution assumption guarantees a safe estimation for the label biases. Effect ofWord Debiasing. To further investigate the effectiveness of word debiasing, we devised three distinct variants: word non-masking (w/o M), all masking (w/ All M), and random masking (w/ Random M). From the results in Table 6, the decreased performance confirms the effect of selectively masking the main-content words. The proposed TFCD enables the baseline to more effectively identify and eliminate

spurious correlations introduced by sarcasmless words, while other masking strategies tend to intertwine with the statistical shortcuts of main-content words to varying degrees.

5.4 Method Analysis

Applicability across Pre-trained Backbones. A natural question that arises is whether our model is effective for diverse pre-trained backbones. To answer the question, we conduct experiments with four variants of CMGCN with and without our proposed TFCD by using different text and image encoders. From the results in Table 7, we observe that the gain from TFCD increases as more advanced pre-trained backbone networks are used. This proves the performance of TFCD doesn't rely on a specially chosen backbone.

Sensitivity of Hyper-parameters. The hyper-parameters α and β in Eq.(6) indicate the degree of subtraction of labeldebiased predictive logits o∗ and word-debiased predictive logits o† from biased predictive logits o, respectively. According to §5.1, we evaluate the scale range setting α, β ∈ [0.0, 1.0] with a step size of 0.1 as shown in Figure 4. We find that as the coefficient incrementally increases, there is a corresponding increase in accuracy. Optimal performance for each debiasing strategy is achieved at distinct points (0.6 and 0.4 for label and word debiasing in practice, respectively), beyond which a downward trend emerges. Furthermore, we find that label debiasing exhibits relatively mild, whereas word debiasing demonstrates heightened sensitivity to coefficient changes, and requires more careful tuning.

Comparison with Other Debiasing Strategies. We also conduct comparison experiments to verify the proposed debiasing framework against other debiasing strategies, and the Table 7: Accuracy performance of using different pre-trained backbones for CMGCN [Liang et al., 2022] on two datasets. "T" and

"V " denote the textual and visual modality, respectively.

results are reported in Table 8. We can find that our framework outperforms typical debiasing strategies, from data manipulation to model balancing. Note that these two debiasing approaches require extra manual or training costs. Whereas, the proposed framework works only in inference and can thus be employed on the previously already trained models, which can serve as a powerful, 'data-manipulation-free' and 'modelbalancing-free' weapon to enhance current MSD baselines.

Robustness under Low-resource Scenario. To further explore the effectiveness of TFCD under the low-resource scenario [Chen et al., 2024b], we conduct experiments following [Qin et al., 2023] to utilize various quantities of training samples including 10%, 20% and 50%. From Figure 6, we observe that Multi-view CLIP with our TFCD, consistently surpasses its baseline counterpart under low-resource scenarios. We attribute this to our proposed debiasing strategies (especially label debiasing), the model can remove spurious correlations even with varying proportions of positive and negative training samples. This demonstrates the robustness of our TFCD against inconsistent distributions of training and test samples, which achieves considerable improvements.

Generalizability on Sentiment Analysis. To verify the generalizability of our TFCD on other tasks, we conduct preliminary experiments on multi-modal sentiment analysis, where we select two representative models (i.e., MulT [Tsai et al., 2019] and DMD [Li et al., 2023]). The results are reported in Table 9. We find that models with TFCD consistently boost performance by approximately 1% over their vanilla counterparts, which demonstrates the generalizability of our TFCD. We refer readers to Appendix for more details.

(a) Testing Sample from MMSD

(b) Testing Sample from MMSD2.0

Label: Sarcastic

Not Sarcastic Sarcastic

Figure 5: Case study on MMSD (a) and MMSD2.0 (b). "Label/word distribution" denote the distribution of label and sarcasmless word (marked red within the sentence) coming from the training set, where the blue and yellow colors denote probabilities of the non-sarcastic and sarcastic labels, respectively. The bars at the bottom demonstrate the prediction distribution of the best baseline Multi-view CLIP with and without the proposed TFCDframework. Better view in color.

Model Metric Base w/ DM w/ MB w/ TFCD

Acc. (%) 83.58 83.95 84.20

84.63

F1 (%) 80.35 80.64 81.06

81.48

Mul-CLIP† Acc. (%) 88.29 88.72 89.04

F1 (%) 86.84 87.17 87.65

89.57

88.13

MOSI

Acc-2 (%) Acc-7 (%) Acc-2 (%) Acc-7 (%)

52.76

83.95

41.47

53.84

85.01

42.88

54.35

85.92

45.87

55.63

86.87

46.94

Table 9: Performance on two multi-modal sentiment analysis datasets. Results with '†' stand for the model we implemented.

5.5 Case Study

In Figure 5, we select one representative example from each dataset to show the performance of the model with and without the TFCD. For instance, in Figure 5(a), the vanilla baseline is misled to predict sarcastic because the MMSD is dominated by the sarcastic class (label bias) and the 'weather' word within the sentence is mostly associated with the sarcastic class (word bias). Applying counterfactual debiasing, TFCD corrects the label and word biases in the model's prediction. In Figure 5(b), the vanilla baseline is misled to predict not sarcastic due to the serious 'beautiful' word distribution in not sarcastic class. Thanks to the proposed TFCD, the model now results in the correct prediction.

6 Conclusion

This paper proposed a training-free debiasing strategy termed TFCD to reduce the harmful bias of statistical label and sarcasmless words for multi-modal sarcasm detection (MSD).

Concretely, TFCD disentangled the causalities among variables via a tailored causal graph and presented a biases extraction module to extract the adverse effect caused by the two biases. These biases were then mitigated by element-wise subtraction to achieve debiased inference. Numerous experiments proved that TFCD could consistently improve existing baselines. The model-agnostic and training-free TFCD undoubtedly has superiority over complex retraining modules employed in previous approaches.

Limitations and Future Work. Although our TFCD has demonstrated promising outcomes, it can still benefit from the following two aspects: (1) Further exploration of the debiasing technique on visual component [Chen et al., 2024a]. (2) In scenarios involving samples with highly balanced classes, the impact of label bias becomes negligible. Future work will extend our framework under more challenging scenarios (e.g., out-of-distribution and long document) for MSD.