Multimodal Inverse Attention Network with Intrinsic Discriminant Feature Exploitation for Fake News Detection

1 Introduction

The rapid evolution of information technology and social networks has shifted news consumption from traditional media to online platforms, enabling individuals to act as publishers and accelerating the delivery of fast, diverse, and personalized information. However, it has also enabled the spread of misinformation, frequently driven by misinterpretations, exaggerations, or deliberate falsifications, which pose significant risks to social security and stability [Shu et al., 2017]. Moreover, advancements in GenAI models [Ouyang et al., 2022;

Terrorists searched the hall with long-range weapons in their hands. (c)

New species of fish found in Arkansas.

(a)

Figure 1: Typical types of fake news. (a)/(b) fabricated fake news where either the text or image is authentic while the other is manipulated to create fake content. Specifically, in (a), the subject in the image features a fish's body alongside a pig's ears and snout, highlighting categorical inconsistencies between regions; In (b), the overall semantics of the news text conflict with specific phrases when placed in the realistic context. (c) mismatched text and images from unrelated real news sources, creating inconsistent narratives.

Li et al., 2022] have further lowered the barriers to creating and distributing fake news, exceeding the capacity of human judgment to identify disinformation effectively [Wang et al., 2024b]. While relevant security departments have established verification mechanisms to combat fake news, the vast amount of content has made manual verification increasingly unfeasible [Zhou and Zafarani, 2020]. Consequently, the development of automated fake news detection techniques has emerged as a critical research focus to safeguard the reliability and integrity of public information.

Research on fake news detection has evolved from relying exclusively on text-based approaches [Yu et al., 2017; Wu et al., 2024] to incorporating multimodal content, driven by the increasing prevalence of multimedia content on social media platforms [Yu et al., 2022]. However, the lack of a systematic categorization of news content poses a substantial barrier to advancing effective multimodal fake news detection methods. To address this limitation, this study introduces a novel perspective that classifies multimodal fake news into two distinct categories: a) fabricated fake news encompasses cases where textual and visual content appear consistent but involve deliberate manipulations or synthetic generation in one or more modalities. It presents a more formidable detection challenge, particularly given the rapid advancements in generative AI technologies [Zhao et al., 2023; Li et al., 2023], as demonstrated in Figure 1 (a) and (b);

b) mismatched fake news refers to instances where textual and visual content are semantically or logically inconsistent. These inconsistencies constitute a form of misinformation capable of distorting public perception and influencing behavior, as illustrated in Figure 1 (c).

On the one hand, detecting fabricated fake news is highly challenging, yet research in this area remains limited. While some methods leverage external knowledge to improve content understanding, they often rely heavily on the timeliness and relevance of such knowledge, introducing potential biases that may undermine performance [Wu et al., 2023; Zhang et al., 2024b]. Therefore, it derives the main research question of our study, RQ 1: How to exploit the intrinsic discriminative information within each modality to identify fabricated fake news without relying on external knowledge? On the other hand, despite recent advancements in detecting mismatched fake news, current methods still face notable limitations. These methods can be broadly categorized into auxiliary task-based approaches [Zhou et al., 2020; Chen et al., 2022] and attention mechanism-based approaches [Qian et al., 2021; Wei et al., 2022]. The former quantifies the similarity between modality representations in a shared subspace or fuses multimodal representations, while the latter captures cross-modal feature similarities to identify inconsistencies. However, both approaches often fail to extract explicit inconsistency or distinguish the inter-modal relationships into consistency and inconsistency features, leading to confusion in the model. Thus, our second research question can be summarized as, RQ 2: How to effectively detect inconsistencies in mismatched fake news while avoiding the overemphasis on cross-modal similarities and implicit pseudo-consistency?

To address these research questions, we propose the Multimodal Inverse Attention Network (MIAN), which leverages unimodal and multimodal intrinsic discriminative information through hierarchical interaction and explicit inconsistency without relying on external knowledge. Specifically, to capture the intrinsic discriminant features of each modality, we build a hierarchical learning module consisting of two aspects: 1) Local-to-Local, which focuses on the contextual associations within the local features; and 2) Localto-Global, which models the relationships between global and local features. This module fully explores unimodal information from news text and images and produces refined modality-specific representations to aid MIAN in detecting fabricated fake news, i.e., targeting for RQ 1. To fuse multimodal features, we propose a cross-modal interaction module that facilitates interaction between the enhanced local features from different modalities, enabling effective detection of mismatched fake news, i.e., targeting for RQ 2. Additionally, both the hierarchical learning module and the crossmodal interaction module equip our designed inverse attention mechanism, allowing MIAN to extract explicit inconsistent features that are crucial for fake news detection. Overall, the contributions of this research can be summarized as:

• We propose MIAN, a multimodal framework that leverages intrinsic discriminant information to improve news content-based fake news detection while offering a novel perspective for capturing discriminative relationships.
• We design a hierarchical learning module that enhances unimodal features through hierarchical relationships and a cross-modal interaction module that enables deep integration of inter-modal dependencies, improving both intra- and inter-feature learning.
• We present an inverse attention mechanism to detect inconsistencies across multiple levels, enabling the model to effectively identify diverse fake news. Extensive experiments demonstrate that MIAN outperforms state-ofthe-art methods in accuracy.

2 RelatedWork

2.1 News Content-basedMethods

The lack of correlation between textual and visual content in news is a key characteristic of certain types of multimodal fake news. Consequently, several studies have focused on measuring multimodal consistency to verify the credibility of news. Zhou et al. [Zhou et al., 2020] utilized a pre-trained image captioning model to convert images into textual descriptions which achieved cross-modal semantic space alignment, then assessed the multimodal consistency between the original text and the generated captions. Similarly, Xue et al. [Xue et al., 2021] employed shared weights to enforce the alignment of textual and visual representations within a unified semantic space, calculating the similarity of the transformed multimodal representations. To mitigate the semantic gap between modalities, Jiang et al. [Jiang et al., 2023] leveraged CLIP [Radford et al., 2021] to extract features from news text and images, subsequently using cosine similarity to guide multimodal fusion. Chen et al. [Chen et al., 2022] introduced cross-modal contrastive learning as an auxiliary task, and then used the Kullback-Leibler (KL) divergence to measure the ambiguity score between the latent distributions of text and image sampled from the autoencoder. Qi et al. [Qi et al., 2021] measured multimodal entity inconsistency by calculating the similarity between entities in the news text and the corresponding visual content.

Under limited available information, some methods refine unimodal features and employ hierarchical cross-attention mechanisms [Lu et al., 2019] to model inter-modal relationships. Qian et al. [Qian et al., 2021] extracted text features at different hierarchical levels and fused them with visual features using a contextual transformer to learn cross-modal contextual features and supplementary information. Wu et al. [Wei et al., 2022] extracted spatial and frequency domain features from images and progressively fused them with textual features through stacked cross-attention mechanisms. However, their approaches primarily focus on exploring intermodal relationships. Even though a few methods attempt to enhance unimodal feature representations, they overlook intra-modal interactions, which are crucial for detecting fake news, particularly fabricated news. Moreover, most existing methods model high-level semantic interactions through implicit pseudo-consistency, making it challenging to explicitly capture cross-modal inconsistencies.

Hierarchical Learning Module

2.2 External Knowledge-basedMethods

Due to limitations in text length or the emergence of novel terms, discrepancies in understanding news content may arise. As a result, several studies have incorporated external knowledge extracted from social media [Zhang et al., 2021], knowledge graphs [Zhang et al., 2019; Dun et al., 2021] or internet retrieval [Zhang et al., 2024a; Qi et al., 2024; Yu et al., 2020] to enhance the performance of fake news detection. Zheng et al. [Zheng et al., 2022] employed an attention mechanism to combine social graph features, composed of user and comment data, with textual and visual features of news, generating discriminative features for fake news detection. Wu et al. [Wu et al., 2023] performed the deep fusion of text and image features based on three distinct reading patterns while leveraging the relationship between news comments and content to capture semantic inconsistencies. Zhang et al. [Zhang et al., 2023] leveraged news text entities as prompts to guide a large-scale vision-language model in generating image entities that enhance the semantic knowledge of the content. Zhang et al. [Zhang et al., 2024b] converted news images and externally relevant knowledge into pure textual representations, which were then combined with the original text and fed into a prompt-based large language model to predict the authenticity of the news. While some methods enhance the representation of news content by incorporating external knowledge, this simultaneously introduces the risk of noise, which is crucial for the accuracy of fake news detection. These challenges significantly hinder the progress of multimodal fake news detection.

3 Methodology

Fake news detection based on multimodal content aims to leverage both textual and corresponding visual information to evaluate the authenticity of news. Given a dataset of fake news detection D = {X, Y }, where X = [XT , XV ] consists of the textual component XT and the visual component XV , and Y ∈ {0, 1} is the corresponding label indicating whether the news is fake (0) or real (1). The objective of the model is to learn a mapping function F : (XT ,XV ) → Y from the training set of D, which maps the input space (XT ,XV ) to the labels Y . The learned function F is then used to predict labels for the test instances.

Our MIAN aims to fully learn intrinsic discriminant features within and across modalities. The detailed architecture is illustrated in Figure 2. We first extract unimodal embeddings from the news text and images using modality-specific encoders. For each multimodal news instance, the text is represented as a sequence of m tokens T = {t1, t2, ..., tm}, where T ∈ Rm×d and each token ti ∈ Rd is a d-dimensional vector obtained from a pre-trained BERT model [Devlin et al., 2018]. Meanwhile, we employ a pre-trained ViT model [Dosovitskiy et al., 2020] to encode the image content into u visual tokens O = {o1, o2, ..., ou}, where O ∈ Ru×d . We then introduce a hierarchical learning module consisting of two components, the Local-to-Global Block and the Localto-Local Block. This module enables multi-granularity interactions to enhance unimodal representations for both text and image content. A detailed description is provided in Section 3.1. Subsequently, MIAN fuses the enhanced multimodal representations using a co-attention mechanism in the cross-modal interaction module. A more detailed description is given in Section 3.2. Additionally, to enable the model to explicitly learn inconsistency features, we design an inverse attention mechanism and integrate it into both the hierarchical learning module and the cross-modal interaction module. The specifics are discussed in Section 3.3. Finally, the enhanced modality-specific representations from the hierarchical learning modules in both the text and image branches, along with the fused representation from the cross-modal interaction module, are concatenated to form the final news representation, which is then fed into the classifier to assess the news authenticity.

3.1 Hierarchical LearningModule (HLM)

To comprehensively explore the internal relationships within news text and image representations and to produce enhanced unimodal features, we propose the Hierarchical Learning Module (HLM). After obtaining modality-specific embeddings from pre-trained encoders for text and images, HLM further models intra-modal associations at different levels from two perspectives: the Local-to-Local Block, which captures contextual interactions among local features, and the Local-to-Global Block, which establishes dependency relationships between local and global features.

Local-to-Local Block. We employ a multi-head attention mechanism [Vaswani et al., 2017] to learn consistent relationships between tokens and capture intrinsic structural features, thereby generating context-aware, refined local representations within each unimodal branch. Specifically, for the text modality, given the text embedding T , we first perform different multi-head linear transformations to derive the query, key, and value matrices: Qit = TW ill-q , Kit = TW ill-k and V it = TW ill-v , and then concatenate the outputs from the attention mechanism.

where all W are learnable parameters, and dk denotes the dimensionality. The n weighted value vectors are concatenated using [·]. Finally, a series of residual connections and feedforward layers are applied to produce the context-aware, local-to-local enhanced features.

where W tfc1 , W tfc2 and b are learnable parameters. LN(·) and ReLU(·) refer to layer normalization and activation function.

Similarly, the ViT encoder transforms the segmented image into a sequential representation of local features, enabling the application of the self-attention mechanism. Following the same procedure as described above, we derive Rll o .

Local-to-Global Block. This block refines local features by integrating global context, using global features to bridge fine-grained local information with overarching semantics. Specifically, for the visual modality, given the image embedding O, we construct the global feature to encapsulate the overall semantics of the image from two complementary perspectives. On the one hand, the mean-pooled vector

the entire image. On the other hand, we use the [cls] token gcls output from the modality-specific encoder to represent the semantic information of the salient objects in the image. Finally, these two representations are concatenated to form the global feature:

In this block, the global features serve as the query guidance vector, while the local features function as the matched vectors. Their inner product is computed to measure the similarity between the global and local features. The resulting similarity scores are normalized via a softmax function, ensuring that the attention weights across all regions sum to 1. Specifically, the attention weights reflecting the consistency between the global high-level semantic features and the wordlevel semantic features of the text are computed as follows:

where all W are learnable parameters, ho denotes the hidden state of the text attention function, and ⊙ represents the element-wise multiplication. The process of calculating the weighted relationship Rlg t between global and local information in the news text branch follows the same procedure as in the image branch.

Finally, we concatenate the enhanced modality-specific features produced by the Local-to-Local Block and Local-toGlobal Block in each modality branch to obtain the hierarchically enhanced representations. Specifically, for the text modality, the representation is Rt = [Rll t ,Rlg t ], and for the visual modality, it is Ro = [Rll o , Rlg o ].

3.2 Cross-modal InteractionModule (CIM)

To fuse multimodal representations, we propose the Crossmodal Interaction Module (CIM), which leverages a coattention mechanism to capture inter-modal dependencies and effectively integrate the hierarchically enhanced representations of news text and images. Specifically, to obtain the text features enriched with image information, given the hierarchical enhanced representations Rt and Ro from the HLM, we apply distinct multi-head linear projections Qco t = RtW co-q t , Kco t = RtW co-k t and V co t = RtW co-v t to derive the query, key, and value matrices, respectively. Due to the similarity of operations to the Local-to-Local Block and space constraints, the specific details of the multi-head attention mechanism are omitted here.

Similarly, after applying the above process, the fused features Rco o are obtained, which capture the influence of the image on the text.

3.3 Inverse Attention

Exploring intrinsic relationships between modalities remains a significant challenge in multimodal fake news detection, especially in cases involving mismatched textual and visual content. Although the widely adopted cross-attention mechanism has shown promise in modeling cross-modal interactions, it often fails to capture inconsistencies between crossmodal features effectively.

Specifically, cross-attention calculates the similarity between query and key vectors to determine which key vectors each query vector should attend to, and then applies the resulting similarity weights to compute a weighted sum of value vectors. While this mechanism effectively emphasizes similar content across modalities, it may inadvertently encourage the network to learn implicit pseudo-consistencies, leading to confusion in fake news detection. To address this limitation, we propose the inverse attention mechanism, which amplifies dissimilar features to enhance the learning of explicit inconsistency signals. The detailed explanation of the inverse attention mechanism employed in CIM is as follows:

where A is the introduced scalar matrix, which is used to subtract from the attention matrix Attco t→o The matrix Attco-ic t→o represents the inverse attention weights, where the values corresponding to inconsistent vectors are larger. ¯Rco-ic t→o denotes the desired explicit inconsistency features. Given the consistency feature ¯Rco t→o and the inconsistency features ¯Rco-ic t→o , which provide distinct information for fake news detection, we employ a gating unit to combine them effectively:

where, Wg and bg represent learnable parameters, while [·] denotes the concatenation operation. The function σ is the sigmoid function, and g is the gating weight computed during the process.

Given that inconsistencies may also emerge within the content of fake news texts or images, the inverse attention mechanism is incorporated into both the Local-to-Global and Localto-Local modules. Specifically, during the computation of Rll t , Rll o , Rlg t , and Rlg o , similar operations to those described in Equations 14 and 15 are applied. This design enables the model to capture inconsistent features at both intra- and intermodal levels, thereby improving its effectiveness in detecting various types of fake news.

Notably, since interactions are performed at the token level, the values of individual elements directly influence the computation of inconsistency weights, particularly when processing textual embeddings. To alleviate the impact of irrelevant or missing information, we introduce a positional mask. Furthermore, due to the potential ambiguity in expression arising from the sequence order of text and the positional information of objects in images, and since the attention mechanism lacks an inherent notion of sequence order, we also incorporate positional encoding. The specific formulation is as follows:

where pos, i and d represent the position index, dimension index, and the dimension of the positional encoding, respectively.

Loss function. Finally, we concatenate the enhanced modalspecific features, enriched with intra-modal inconsistency signals, and the multimodal fused features, capturing intermodal inconsistencies, to construct the news representation Rn. This representation is subsequently input into a multilayer perceptron-based classifier to predict confidence scores, ˆ y = MLP(Rn). We define the loss function L using crossentropy as follows:

4 Experiments

The experiments in this study aim to address several key challenges in fake news detection by exploring the following research questions:

• Q1: Can MIAN effectively improve the performance of fake news detection?
• Q2: Does each component contribute to improving detection?
• Q3: Can the proposed designs improve the accuracy of detecting specific types of fake news?

4.1 Configurations

Dataset. Our experiments are evaluated on four real-world datasets: Weibo17, Weibo21, GossipCop, and PolitiFact, which cover nearly all publicly available datasets in this field. These datasets used in the experiments not only cover multiple domains but also include specific domain datasets where fake news frequently occurs. The detailed descriptions of these datasets are provided in Appendix A.

Baselines. To evaluate the effectiveness and performance of the proposed method in multimodal fake news detection. The baseline methods include classic models that leverage deep neural networks for extraction such as EANN [Wang et al., 2018], Spotfake [Singhal et al., 2019], HMCAN [Qian et al., 2021] and RaCMC [Yu et al., 2025], as well as methods exploring cross-modal relationships such as CAFE [Chen et al., 2022], CMC [Wei et al., 2022], BMR [Ying et al., 2023] and MSACA [Wang et al., 2024a], and integrating external knowledge for enhanced detection accuracy such as MRHF [Wu et al., 2023]. The detailed descriptions of these methods can be found in Appendix B.

Implementation. All experiments were implemented using the PyTorch toolkit on an NVIDIA A100 GPU. We pad the sequence length of news text to 196 for all datasets, denoted as m = 196, and put the processed text into the BERT. Images were resized to 224×224 pixels and subsequently divided into u = 14×14 patches, which were then input into the vit-patch16-224. The multi-head attention used in both the Local-to-Local Block and the Cross-modal Interaction Module consists of 2 layers with 12 heads. The initial learning rate was set to 2e-6 , and we utilized the StepLR decay strategy with a decay step of 20 and a decay rate of 0.5 to help the model converge better.

4.2 Overall Performance

Table 1 presents the performance results of the proposed method and the comparison methods on the public datasets. Although our method does not achieve the highest precision or recall, it outperforms the other comparison methods in terms of accuracy and F1 score, demonstrating its superior balance and overall performance, thus resolving Q1.

EANN and SpotFake, as early classic methods, simply concatenate the multimodal features obtained from modalspecific encoders to represent news, and do not perform well. This is likely due to their failure to further explore unimodal features and the lack of cross-modal interactions. CAFE and BMR highlight the importance of unimodal features in decision-making, with BMR achieving suboptimal results on certain datasets, suggesting that unimodal news content contributes to fake news detection. HMCAN and MSACA attempt to extract multi-level unimodal features from modalspecific encoders but do not yield strong performance, indicating that when exploring unimodal features, it is crucial to consider their specific role in the fake news detection task and fully leverage the interactions between them. Moreover, HMCAN exhibits a significantly higher F1 score for real news compared to fake news, which could be due to the use of a co-attention mechanism to model cross-modal information, where the model overemphasizes modal consistency and overlooks the stronger inconsistencies present in fake news.

Additionally, we provide the t-SNE visualization of the features for real and fake news in Figure 3. Since the LII model also generates discriminative features through enhanced intra- and inter-modal methods, it serves as a comparisonmodel toMIAN. It demonstrates that the features learned by the proposed model are more discriminative.

4.3 Ablation Studies

Impact of each component. In order to address Q2, we evaluate MIAN against its variants by systematically removing specific components. Specifically, w/o intra-lg refers to eliminating the Local-to-Global Block, thereby removing the interaction between global and local features within both text and image branches. w/o intra-lg-ic and w/o intra-ll-ic refer to removing the inconsistent features generated by inverse attention at different levels within the same modality (Local-toGlobal and Local-to-Local, respectively). w/o inter-ic refers to excluding the inconsistent features in the CIM.

Table 2 presents the results of the ablation experiments comparing various method variants, demonstrating the following key findings: (1) In the multimodal fake news detection task, comprehensive and in-depth unimodal information is essential. Extracting features from text or images aids the model in understanding the news content more effectively. (2) The loss of inconsistent information at any stage significantly impacts the accuracy of the detection results, particularly for fake news. Other metrics and results on the GossipCop dataset can be found in the Appendix C.1.

Impact of different types. To further validate the effectiveness of the proposed method in identifying both fabricated news and mismatched news, we introduce the Fakeddit dataset [Nakamura et al., 2019], which includes multiple fake news categories. In this dataset, label 0 represents real news, label 4 corresponds to False Connection, and label 5 denotes Manipulation Content, all of which align with the focus of our study. This experiment answers Q3, demonstrating that our specific design significantly enhances the accuracy of detecting various types of news.

For the comparative experiments, three representative methods were selected for comparison with our approach: HMCAN, which employs a co-attention mechanism to fuse multimodal information; CAFE, which utilizes the semantic differences between modalities as auxiliary tasks; and LII, which simultaneously leverages both intra-modal and inter-modal relationships. The results, presented in Table 3, demonstrate that our method achieves superior performance.

Furthermore, the ablation study, shown in Figure 4, reveals that removing unimodal modeling or inconsistency feature extraction results in notable accuracy degradation in categories 5 (Manipulated Content) and 4 (False Connection). This indicates that the proposed modules effectively identify both fabricated news and mismatched news. The details of the comparison and ablation experiments on other categories of the Fakeddit dataset can be found in Appendix C.2.

5 Conclusion &Limitations

In this paper, we propose MIAN, a novel multimodal fake news detection method that effectively identifies various types of fake news. MIAN explores hierarchical interactions within news text and image contents to generate enhanced unimodal representations, and then utilizes a coattention mechanism to model inter-modal dependencies for multimodal feature fusion. Additionally, we introduce an inverse attention mechanism from a new perspective to explicitly learn and extract intra- and inter-modal inconsistencies. Experimental results demonstrate that the proposed method achieves optimal accuracy and effectively detects different types of fake news.

Despite the promising results, our method currently relies on a single image per news article, which could lead to information bias between the two modalities. Future work could address this by extending the approach to incorporate multiple images per news article.

Acknowledgments

This work was supported by the Joint Project for Innovation and Development of Shandong Natural Science Foundation, (No. ZR2022LZH012).

Contribution Statement

Tianlin Zhang and En Yu contributed equally to this work and should be considered as co-first authors. Corresponding Author: Jiande Sun.