MFAN: Multi-modal Feature-enhanced Attention Networks for Rumor Detection

1 Introduction

With the rapid development of social media such as Twitter and Weibo, rumors can quickly spread over these platforms, which can lead to significant negative impacts on society. For example, the rumor blaming 5G for the coronavirus pandemic had led to arson attacks on more than 70 cell phone towers in the UK in 2020 1 . Due to the large amounts of user-generated content every day, it is desirable to automatically identify rumors to minimize the harmful impacts.

Traditional rumor detection models mainly rely on extracting textual features as source post representations for classification, either with traditional learning models such as decision trees [Castillo et al., 2011] or deep neural networks (DNN) based models such as recurrent neural networks

∗Corresponding author

1 https://www.cnet.com/health/5g-coronavirus-conspiracy-

theory-sees-77-mobile-towers-burned-report-says/

(RNN) and convolutional neural networks (CNN) [Ma et al., 2016; Yu et al., 2017]. With the prevalent of multimedia posts on social media, rumor spreaders tend to utilize visual content together with textual content to attract more attention and get rapid dissemination. To address this issue, a line of multimedia rumor detectors have been proposed to fuse textual and visual features based on DNN to produce multimodal post representations, which have shown better performance than solely using the textual data [Khattar et al., 2019; Wang et al., 2018; Zhou et al., 2020]. However, one common limitation of these studies is that they didn't consider the graphical social contexts simultaneously, which have been proved to be beneficial to improve the detection performance [Yuan et al., 2019].

The social context of a source post commonly involves its forwarding users and the corresponding comments. Based on these entities and their connections, a heterogeneous graph can be constructed to model the structure information. Then graphical models such as graph attention networks (GAT) [Veliˇckovi´c et al., 2017] and graph convolutional networks (GCN) [Kipf and Welling, 2016] can be utilized to aggregate adjacent node information to obtain better node representations for rumor detection [Yuan et al., 2019; Yang et al., 2021]. With the help of graphical models, connected instances can exchange information and facilitate each other's learning. However, the existing graph-based detectors suffer from several limitations: (1) the quality of node representation learning depends highly on reliable links among entities. Due to the privacy issue or data crawling constraint, the available social graph data is very likely to lack some important links among entities. Therefore, it is necessary to complement latent links on the social graph to achieve a more accurate detection; (2) there may be various latent relations between adjacent nodes on a graph, while the conventional neighborhood aggregation procedure of graph neural networks (GNN) may not be able to differentiate their effects on the representation of a target node, leading to inferior performance; (3) how to effectively integrate the learned social graph features with other modality features (e.g., visual features) is less explored in existing studies.

To address the above challenges, we propose a novel Multi-modal Feature-enhanced Attention Network (MFAN) for multimedia rumor detection. We make the first attempt to jointly model textual, visual, and social graph features in one framework. Given the multimedia post features and the social graph features, a straightforward solution is to adopt the attention mechanism between the two groups of features and effectively aggregate them to produce distinctive features for rumor detection. In this work, we improve the multi-modal fusing mechanism by considering the cross-modal semantic alignment. Specifically, a self-supervised loss is introduced to align the source post representations learned from two distinct views, i.e., the textual-visual view and the social graph view, aiming to improve the representation learning in each view. In addition, to obtain better social graph structures, we improve the graph representation learning from two perspectives. On the one hand, we propose to infer potential links between nodes in the social graph to alleviate the incomplete link issue. On the other hand, we utilize a signed attention mechanism to capture both positive and negative neighborhood correlations to achieve better node representations. Through the above enhanced cross-modal fusion and social graph representation learning, we can promote the performance of multimedia rumor detection.

The main contributions can be summarized as follows:

• We propose a multi-modal feature-enhanced attention network for multimedia rumor detection, which can effectively combine textual, visual, and social graph features in one unified framework.
• We introduce a self-supervised loss to align the source post representations in different views to achieve better multi-modal fusion.
• We improve the social graph feature learning by enhancing both the graph topology and neighborhood aggregation procedure.
• We empirically show that the proposed model can effectively identify rumors and outperform the state-of-theart baselines on two large-scale real-world datasets.

2 RelatedWork

Early rumor detection methods usually manually extract features from text content for classification [Castillo et al., 2011; Popat, 2017], which requires massive human efforts. More recently, DNN-based methods [Ma et al., 2016; Karimi et al., 2018; Ma et al., 2018] have been proposed to learn the post representations to promote the detection performance. However, these studies don't consider the visual features and social context features that would be beneficial.

Multi-modal data have been exploited by a set of studies to facilitate the detection. [Khattar et al., 2019] uses bidirectional long short-term memory (Bi-LSTM) to extract textual and visual representations. [Wang et al., 2018] jointly considers textual and visual information and removes the event-specific features. [Yuan et al., 2019] combines text and heterogeneous graph structures for rumor detection. However, they adopt simple fusion methods and may not be able to capture the complex relationships among multi-modal data.

To explore the rich correlations between different modalities, [Qian et al., 2021] jointly models the multi-modal context information and the hierarchical semantics of text in a unified model. [Lu and Li, 2020] learns the representations of user interactions, retweet propagation, and their correlation with source tweets. [Wu et al., 2021] stacks multiple coattention layers to fuse the multi-modal features. [Jin et al., 2017] jointly considers the fusion of text, image, and social context information but ignores the social graph structure.

Graph-based rumor detectors have been proposed to exploit the social graph to enhance the post representations [Yuan et al., 2019; Yang et al., 2021]. However, they ignore the incomplete link issue in the dataset. Although [Wei et al., 2021] has considered the propagation uncertainty between different nodes, it doesn't complement the missing links on the social graph nor conduct the multi-modal fusion to promote the representation learning.

A comparison between our work and the related studies is shown in Table 1. The uniqueness of our work lies in: jointly using textual, visual, and social graph features, involving multi-modal alignment for better fusion, and utilizing potential relationships to enhance the graph features.

3 ProblemDefinition

Let P = {p1, p2, · · · , pn} be a set of multimedia posts on social media with both texts and images. For each post pi ∈ P , pi = {ti, vi, ui, ci}, where ti, vi and ui denote the text, image and user who have published the post. ci = {c1i , c2i , · · · , cji } represents the set of comments of pi. Moreover, each comment cji is posted by a corresponding user uji .

In order to represent user behaviors on social media, we establish a graph G = {V,A,E}, where V is a set of nodes, including user nodes, comment nodes, and post nodes. A ∈ {0, 1}|V |∗|V | is an adjacency matrix between nodes to describe the relationships between nodes, including posting, commenting, and forwarding. E is the set of edges.

We define rumor detection as a binary classification task. y ∈ {0, 1} denotes class labels, where y = 1 indicates rumor, and y = 0 otherwise. Our goal is to learn the function F(pi) = y to predict the label of a given post pi.

4 Methodology

The focus of our proposal is to effectively combine textual, visual, and social graph features to improve the rumor detection. To this end, we first extract the three types of features. In order to produce better social graph features, we propose enhancing both the graph topology and aggregation procedure based on GAT. We then capture cross-modal interaction and alignment to achieve better multi-modal fusion. Finally, we concatenate the enhanced multi-modal features for classification. We also apply adversarial training to promote the robustness. The overall architecture is illustrated in Figure 1.

4.1 Textual and Visual Feature Extractor

Textual Representations

We employ CNN with pooling to extract the semantic feature of sentences. Firstly, for each post pi, its text ti is padded or truncated to have the same number of tokens, i.e., L, which is represented as

where o ∈ Rd , d is the dimension of word embeddings and oij denotes the word embedding of the j-th word of ti.

Then, we apply convolution layer on the word embedding matrix Oij:j+k-1 to get the feature map sij, where k is the size of the receptive field. We denote si = {si1, si2, · · · , si(L-k+1)}. Then, we use max pooling over si to obtain ˆ si = max(si ). We use d/3 filters with varying receptive filed k ∈ {3, 4, 5} to obtain semantic features of different granularities.

Finally, we concatenate all filters' outputs to form the overall textual feature of ti:

Visual Representations

We use the pre-trained model ResNet50 [He et al., 2016] trained over the ImageNet database to extract image vi's feature. Firstly, we extract the output of the second last layer of ResNet50 and denote it as V i r . Then, we pass it through a fully connected layer to obtain the final visual feature with the same dimension as the textual feature, that is,

where Wv is the weight matrix of the fully connected layer and σ(·) is an active function such as sigmoid.

4.2 Enhanced Social Graph Feature Learning Inferring Hidden Links

To alleviate the missing link issue, we propose to infer the hidden links between nodes in social networks. According to network homophily, similar nodes may be more likely to attach to each other than dissimilar ones. We thus calculate the feature similarity between different nodes and infer links between nodes with high similarity. Specifically, we define the node embedding matrix as X ∈ R|V |×d , where d is the dimension size. There are three types of nodes in X, we use the sentence vectors as the initial embeddings of the post and comment nodes, and use the average value of the post node embeddings posted by the user as the initial user embedding.

Then we calculate the correlation βij between node ni and nj based on their cosine similarity as

where xi and xj are node embeddings of ni and nj. We then infer there exists a potential edge between them if the similarity is above 0.5, that is,

Then we enhance the original adjacency matrix A ∈ R|V |×|V | with the inferred potential edges. aij denotes the element of A, where aij = 1 indicates there is an edge between ni and nj and aij = 0 otherwise. Then the element a′ij of the enhanced adjacency matrix A′ is defined as

Capturing Multi-aspect Neighborhood Relations

Here we aim to capture the social graph structure information with GAT. Different from conventional GAT, we introduce the signed attention mechanism to capture the positive and negative correlations between neighboring nodes to obtain better graph features. We start from the vanilla GAT [Veliˇckovi´c et al., 2017] and then propose our signed attention based GAT to capture the multi-aspect correlations.

The key of GAT is the aggregation of the neighborhood information. For node ni and its neighbor node set Ni = {n′1 , n′2 , · · · , n′|Ni|}, we first calculate the set of attention

weights Ei = {e′i1 , e′i2 , · · · , e′i|Ni|} between ni and each node in Ni by

where ∥ means concatenation operation, ˆa and W are learnable parameters, xi and x′j denote the node embeddings of node ni and n′j , j ∈ {1, 2, · · · , | Ni |} and n′j ∈ Ni.

Then, we use the softmax function to perform the weight normalization operation on the attention weights. The attention weight e′ij between ni and n′j may be a negative value, which will become extremely small after softmax function. In fact, the attention weight between nodes contains a potential positive and negative relationship, which will be ignored by directly using the softmax function. For example, for a specific node nt, we obtain its weights with the neighbor nodes as Et = {0.7, 0.3,-0.1,-0.9}. After normalization by the softmax function, the weights become E ′ t = {0.43, 0.29, 0.20, 0.09}. It can be seen that the node corresponding to the value "-0.9" in the weight vector has the smallest contribution to the output. However, "-0.9" may indicate that the two node vectors are in opposite directions. This kind of large negative relations may also be beneficial for rumor detection. For instance, it may reflect camouflage behaviors such as a rumor spreader buying some honest users as fans or a comment opposing a source post [Yang et al., 2021], and their node vectors can be intrinsically negatively correlated. Unfortunately, existing GATs ignore such negative correlations.

To address this issue, inspired by QSAN [Tian et al., 2020], we design a signed attention based GAT, namely Signed GAT, which uses signed attention to involve both the positive and negative relationships between nodes. Specifically, for node ni, we denote the inversion of the attention weights Ei of its neighbor nodes as ˜ Ei = -Ei. We then calculate the normalized weights for both Ei and ˜ Ei with the softmax function,

In order to capture both positive and negative relations between nodes, we utilize E ′ i and - ˜ E ′ i respectively to obtain the weighted sum of the neighbor nodes' features. Then we concatenate the two vectors together and pass it through a full connected layer to obtain the final node feature. For instance, the node feature of ni can be obtained by

where Wn is the weight matrix of the fully connected layer, σ(·) is an active function and Xj is the feature matrix of Ni.

Graph Feature Extractor

We then introduce how to obtain the social graph feature based on the enhanced social graph and the signed GAT. Firstly, we enhance the original social graph by augmenting the inferred potential edges, and initialize three types of nodes in the graph. For post and comment nodes, we use their textual features as the initial embeddings. For user nodes, we use the average of their post and comment embeddings as the initial embeddings to reflect the user characteristics.

Then we use Signed GAT to extract graph structure features from the enhanced social graph. For each node, we update its embedding according to Eqn. (9) and obtain the updated node embedding matrix ˆX ∈ R|V |×d , where |V | is the number of nodes and d is the dimension size. Then a multihead attention mechanism [Vaswani et al., 2017] is adopted to capture features from different perspectives. We concatenate the updated node embeddings of each head together as the overall graph feature:

where H denotes the number of heads. Then the graph feature Rig of the i-th post pi corresponds to the i-th column of ˆG.

4.3 Multi-modal Feature Fusing

In this work, as there are three types of modalities, we adopt a hierarchical fusion schema with the co-attention method [Lu et al., 2019]. In order to capture different aspects of crossmodal relationships and enhance the multi-modal features, we propose to enforce the cross-modal alignment with a selfsupervised loss.

Cross-modal Co-attention Mechanism

We use the co-attention mechanism to capture the mutual information between different modalities. It learns the attention weights between different modal features to enhance the cross-modal feature.

Specifically, for each modal, we first use multi-head selfattention [Vaswani et al., 2017] to enhance the intra-modal feature representation. For example, for the text feature Rit , we use Qit = RitW Q t , Ki t = RitW K t , and V i t = RitW V t to calculate its query matrix, key matrix and value matrix respectively, where W Q t ,W K t ,W V t ∈ Rd× dH are linear transformations and H is the number of heads. We then produce the multi-head self-attention feature of the text modal as

where h denotes the h-th head, and W O t ∈ Rd×d is the output linear transformations. We perform the same operations on Riv and Rig to obtain the corresponding features Z iv and Z ig .

Then we use the co-attention mechanism to produce the enhanced multi-modal features. Specifically, in order to perform the textual-visual co-attention for pi, we first perform a similar operation as the above self-attention, but replace Rit with Z iv to get the query matrix Qiv , and replace Rit with Z i t to get the key matrix Ki t and value matrix V i t . Thenwe obtain the cross-modal enhanced feature Z ivt as

where W O vt ∈ Rd×d is the output linear transformations.

Note that Z ivt represents the enhanced textual feature with the visual feature based on their correlations. Based on the same co-attention procedure, we can obtain the enhanced visual feature Z i tv with the textual feature by exchanging the roles of the two modalities in Eqn. (12).

Multi-modal Alignment

Based on the co-attention mechanism, we can obtain the enhanced textual feature with the visual feature and vice versa. However, for the source post, its representations in different modalities should be intrinsically related. Such inter-modal correspondences are not covered by the co-attention mechanism. We thus introduce the modal alignment via enforcing the enhanced textual feature of the post close to its enhanced graphical features in order to refine the representations learned in each modality.

Specifically, for a post pi, its enhanced graph feature Z ig and enhanced textual feature Z ivt are transformed into the same modal feature space, that is,

where Wg ′ and Wt ′ are learnable parameters. Then we narrow the distance between Z ig ′ and Z i t ′ with the MSE loss for modal alignment:

where n is the total number of posts. We then get the alignment-refined textual feature ˜ Z i t and graphical feature ˜ Z ig , which is used for the following multi-modal fusion.

Fusing the Above Multi-modal Features

We again perform the aforementioned cross-modal coattention mechanism among each pair of the three modal features, i.e., ˜ Z i t , ˜ Z ig and Z iv , and finally get six cross-modal enhanced features: ˜ Z i tv , ˜ Z ivt , ˜ Z igt , ˜ Z i tg , ˜ Z igv , and ˜ Z ivg . We then concatenate them as the final multi-modal feature:

4.4 Classification with Adversarial Training

We feed the final multi-modal feature Z i of post pi into the fully connected layer to predict whether pi is a rumor or not,

where ˆ yi denotes the predicted probability of pi being a rumor. Then we use the cross-entropy loss function as

The final loss can be written as follows:

where λc and λa are used to balance the two losses.

As the text contents in social media may not follow the strict grammar rules, in order to adapt to such grammatical irregularity, we add adversarial perturbations at the text embedding level to enhance the robustness of the model. We use PGD [Madry et al., 2017], a widely used adversarial training method. Specifically, we calculate the gradient for the textual feature in each training iteration and use it to calculate the adversarial perturbation, which is added to the textual feature. We then recalculate the gradient on the updated textual feature. We repeat this process k times and use a spherical space to limit the extent of the perturbation. Finally, the above adversarial gradients are accumulated to the original gradient, which is then used for parameter updating. More details can refer to [Madry et al., 2017].

5 Experiments

5.1 Datasets

We evaluate our model on two real-world datasets: Weibo [Song et al., 2019] and PHEME [Zubiaga et al., 2017]. The Weibo dataset is collected from the most popular social media in China. PHEME is constituted by tweets on the Twitter platform and based on five breaking news. Each dataset contains texts, the attached images, and comments. In this work, we focus on detecting rumors with three modal features, i.e., the textual, visual, and social graph features. Thus, we remove the data instances without any text or image. Table 2 shows the statistics of the resulting two datasets after removal.

and EBGCN outperform most other methods, indicating that the social graph information is beneficial for rumor detection. For methods that consider both textual and visual information, SAFE outperforms other methods, indicating the importance of considering interactions between modalities. Our MFAN outperforms GLAN and EBGCN, demonstrating that considering visual data, latent links, and modal alignment can further improve detection performance.

5.5 Performance of the Variations

To show the effectiveness of different components in MFAN, we compare it with the sub-models "-w/o V", "-w/o G", "-w/o P", and "-w/o A". They denote the variant of MFAN without considering the visual information, social graph information, potential links, and modal alignment, respectively. The comparison results are shown in Table 4. We can observe that all ablation variants perform worse than the complete MFAN model on both datasets. The results indicate that: (i) visual modal and graph features are both important for rumor detection; (ii) the modal alignment can facilitate the multi-modal fusion; (iii) considering latent links can significantly improve the social graph feature representations.

6 Conclusions

In this paper, we propose a multi-modal rumor detection framework that, for the first time, incorporates three types of modalities, i.e., text, image, and social graph. To improve the social graph feature learning, both the graph topology and neighborhood aggregation procedure are enhanced based on GAT. Our framework enables more effective multi-modal fusion by introducing cross-modal alignment. Evaluations and comparisons on both Chinese and English datasets demonstrate that our model can outperform the state-of-the-art baselines for multimedia rumor detection.

5.2 Baselines

We compare our model with the following strong baselines:

• EANN [Wang et al., 2018] is a GAN-based model exploiting both text and image data. It derives eventinvariant features and benefits newly arrived events.
• MVAE [Khattar et al., 2019] uses a bimodal variational autoencoder coupled with a binary classifier for multimodal fake news detection.
• QSAN [Tian et al., 2020] integrates the quantum-driven text encoding and a novel signed attention mechanism for false information detection.
• SAFE [Zhou et al., 2020] jointly exploits multi-modal features and cross-modal similarity to learn the representation of news articles.
• EBGCN [Wei et al., 2021] rethinks the reliability of latent relations in the propagation structure by adopting a Bayesian approach.
• GLAN [Yuan et al., 2019] jointly encodes the local semantic and global structural information and applies a global-local attention network for rumor detection.

Among them, EANN, MVAE and SAFE exploit both textual and visual data. QSAN only exploits the textual data. Social graphical features are considered by EBGCN and GLAN. None of them consider data from all three modalities like our proposed model.

5.3 Implementation Details

We split the datasets for training, validation, and testing with a ratio of 7:1:2. The evaluation metrics include Accuracy, Precision, Recall, and F1. We use word vectors provided in [Yuan et al., 2019] as initialized word embeddings. The number of heads H is set to 8. λc and λa are set to 2.15 and 1.55. We choose the best parameter configuration based on the performance of the proposed model. We use Adam [Kingma and Ba, 2014] to optimize our objective function. The learning rate used in the training process is 0.002. We perform 5 runs throughout all experiments and report the average results and standard deviation results.

5.4 Results and Discussion

Table 3 shows the performance of the comparison methods. On both datasets, our model MFAN significantly outperforms all the other approaches in all the metrics. GLAN

Acknowledgments

This work was supported by the Natural Science Foundation of China (No.61976026, 61902394) and 111 Project (B21049).