A Generalized Deep Markov Random Fields Framework for Fake News Detection

1 Introduction

Nowadays, online social media such as Twitter and Weibo have been woven into the fabric of people's lives, providing a convenient platform for users to acquire instantaneous information and share personal opinions. Unfortunately, with this prevalent trend, massive false information known as fake news also proliferates extensively across social media. As fake news may manipulate major public events and affect social safety by misleading users' views [Allcott and Gentzkow,

∗Corresponding author

2017], developing automatic fake news detectors has been in the limelight over the past few years.

Thus far, many methods have achieved impressive performance with extra social contexts such as news propagation networks [Bian et al., 2020] and user comments [Cheng et al., 2020], but those additional social features do not always exist especially when a news article is just emerging, which inspire us to focus on mining the news content itself. Lately, online news content includes a wealth of multimodal resources, involving texts, images, etc. So many works aggregating multimodal content information have emerged. Besides, with the advancement of neural network technology, numerous content-based methods utilize deep neural networks (DNN) to extract unimodal or multimodal features for identifying fake news [Zhou et al., 2020; Chen et al., 2022].

However, these existing DNN-based approaches merely learn the implicit relevance of labeled news. Actually, there are obvious shared characteristics such as negative emotional words and inflammatory expressions among all labeled and unlabeled fake news [Ajao et al., 2019], which is often ignored by existing works. Therefore, we aim at a framework that leverages the implicit relationship modeling capabilities of DNN while effectively incorporating the explicit correlations among news when determining their credibility.

Intuitively, similar news articles are inclined to share the same labels. Nevertheless, a pair of news may be irrelevant even though their labels are consistent if they discuss totally irrelevant events [Zhang et al., 2021]. We believe the eventspecific information could contradict the intuitive judgment somehow. To investigate the impact of event information, we calculate the percentage of label consistency news pairs with varying cosine similarity scores on the Twitter dataset [Boididou et al., 2018]. Specifically, the four curves in Fig. 1 correspond to three event-specific news set E1, E2, and E3 (E1: Boston Marathon, E2: Hurricane Sandy A, E3: Hurricane Sandy B) and the union of these three sets EAll respectively. From Fig. 1, our major findings are two-fold. Firstly, the increasing trend of all curves demonstrates that as the similarity of news pairs rises, it is likely that have the same labels. Secondly, EAll curve is much lower than the E1, E2, and E3 curves, with an average ratio of 11.7%, 10.2% and 34.5%, indicating that event-specific information would weaken the correlation between label consistency percentage and similarity scores.

Having the aforementioned insights, we would like to model our intuition using Markov Random Fields (MRF) [Jin et al., 2019] while minimizing the impact of event-specific information on the correlation coefficient. Thus, we develop a Generalized Deep Markov Random Fields Framework (GDMRFF) based on graph theory for fake news detection, which inherits the power of deep learning and captures dependencies among news articles jointly. In our framework, we first learn the implicit relations for all news with the superior capability of DNN, which are unfolded to the unary function of MRF. The pairwise potential functions encapsulating our intuition are designed to model the explicit corrections among all samples (including labeled and unlabeled samples), which helps refine the initial results made by DNN. Furthermore, we introduce an event removal module to progressively remove event features during the training process. Note that the proposed GDMRFF is trained in a semisupervised setting, which decreases the reliance on labeled data while maximizing the potential of unlabeled data. To clearly analyze the effectiveness of our framework, we further propose an Ambiguity Learning Guided MRF (ALGM) model for multimodal fake news detection as a concretization of GDMRFF, which is derived by replacing the content features extraction module using DNN in our framework.

To summarize, the contributions of this paper include:

• To the best of our knowledge, this is the first semisupervised fake news detection framework that explicitly explores the characteristics shared among fake news only utilizing news content at the early stage of news dissemination.
• We first develop GDMRFF, a general semi-supervised framework for fake news detection, in which a MRF layer capturing shared characteristics is stacked to refine initial predictions made by DNN while removing eventspecific features during training. Most existing DNNbased supervised models can be adapted to our framework, lowering the amount of labeled data required and enhancing performance. To verify this, we further present an Ambiguity Learning Guided MRF (ALGM)

model as a materialization of GDMRFF.

• Experimental results on two real-world datasets demonstrate ALGM outperforms the state-of-the-art models. In addition, by displaying the latent embeddings, we highlight the improvement effects of GDMRFF's refinement.

2 Methodology

2.1 Task Definition

We focus on identifying fake news in social media using its contents only. Specially, given a news dataset Dall = {X, Y } containing news articles X and their ground truth labels Y ∈ {0, 1}. Each news item in X can incorporate features from one or more modalities. In our ALGM, we consider two primary modalities, i.e., text and image for each news. The dataset Dall could be divided into a training set Dtra and a test set Dtes with a specific ratio. Given a training set Dtra and a test set Dtes without labels, our goal is to assign label to each news in test set by investigating its own content information and associations across all data. Mathematically,

2.2 Model Overview

Our proposed ALGM model aims to explicitly explore the universal features shared among all the fake news. As illustrated in Fig. 2, it integrates three major modules: a) Multimodal Feature Extraction that extracts features from two modalities and derives a fused embedding for each news item (This module can be replaced by any other DNN-based methods, allowing our Generalized Deep Markov Random Fields Framework (GDMRFF) to improve their performance); b) Event Removal that includes an event discriminator to minimize event-specific feature in the joint news representation; and c) MRF Inference that explicitly captures shared fake news properties based on the entire datatset.

Given a dataset Dall with N articles, we begin by extracting features from their texts and attached images separately, then aligning and fusing the unimodal features in an adaptive manner. After obtaining joint multimodal representations, they are passed into an event removal module to eliminate event-related characteristics as much as possible. In the meanwhile, they are also converted into preliminary predicted probabilities, which are then served as the unary potential functions for MRF. At last, we construct a global news similarity graph utilizing the multimodal representations of all news, upon which our pairwise potential functions of MRF are designed to model our intuition.

2.3 Multimodal Feature Extraction

Unimodal Feature Extraction. To exploit underlying semantic information of each news xi, ∀i ∈ {1, ..., N}, we employ the pre-trained BERT [Kenton and Toutanova, 2019] and ResNet34 [He et al., 2016] to extract latent text and image features f t i and f v i in xi separately. Subsequently, we feed f t i and f v i into linear layers to have a fixed dimension df . Cross-modal Alignment. Before fusing text and image features, an issue that needs to be considered is the inherent information gaps persisting across heterogeneous modalities

[Hazarika et al., 2020]. We handle this by introducing a cross-modal alignment component, which embodies a shared fully connected layer coupled with a Leaky Rectified Linear Units (LeakyReLU) to map features from different spaces into a common semantic space as follows:

where eti , evi ∈ Rde denote aligned representations, de is the feature dimension after alignment, W1 ∈ Rde×df , b1 ∈ Rde are learnable parameters of the shared layer.

Ambiguity Guided Feature Fusion. To adaptively incorporate features from different modalities, we utilize an ambiguity guided feature fusion component following [Chen et al., 2022], which considers the inherent ambiguity (i.e., degree of information gap) between different content modalities. The ambiguity score is estimated by learning the Kullback-Leibler (KL) divergence between two unimodal distributions that are approximated by two variational autoencoders separately. For each news xi with aligned textual and visual representations, the variational posteriors could be denoted as:

where µ is the mean and σ2 is the variance. Taking the entire dataset into account, they are

Then, an averaged KL divergence is served as the ambiguity score ηi of different modalities as:

where DKL represents the calculation of KL divergence and sigmoid is a normalization function.

The interaction between modalities is conducive to induce the complementary features from each others, especially when unimodal features alone emerge as strongly incongruous. So an interaction vertor ci ∈ Rde is computed as follows:

where softmax is an activation function for normalizing, [·]T indicates matrix transposition, ⊗ denotes the outer product, and W2 ∈ R1×de , b2 ∈ R1×de are learnable parameters.

To get the final feature representation, unimodal and multimodal features are concatenated adaptively guided by the ambiguity score as follows:

where ⊕ denotes the concatenation of vectors.

2.4 Event Removal

Due to event-specific information has an adverse impact on the correlation between news label consistency percentage and similarity scores statistically, we intend to remove as many event related features as possible in each news vector mi. We first adopt an event discriminator containing two fully connected layers followed by a softmax function, whose original purpose is to learn event-specific features to distinguish among different events, while our aim is exactly the opposite. Accordingly, we apply a gradient reversal layer [Ganin and Lempitsky, 2015; Wang et al., 2018] before the discriminator to maximize the cross-entropy loss function over the whole dataset defined as below:

where ψa and ψb are all parameters to be trained in the multimodal feature extraction module and the event removal module respectively. τi is the ground-truth event label distribution of news xi, which is obtained by k-means clustering using the text embeddings f t = {f t 1 , f t 2 , ..., f tN } of all data from the BERT extractor, ˆ τi is the predicted event distribution of news xi, and db is the number of event categories.

2.5 MRF Inference

Considering the explicit relations among all the news, we aim to capture those relations utilizing a Markov Random Field with its ability in modeling joint probability distribution over dependent random variables. Our MRF is built on top of the multimodal feature extraction module, thus taking the final feature representations m = {m1,m2, ...,mN} as input. Above all, we construct a global news similarity graph S = (V, E) based on m, where the i-th node in V (|V| = N) denotes the news xi and edges in E link similar news pairs. Then the pairwise potential functions of MRF are designed over this graph.

In our MRF, each xi is associated with a random variable ωi, which represents its label. The set of all nodes' label assignments is represented as random variables ωV with domain L = {0, 1}. The MRF can be estimated in this Gibbs distribution form:

where Z is a constant for normalizing, (u, r) ∈ L are the labels of news items. Energy function E(ωV) includes the unary potential Φ(ωu i ) that measures the cost that assign label u to node xi and the pairwise potential Ψ(ωu i , ωr j ) denoting the cost that nodes xi and xj have labels u and r respectively. α is a trainable parameter.

As the exact distribution of P(ωV) is infeasible, we employ the mean-field theory [Koller and Friedman, 2009] to approximate P(ωV) by a factorizable distribution Q(ωV) = ∏i∈V Qi(ωi). We define Qi(ωi = u) as the probability that the node xi has the label u. Here we use qu i to indicate Qi(ωi = u), which can be iteratively updated as follows:

where Zi is a normalization term.

We use the unary potential function Φ(ωu i ) to serve as an interface between DNN-based module (i.e., the multimodal feature extraction module) and MRF, fully absorbing the ability of DNN in learning high-level representations into our MRF layer. Concretely, the unary term is calculated by:

where the probability q(0) i that inherits the power of DNN is derived by projecting the final representation mi of node xi from the DNN-based feature extraction module into two dimensions, and q(0) i [u] is the component value of the u-th dimension of q(0) i , which is used as the prior probability that node xi has label u.

Since we have experimentally proved that similar news pairs tend to share the same labels, we now model this correlation with pairwise potentials of MRF. In order to measure the similarity between news pairs, we introduce a parameter matrix-guided cosine distance metric function whose computed values are taken as edge weights in the global news similarity graph S:

where ⊙ represents the Hadamard product. mi and mj are final vectors of node xi and node xj from the multimodal feature extraction module respectively. The parameter wi,j has the same dimension as mi. Besides, a threshold δ is preset to control the sparsity of graph S, that is, once si,j is lower than δ, its value will be displaced by 0.

Based on this, si,j reflects how strong the similarity between two nodes is, then the pairwise term is designed as:

where γ(u, r) is the label compatibility, if u! = r, it equals 1, otherwise it equals 0. In our pairwise, similar node pairs with inconsistent labels will be penalized. Moreover, the more similar the node xi and the node xj are, the larger the values of si,j and Ψ(ωu i , ωr j ), yet considering the negative sign in Equation (14), the value of qu i will decrease, i.e., the penalty will increase accordingly. In this way, the relation between label consistency percentage and similarity is embodied in the MRF layer.

Then we extend Equation (13) to the entire dataset and stack K MRF layers, and we get the following matrix form that can be updated iteratively:

where k denotes the k-th MRF layer. The output of the previous layer Q(k-1) is sent to the k-th layer as an input. Q(0) ∈ RN×dc is the preliminary probability matrix from the multimodal feature extraction module, whose i-th row vector equals to q(0) i , the adjacency matrix S ∈ RN×N comprises all the edge weights in the news similarity graph, and Γ ∈ Rdc×dc is the label compatibility matrix. dc is the dimensionality of news authenticity labels.

Algorithm 1 Model Training of ALGM

Input: Dataset: D, hyper-parameter: df , de, db, dc, δ,K, λ, and learning rate: ρ

Output: Parameters: ψa , ψb , ψc

• 1: while not converge do
• 2: update parameters ψa in the multimodal feature extraction module: ψa ← ψa - ρ ( ∂Lc ∂ψa -λ ∂Lb ∂ψa )
• 3: update parameters ψb in the event removal module: ψb ← ψb - ρ ∂Lb ∂ψb
• 4: update parameters ψc in the MRF inference module: ψc ← ψc - ρ ∂Lc ∂ψc
• 5: end while

We regard the output Q(K) of the last MRF layer as the predicted probability of news credibility. For fake news detection, a cross-entropy loss function is devised as follows:

where ψc absorbs all parameters to be trained in the MRF inference module, yi and ˆ yi are the ground-truth and predicted label distribution of the node xi separately.

2.6 Model Learning

Combining the event removal loss Lb and the fake news detection loss Lc, the overall loss function is defined as:

where λ is a trade-off parameter between the two terms. The parameters ˆ ψa , ˆ ψb , ˆ ψc that we desire are the saddle point of the overall loss function:

Here a gradient reversal layer is introduced to achieve the objective in Equation (20), which multiplies the gradient by -λ during the backpropagation process. The overall training process of our ALGM is outlined in Algorithm 1.

3 Experiments

3.1 Experimental Setup

Datasets. We use two widely-used datasets collected from Twitter and Weibo for fair evaluation. The Twitter dataset is part of MediaEval released for Verifying Multimedia Use task [Boididou et al., 2018], which is spilt into the development set and test set. The Weibo dataset was released by [Jin et al., 2017]. The real news is collected from Xinhua News Agency, an authoritative news source of China, while the fake one is verified by the official rumor debunking system of Weibo. We provide comprehensive statistics for these two datasets in Table 1. Before training, we carry out a series of pre-processing steps including removing posts with attached videos, deleting special symbols and hyperlinks in texts and normalizing images for each dataset.

Compared Baselines. To evaluate the performance of the proposed ALGM, we choose two categories of compared models, namely: 1) Unimodal methods, i.e., Image and Text, which adopts a pre-trained ResNet34 model and BERT model coupled with a fully connected layer, respectively; 2) Multimodal methods, i.e., NeuralTalk [Vinyals et al., 2015], att RNN [Jin et al., 2017], EANN [Wang et al., 2018], MVAE [Khattar et al., 2019], SAFE [Zhou et al., 2020], BTIC [Zhang et al., 2021], CAFE [Chen et al., 2022]. These methods typically apply deep neural networks such as CNN, RNN, and attention mechanisms to extract multimodal features, combined with some well-designed sub-task or strategy like cross-modal alignment and contrastive learning, then they are trained to suitable models based on general rules of supervised classification.

Implementation Details. In the multimodal feature extraction module, we fix the dimension df = 128, de = 64. We set the number of event categories db as 20 on Twitter and 50 on Weibo. The sparsity threshold δ of the news graph S is set to 0.4. We optimize parameters in our model with Adam [Kingma and Ba, 2014] optimizer. The learning rates on the two datasets are equal to 10-4 and 10-2 respectively. To prevent over-fitting, we adopt a random dropout with a probability of 0.5. Our model is trained with a maximum epoch of 200 and we use the early stopping strategy.

3.2 Performance Comparison

The performances of baselines and our proposed ALGM are presented in Table 2. Experimental results clearly exhibit that our ALGM achieves the best performance on both datasets in terms of accuracy, precision, recall, and macro F1 score. Specifically, ALGM reaches the highest accuracy of 89.1% and 84.6%, surpassing the state-of-the-art method's 3.2% and 2.0% on Twitter and Weibo datasets respectively.

We can observe many commonalities between the two datasets. Considering the unimodal methods, Text (BERT) performs much better than Image (ResNet34). This implies that textual features in news provide more evidence for determining the veracity of news. In addition, multimodal techniques beat unimodal approaches on nearly all metrics, proving that incorporating multimodal features can provide supplementary support in detecting fake news. On both datasets, the proposed ALGM outperforms all compared models significantly. We believe this is the result of our realistic simulation of the general characteristics of fake news and the full use of the advantages of the semi-supervised setting, i.e., in addition to labeled data, we extensively mine the information from unlabeled data (which is impossible to achieve by the full-supervised model) by the generalized deep Markov Random Fields framework (GDMRFF).

3.3 Ablation Study

Effectiveness of Key Components. Firstly, we investigate the impact of each key component in ALGM on performance by removing it from the entire model. Concretely, we get the ALGM (w/o event) and ALGM (w/o mrf) by omitting the event removal and MRF inference module respectively. Then we eliminate the above two modules together and obtain the ALGM (w/o both). When the MRF inference module is omitted from these variants, we simply append a fully connected layer with a softmax function to the end of the feature extration module for final prediction.

As shown in Table 3, ALGM (w/o event) yields inferior results, demonstrating that event-related characteristics do indeed impede the utility of MRF and confound our model. ALGM (w/o mrf) also yields poorer performance, indicating that the apparent consideration of shared features in fake news is crucial for our task. ALGM (w/o both) performs the worst of the three variants. This proves that both modules provide significant discriminability when identifying fake news, and that integrating them could improve their effectiveness.

Effectiveness of GDMRFF. The second experiment is to explicate the effectiveness of the generalized deep Markov Random Fields framework we proposed. Our framework can be adapted to any other supervised DNN-based model for detecting fake news, reducing the model's dependency on labeled data and improving its performance. Consequently, we select three supervised models to verify this, including an image model (ResNet34), a text model (BERT), and a classical multimodal fake news detection model (SpotFake [Singhal et al., 2019], which concatenates textual features from the BERT model and visual features from the VGG-19 [Simonyan and Zisserman, 2014] model). Specifically, we replace the multimodal feature extraction module in our framework with each selected model. Then, we compare the performance of the original models to that of the framework-modified models. The results are plainly displayed in Fig. 3. The performance improvement of each model with varying degrees strongly proves the generalization and effectiveness of GDMRFF.

3.4 Data Ratio Analysis

In real scenarios, labeled news articles are quite sparse and limited due to the high cost of acquisition, while there is a sheer volume of unlabeled data. A pivotal issue here is how to make full use of those unlabeled data, so in this paper, we develop a semi-supervised framework to capture dependencies among all the samples (unlabeled data included). To explore the performance of our model with little labeled data, we vary the number of labeled samples in the training set (i.e., ranging from 1% up to 80%). The best-performing baseline CAFE is chosen for comparison. Results in Fig. 4 display that ALGM always outperforms CAFE in all proportions on both datasets. Despite the fact that all models perform poorly with less training data, it is clear that ALGM degrades more slowly, suggesting that ALGM is more useful in the real world, when labeled data is extremely sparse. We attribute this to the collaborative modeling of all labeled and unlabeled data by GDMRFF, which also takes advantage of the rich features contained in unlabeled news.

On the Twitter dataset, we can see when the proportion of labeled data decreases from 80% to 20%, yet the performance of our model has little or no variation. This is because this dataset contains a big number of tweets that are quite similar. Consequently, the relationships between news are strengthened, allowing GDMRFF to play a more significant role.

3.5 Visualization

To further evaluate our model intuitively, we visualize the news representations learned by ALGM, ALGM (w/o both) and CAFE (a best-performing baseline) with the t-SNE tool [Van der Maaten and Hinton, 2008] on the Weibo dataset. Fig. 5 shows the visualized results, from which we can observe that ALGM can notably generate more discriminative news representations than ALGM (w/o both) and CAFE. As depicted in Fig. 5(a) and Fig. 5(b), these two models are also able to separate news into two categories roughly, but the vectors still have a certain part of intersection and entanglement at the boundary.

In contrast, the vector representations learned by our integrated model have a more pronounced separation boundary and larger inter-class distance shown in Fig. 5(c). We believe this benefits from the refinement of GDMRFF. During training, with the aid of the event removal module, we can get features that are only specific to news categories. After modeling the connection within each category of news through MRF, all news can have smaller intra-class distances with the larger inter-class separated areas.

3.6 Case Study

We present an example in Weibo that was misclassified by the DNN-based module in Fig. 6(a) but was corrected by our MRF module in Fig. 6(b). Nodes represent news articles, the thickness of the edges represents the similarity between articles, and the color of each node represents its predicted label. Nodes within the gray dotted circle are related to the same event. For the node with id 710 in Fig. 6(a), it was erroneously assigned to "real", since the other two nodes in the same event (these three nodes are all about shampoo) are labeled as "real" and the DNN-based module tends to classify nodes with similar semantic information into the same category. In comparison, the MRF module correctly assigned this node to "fake" as shown in Fig. 6(b). Because the MRF layer removes event-specific information and refines the initial results by using information from its all neighbors in the global similarity graph that models the unique characteristics shared among all fake news.

4 Conclusion

In this work, we focus on content-based fake news detection. A disadvantage of existing DNN-based methods is their inability to model the explicitly shared features in fake news. Thus, we propose a Generalized Deep Markov Random Fields Framework (GDMRFF) based on graph theory, which leverages the superiorities of DNN in capturing highlevel features, and of MRF in integrating the explicit correlations among all labeled and unlabeled data. Moreover, based on the proposed GDMRFF, we also develop a concrete ambiguity learning guided MRF model, named ALGM. Experiments on two public datasets demonstrate the effectiveness of GDMRFF. In future work, we plan to extend our framework to be unsupervised to further diminish the reliance on labeled news and make it more applicable to real-world scenarios.

Acknowledgments

The work was supported by the National Natural Science Foundation of China (No. 62276187, 62272340, 62172056).