Crowd, Expert & AI: A Human-AI Interactive Approach Towards Natural Language Explanation Based COVID-19 Misinformation Detection

1 Introduction

The proliferation of COVID-19 misleading information on social media has posed a serious threat to the public health and online news ecosystem [Shang et al., 2022a]. For example, nearly 800 people have died in the first quarter of 2020 as they believe that consuming bleach could disinfect the body and kill the COVID-19 virus [Islam et al., 2020]. Therefore, the COVID-19 misinformation detection problem has gained increasing research attention and public interest in promoting Good Health and Well-being (i.e., the United Nations' Sustainable Development Goals (SDGs)). However, it is often difficult for AI algorithms to effectively detect the COVID19 misinformation and explain the detection results due to the lack of accurate COVID-19 professional knowledge in the training data to optimize the AI models. To address this challenge, we focus on a natural language explanation (NLE) based COVID-19 misinformation detection problem where the goal is to accurately identify COVID-19 misinformation on social media and provide explanations for the identified misinformation in natural language.

∗The first two authors contributed equally to this work.

Several initial efforts have been made to study the explainable COVID-19 misinformation detection problem in machine learning and human-computer interaction (HCI) communities [Ayoub et al., 2021; Kou et al., 2022; Shang et al., 2022a]. We show three types of explanations in Figure 1 for a misleading COVID-19 post to illustrate the limitations of current approaches. For Figure 1a, Ayoub et al. [Ayoub et al., 2021] proposed an attention-based AI framework that identifies misleading COVID-19 posts and extracts specific words from the posts as explanations [Jain and Wallace, 2019]. However, people cannot directly understand the reason of the misinformation simply by reading the extracted words due to the lack of semantic context between the words. Kou et al. [Kou et al., 2022] tasked online crowd workers to analyze COVID-19 articles (e.g., fact-checking articles, medical literature) and extracted COVID-19 knowledge triples from the articles as explanations (e.g., "bleach" not cure ----→ "COVID-19" in Figure 1b). However, people may still not be able to understand why the bleach is incapable of killing COVID-19 in human's body due to the lack of explicit justification on the knowledge fact (e.g., bleach corrodes human tissues). In contrast, the NLE in Figure 1c is an example of the objective of this paper that can effectively address the above limitations by providing critical details in natural human language to explicitly explain the exact reason why the post is misleading.

Motivated by the above observations, we develop CEACOVID, a tripartite Crowd-Expert-AI interactive framework to address the NLE-based COVID-19 misinformation detection problem. In our solution, we leverage the strengths of three parities: i) crowd workers who often can easily understand COVID-19 posts on social media, ii) COVID-19 expert who have professional COVID-19 knowledge, and iii) AI models that can quickly navigate COVID-19 articles and extract relevant knowledge facts (e.g., the statement in Figure 1c), to collaboratively identify misleading COVID-19 posts and generate NLEs for the identified misinformation. However, the design of such an NLE-based COVID-19 misinformation detection framework is non-trivial due to two research challenges: i) how to design a crowd-AI collaboration model to analyze the COVID-19 posts and extract highly relevant COVID-19 knowledge facts from the COVID-19 articles as the NLE for the post? 2) How to leverage the professional knowledge from the COVID-19 experts to justify the uncertain COVID-19 posts identified by crowd workers and noisy knowledge facts extracted by the AI model?

To address the first challenge, we develop a novel logicdriven COVID-19 knowledge graph that can effectively identify COVID-19 knowledge facts from COVID-19 articles by exploring the semantic logical relations between the COVID19 posts and the knowledge facts. To address the second challenge, we design an expert-guided knowledge updating strategy to dynamically add the latest COVID-19 knowledge facts to our knowledge graph by leveraging the professional COVID-19 knowledge from the COVID-19 experts. To our best knowledge, CEA-COVID is the first human-AI interactive framework to solve the NLE based COVID-19 misinformation detection problem by jointly leveraging the intelligence from crowd workers, COVID-19 experts and AI models. We evaluate CEA-COVID on two large-scale COVID-19 misinformation datasets on social media. The results show that CEA-COVID not only correctly identifies the misleading COVID-19 posts but also generates more explicit and accurate NLEs for the detected posts than the state-of-the-arts.

2 RelatedWork

MisinformationDetection andExplanation. Explainability is a critical aspect of misinformation detection, which aims to provide justifications for the misinformation detection results [Brand et al., 2021; Karagiannis et al., 2020; Shang et al., 2022b]. A few explainable misinformation detection solutions have been proposed to explain the identified misinformation with various types of explanations, including top misleading keywords [Ayoub et al., 2021] and knowledge fact triples [Cui et al., 2020]. However, these solutions cannot be applied to solve our problem since they are insufficient to generate NLEs that contain detailed and explicit explanations of the detected misinformation. Moreover, several recent efforts in natural language processing have been developed to generate NLEs for question answering tasks [Danilevsky et al., 2020]. Current NLE generation approaches always require a large number of accurate NLE training labels (e.g., several millions of open-domain questions) that are usually annotated by crowd workers. However, crowd workers are often incapable of generating NLE labels for COVID-19 posts due to their lack of COVID-19 professional knowledge. We develop a crowd-expert-AI interaction framework that leverages the collaboration between crowd workers and COVID19 experts to jointly generate the NLEs for COVID-19 posts by providing the COVID-19 knowledge facts retrieved from COVID-19 articles by the AI model.

Crowdsourcing Intelligence. Our work is also closely related to leveraging collective human intelligence from a large number of crowd workers to improve AI models. Traditional crowdsourcing solutions often utilize crowdsourcing efforts for conventional annotation tasks, such as object annotation [Barbu et al., 2019] and image captioning [Zhou et al., 2020]. A few recent crowdsourcing solutions have been proposed to solve domain-specific annotation problems by leveraging the human background knowledge from crowd workers [Kou et al., 2022; M´endez M´endez et al., 2019]. However, these solutions often require crowd workers to understand the domain-specific crowdsourcing tasks well by reading various professional articles (e.g., COVID-19 articles), which is largely inefficient as crowd workers are not interested in the reading tasks and may generate unexpected annotation noise. We develop a knowledge-independent crowdsourcing strategy that exploits the logic reasoning ability of crowd workers to efficiently extract the semantic logic information from the COVID-19 posts for the NLE generation.

3 Problem Statement

Definition 1. COVID-19 Post (p): A COVID-19 post is a piece of short text from social media that describes a news topic or statement related to COVID-19 (e.g., the post in Figure 1). In particular, we define a set of N COVID-19 posts as P = {p1, . . . , pN}.

Definition 2. COVID-19 Post Label (y): A COVID-19 post pn is considered as misleading (yn = 1) if it contains partial or entire false information. Otherwise, the COVID-19 post is considered as non-misleading (yn = 0). In this paper, we assume that a COVID-19 post only contains one COVID-19 related claim that is either misleading or non-misleading.

Definition 3. COVID-19 Article (a): An article related to COVID-19, such as a COVID-19 related medical publication (e.g., the New England Journal of Medicine) or a debunking article from credible fact-checking websites (e.g., FactCheck.org). Such articles usually contain COVID-19 related natural language texts that explicitly disprove the misinformation in COVID-19 posts. In particular, we define a set of M COVID-19 articles as D = {a1, a2, . . . , aM}, where am = {sm,1, . . . , sm,Ls } is the mth article from D and sm,l is the lth COVID-19 statement (i.e., lth sentence) in am. Given a COVID-19 post, a COVID-19 statement is considered as a potential NLE if it can be used to partially or completely judge the truthfulness of the posts.

Definition 4. COVID-19 Expert (X): The COVID-19 experts are professionals who have COVID-19 medical or healthcare knowledge to fully understand COVID-19 posts.

Definition 5. Crowd Worker (C): The crowd workers are a set of users in online crowdsourcing platforms (e.g., Amazon MTurk) who are not proved to have COVID-19 related professional knowledge.

Definition 6. COVID-19 Natural Language Explanation (NLE) (z): Given a COVID-19 post pn, a COVID-19 NLE is a statement that explicitly explains why the post is misleading or not (e.g., the texts in Figure 1c). In particular, the NLE of pn is defined as zn = {zn,1, . . . , zn,Lz } where Lz is the total number of words in zn.

Definition 7. NLE-based COVID-19 Misinformation Detection Model (M): Given a COVID-19 post pn, our NLEbased COVID-19 misinformation detection model M is expected to generate 1) the binary prediction to indicate if pn is misleading or not; 2) the NLE that explains its prediction.

Using the definitions above, our NLE-based COVID-19 misinformation detection problem is formally defined as:

where ˆ yn is the misinformation prediction and ˆ zn is the generated NLE by M. We expect CEA-COVID to maximize both the misinformation detection accuracy and the explainability.

4 Solution

4.1 Crowdsourcing Logical Reasoning Interface

The first module in our scheme is Crowdsourcing Logical Reasoning Interface (CLRI) that is designed to create crowdsourcing tasks for online crowd workers to explain why the assigned COVID-19 posts are misleading (or not). We observe that it is challenging for crowd workers to generate reasonable explanations (e.g., the NLE in Figure 1c) even if they are given the truthfulness of the COVID-19 posts due to their lack of professional COVID-19 knowledge.

To solve the above problem, we leverage the general logical reasoning ability of crowd workers to analyze the semantic logic embedded in the COVID-19 posts without requiring them to understand domain-specific terms (e.g., "Chlorine"). We show an example crowdsourcing task in Figure 2. In particular, we hide the binary label of the assigned COVID-19 post and assume all the posts as non-misleading, which effectively saves the time of crowd workers from judging the truthfulness of the post and identifying the misinformation. We formally define the semantic logic below.

Definition 8. Semantic Logic: The semantic logic of a COVID-19 post denotes the relations between different entities in the post regardless of the semantic meaning of the entities. For example, a crowd worker may not know the semantic meaning of "Ethylene oxide" in Figure 2 and hence cannot generate related NLEs. However, it is not difficult for the crowd worker to understand the semantic logic of the post (e.g., "Ethylene oxide somehow damages human DNA").

Given a crowdsourcing task with a COVID-19 post, a crowd worker is expected to generate a piece of semantic logic of the post and submit it as the response. Therefore, the semantic logic of the post is expected to be exactly contradicted or consistent with the truthfulness of the corresponding COVID-19 post (i.e., misleading or non-misleading) (i.e., ground-truth label of the post). We adopt human intelligence from the crowd to create the semantic logic because such logic information is non-trivial to be captured by data-driven AI models that usually lack semantic reasoning ability to generate abstracted logic information [Ayoub et al., 2021]. Given PC = {p1, . . . , pNc } as the set of COVID-19 posts in crowdsourcing tasks, we define Ωc = {ω1, . . . , ωNc } as the collected semantic logic from crowd workers.

4.2 Logic-driven NLEGraph Network

Given the generated semantic logic of COVID-19 posts from the CLRI, we develop the Logic-driven NLE Graph Network (LNGN) module that extracts high-relevant knowledge facts from the COVID-19 articles as potential NLEs of the posts and leverages the semantic logic as effective regularization. In particular, we first model the set of COVID-19 articles D as a multi-relational knowledge graph. We extract the pre-annotated entities from COVID-19 posts (e.g., "Chlorine", "human tissues") as initial graph nodes and then identify both the related entities (e.g., compound entities) as graph nodes and their relations as graph edges (e.g., "corrosive injury") by developing heuristic semantic word extraction scripts. The reason of constructing the knowledge graph is to effectively integrate the COVID-19 posts with the structured knowledge facts that are converted from the unstructured human language texts of COVID-19 articles. In particular, given each COVID-19 statement sm,l of the COVID19 article am ∈ D, we define Vm,l = {v1, . . . , vKm,l } and Rm,l = {r1, . . . , rQm,l } as the set of entities and relations extracted from sm,l, respectively. If two entities vk1 and vk2 are connected through a relation rq, we denote them as a knowledge triple Tm,l,k1,k2 = (vk1 , rq, vk2 ). Therefore, we construct the NLE graph network (NLE-GN) from all COVID19 articles and denote it as G = (V,R, T ,A) where V = {v1, . . . , vK}, R = {r1, . . . , rQ} and T = {T1, . . . , TE} represent the union entities, relations and knowledge triples of G, respectively. A denotes a K ×K matrix that contains binary values to denote whether two entities from V are connected or not.

Given the constructed NLE-GN G and a COVID-19 post pn, LNGN aims to classify pn as misleading or not by identifying the relevant knowledge facts with pn from G. Such knowledge facts can be further traced back to original COVID-19 statements as potential NLE labels. Specifically, LNGN first embeds G and pn into high-dimensional embeddings by applying a BERT-based entity encoder to accurately encode each term from G and pn into the same vector space. We denote the generated embeddings of the entities, relations, COVID-19 posts and semantic logic as ˜ V ∈ RK×d , ˜R ∈ RQ×d , ˜ pn ∈ R1×d and ˜ωn ∈ R1×d respectively.

We then develop a post-guided knowledge graph propagation strategy for the LNGN to identify the relevant knowledge facts from G for the input COVID-19 post pn by jointly exploring the semantic information of both G and pn as ˜ vi = ReLU(∑(j,r,i)∈T ∗ 1 zj Wi,j ˜ vjAi,j) where ˜ vi and ˜ vj are ith and j th encoded entities from ˜ V . T ∗ ⊂ T denotes the set of knowledge triples consisting of ˜ vi. A is K × K matrix derived by jointly aggregating the semantic information from pn and the adjacent matrix A with binary values as A = (˜ VW1(˜ pn)T + ˜ pnW2(˜ V)T ) ⊙ A. After the knowledge propagation, we generate the embeddings of knowledge triples from G by multiplying the entity embeddings with the relation embedding as ˜ Ti,q,j = ˜ vi ⊙ ˜ rq ⊙ ˜ vj . Each knowledge triple embedding corresponds to a COVID-19 statement from COVID-19 articles as potential NLE label of pn. However, it is extremely challenging for the LNGN to accurately identify the relevant knowledge triples due to the lack of direct training supervision on the knowledge triple retrieval process.

To solve the above problem, we leverage the semantic logic ωn of pn from the CLRI to effectively retrieve relevant knowledge triples by exploring the logical consistency between ωn and T . We define the logic consistency as:

Definition 9. Logical Consistency (LC): Given a knowledge triple Te ∈ T and ωn of pn, the logical consistency is a binary value (i.e., -1 or 1) to denote if the semantic information of Te matches with ωn. For example, the LC is -1 between the example semantic logic in Figure 2 and the knowledge triple ("Ethylene oxide" not affect -----→ "Human DNA").

Given the above definition, the logical consistency between the embedding ˜ Te and ˜ωn is expected to be high if pn is nonmisleading (i.e., yn = 0). Otherwise, the logical consistency is low because the ωn is generated by falsely assuming pn is non-misleading. Therefore, we define the logical consistency loss as Llogic = |cos(˜ωn, ˜ Ttop) - (-1)1[yn=1] | + ϵ where ˜ Ttop ∈ ˜ T denotes the embedding with the highest attention score that is generated by transforming the embeddings ˜ T to 1-dimensional vector as ATTn = Softmax(Watt(˜ T )T ). We then multiply ˜ T with ATTn and apply mean-pooling to generate a single post-level embedding zn ∈ R1×d that is further transformed to ˆ yn ∈ R2 as the output probability of pn being misleading or not. The cross entropy loss Lce is finally applied to train LNGN with gradient descent optimization.

4.3 Expert-driven Graph Knowledge Updater

After optimizing the LNGN, the Expert-driven Graph Knowledge Updater (EGKU) aims to update the knowledge facts from G by efficiently tasking COVID-19 experts to investigate the truthfulness of both the knowledge triples from G and the semantic logic of COVID-19 posts. However, it is extremely time-consuming if the COVID-19 experts are tasked to check each semantic logic and knowledge triple due to the large amount of available COVID-19 posts and articles.

To address the limitation, we only expect COVID-19 experts to investigate specific pairs of semantic logic and knowledge triples by specifying the logical confidence relation below.

Definition 10. Logical Confidence Relation (LCR): Given a knowledge triple Te ∈ T and a semantic logic ωn of pn, the LCR indicates how confident we believe that they have certain consistent or contradicted local consistency. In particular, the LCR between them is considered to be high if cos(˜ωn, ˜ Ttop) in Llogic is lower than ξ and higher than 1 - ξ where (0.5 < ξ < 1) is the pre-defined threshold. Otherwise, the LCR is considered to be low.

For the pairs of semantic logic and knowledge triples with low LCR, we design a ranking-based knowledge graph updating strategy that tasks the COVID-19 experts to judge which semantic information (i.e., semantic logic or knowledge triple) is considered as true. The reason of applying the ranking-based strategy is to minimize the potential opinion bias of COVID-19 experts given the fact that humans usually make more fair estimations if they compare different candidate options rather than solely judging a single option [Burton, 2004]. In particular, if the COVID-19 experts believe the semantic logic contains true COVID-19 related information, the knowledge triple in the pair is removed from NLEGN while the semantic logic is added. Otherwise, the crowd worker who submitted the semantic logic will be notified to provide more reasonable logic information in the following crowdsourcing tasks. After the COVID-19 experts update the NLE-GN, we levearge the updated NLE-GN to optimize the LNGN again. We repeat such process in LNGN and EGKU for Φ rounds defined as a hyper-parameter of CEA-COVID.

4.4 Knowledge Graph-based NLEGenerator

After the above optimization process, we generate the relevant knowledge triples for COVID-19 posts and track the knowledge triples back to corresponding COVID-19 statements. We build the Knowledge Graph-based NLE Generator (KGNG) and optimize KGNG with the retrieved COVID-19 statements as training data. We develop the KGNG based on the multi-head attention-based transformer framework [Ma et al., 2019] and propose a novel word-level weighted discriminator as the loss function. In particular, the KGNG takes each COVID-19 post pn = {wn,1,wn,2, . . . ,wn,Lp } as input and outputs the predicted NLE as ˆ zn = {ˆ zn,1, ˆ zn,2, . . . , ˆ zn,Lz }. Given Top-K retrieved COVID-19 statements from the NLEGN S∗n = {s1, s2, . . . , sK}, we calculate the weighted per-word cross entropy loss as LNLE = 1Z ∑w N(w) × CrossEntropy(w, ˆw) + LD(ˆ zn, S∗n ) where w denotes each word from the COVID-19 statements and ˆw is the prediction. N(w) is the word frequency. LD denotes the language generation discriminator that discriminate the generated NLE from the COVID-19 statements to generate more smooth human natural language textual results [Chen et al., 2020].

5 Evaluation

We conduct extensive experiments on two real-world COVID-19 misinformation datasets to explore Q1) the COVID-19 misinformation detection performance, Q2) the explainability, and Q3) the robustness of CEA-COVID.

5.1 Dataset and Experiment Setup

Dataset. We use two public COVID-19 misinformation datasets in our experiments: i) COVIDRumor [Cheng et al., 2021] is a COVID-19 misinformation dataset that consists of 3, 580 misleading and 1, 699 non-misleadinig COVID-19 news and social media posts, and ii) CONSTRAINT [Patwa et al., 2021] is a large-scale COVID-19 fake news dataset that consists of 5, 600 misleading and 5, 100 non-misleading posts. We collect two types of COVID-19 articles to extract COVID-19 knowledge facts as follows: i) professional factchecking articles that are published by fact-checking journalists on major fact-checking websites (e.g., politifact.com), and ii) COVID-19 medical papers that contain the latest COVID-19 related publications from major archival services (e.g., bioRxiv). In particular, we collect 1, 132 professional fact-checking articles and 247 COVID-19 medical papers that are considered as creditable after manual verification process.

Crowd & Expert Configurations. For each COVID-19 misinformation dataset, we generate 2,000 crowdsourcing tasks with each task containing a COVID-19 post randomly selected from the dataset. For each crowdsourcing task, we assign three independent Amazon MTurk workers to generate the semantic logic for the post. The crowd workers are selected only if they have a 95% or higher Human Intelligence Task (HIT) rate to ensure high quality of their answers. We set the payment to all crowd workers well above the minimum requirement fromMTurk. For COVID-19 experts, we design a screening exam that contains 50 multi-choice questions on human health and COVID-19. We invite participants with medical and COVID-19 knowledge to take the exam and finally recruit 5 experts who achieve the full score in the exam.

5.2 Baselines

BertCOVID [Ayoub et al., 2021]: an explainable COVID19 misinformation detection framework that applies wordlevel attention as explanations.

HC-COVID [Kou et al., 2022]: a knowledge graph based COVID-19 misinformation detection framework that constructs a crowdsourced knowledge graph to retrieve the knowledge facts from the knowledge graph as explanations.

ExpertLoop [M´endez M´endez et al., 2019]: an expertbased annotation framework that tasks experts to validate the uncertain data annotations from common crowd workers.

ATT-KG [Wang et al., 2019]: an attentive knowledge graph recommendation network that extracts the relevant knowledge from knowledge graph based on attention degrees.

We also adopt two non-explainable COVID-19 misinformation detection models, EnsembleCOVID [Al-Rakhami and Al-Amri, 2020] and COVID-RNN [Alenezi and Alqenaei, 2021], to rigorously evaluate the misinformation detection accuracy of CEA-COVID when there is no regulation on model explainability during optimization.

5.3 Evaluation Results

COVID-19 Misinformation Detection Performance (Q1)

To answer Q1, we first evaluate the COVID-19 misinformation detection performance of the CEA-COVID and all the baselines on both datasets. The evaluation results are shown in Table 1. We observe that CEA-COVID consistently outperforms all baselines on both datasets. Such a performance gain can be attributed to the dynamic COVID-19 knowledge facts in the NLE-GN that effectively infer the truthfulness of unseen COVID-19 posts. Moreover, we observe that the CEA-COVID significantly outperforms ExpertLoop that also utilizes the same crowd workers and COVID-19 experts to generate NLE labels. Such an observation further verifies that the performance gain of CEA-COVID is not due to the crowdsourced labels as additional supervision but the effective human-AI interaction strategy in the LNGN and EGKU modules that accurately identifies relevant knowledge facts from NLE-GN for misinformation detection.

COVID-19 Post Explanation Performance (Q2)

To answer Q2, we study the explainability performance of the CEA-COVID through real-world user studies. In particular, we compare the explainability of CEA-COVID with the BertCOVID, HC-COVID, ExpertLoop and ATT-KG which generate explanations for the misinformation detection results. We randomly select 50 misleading and 50 non-misleading COVID-19 posts from the testing set of both datasets and perform the explainability evaluation. In the user study, we carry out two experiments using Amazon MTurk.

First, we study the explainability performance by comparing the quality of the NLEs generated from CEA-COVID with the outputs of other schemes. For the generated explanation of each compared scheme and each COVID-19 post, we recruit 5 MTurk workers to select one scheme from the five compared schemes that can best explain the detection results of each input COVID-19 post. The explainability performance is evaluated with two commonly-adopted metrics as in [Kou et al., 2022]: 1) the Percentage of Posts from which the majority of workers pick their preferable explanation by a scheme, and 2) the Percentage of Workers who choose the explanations from a scheme as their favorite. The results are summarized in Figure 3. We observe that CEA-COVID significantly outperforms the compared baseline schemes in terms of both metrics. The performance gains demonstrate the effectiveness of the KGNG module that optimizes the NLE generation with crowdsourced NLE labels to obtain more relevant and accurate NLEs than the baselines.

Second, we evaluate the efficiency of the crowdsourcing logical reasoning tasks from the CLRI module by investigating the time and quality of the responses from crowd workers. In particular, we select HC-COVID and ExpertLoop as compared schemes because they are the only two schemes that generate explanation labels of COVID-19 posts by utilizing the crowd intelligence. For each COVID-19 post, we recruit 3 crowd workers to complete the crowdsourcing tasks specified by each scheme. We design three different metrics to evaluate the efficiency of each compared scheme: 1) Anno. that indicates the average time of a crowd worker to complete each crowdsourcing task; 2) Subm. that indicates the total waiting time to collect all responses from the crowd workers and 3) Invalid that indicates the number of invalid responses (e.g., randomly typed texts) that are manually identified by the researchers. We summarize the evaluation results in Table 2. We observe that CEA-COVID significantly outperforms the two compared baselines with a much lower time cost and a smaller number of invalid answers.

Robustness Study (Q3)

We study the robustness of CEA-COVID by tuning 1) the percentage of crowdsourcing tasks from the CLRI module (Ω), and 2) the number of iteration rounds for optimizing CEACOVID (Φ). We vary Ω from 0% to 100%, and Φ from 1 to 4. We report the results in Figure 4. We observe that the detection performance of CEA-COVID gradually plateaus as Ω increases. This is because the increased number of tasks may contain similar COVID-19 posts with duplicated semantic logic. We also note that CEA-COVID's performance gradually plateaus as Φ increases and even decrease a bit in the COVIDRumor dataset. A possible reason is that too many optimization rounds in CEA-COVID may reduce the accuracy of semantic logic from crowd workers or knowledge facts from COVID-19 experts as the workload increases.

6 Conclusion

This paper presents a novel CEA-COVID framework to address the NLE-based COVID-19 misinformation detection problem. CEA-COVID designs a tripartite crowd-expert-AI framework that jointly leverages the collective strengths of crowd workers, COVID-19 experts, and AI models to generate NLEs for detecting and explaining COVID-19 misinformation. Evaluation results on two real-world datasets demonstrate a higher detection accuracy and better explainability of CEA-COVID than state-of-the-arts in identifying COVID-19 misinformation on social media.

Acknowledgments

This research is supported in part by the National Science Foundation under Grant No. CHE-2105032, IIS-2008228, CNS-1845639, CNS-1831669. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.