On Optimizing Model Generality in AI-based Disaster Damage Assessment: A Subjective Logic-driven Crowd-AI Hybrid Learning Approach

1 Introduction

The increasing frequency and severity of natural disaster events (e.g., hurricanes, earthquakes, wildfires) have posed serious challenges to human society with significant casualties and enormous economic losses [McEntire, 2021]. For example, the recent Turkey-Syria earthquake has directly impacted 23 million people across Turkey and Syria along with over 50, 000 deaths and $80 billion loss. 1 Damage assessment is an essential process during disaster response that aims to acquire accurate and timely information about the damages caused by the disaster, assist local/federal authorities (e.g., FEMA, public health agencies, police departments), and civil society stakeholders (e.g., regional red cross society, community disaster experts, local NGOs) in their decision making process, and prevent further damages [Kankanamge et al., 2020]. The recent advances in AI have enabled more effective and scalable AI-driven solutions for timely disaster damage assessment, which aligns well with the "disaster risk reduction" objective under "Sustainable Cities and Communities" (i.e., Goal 11 of United Nations' Sustainable Development Goals (UN SDGs)). Meanwhile, the proliferation of social media also provides a pervasive data source to obtain realtime situation awareness of disaster events from common citizens [Wang et al., 2019a]. In this paper, we focus on the AI-based damage assessment (ADA) application, where the goal is to leverage the advanced AI techniques (e.g., deep convolutional network, graph neural network, transformer) to automatically assess the disaster damage severity using social media imagery data [Nguyen et al., 2017]. Specifically, we focus on addressing the limitation of current ADA models that are often optimized only for a single disaster event and lack the generality to provide accurate performance across different disaster events. We refer to such a knowledge gap as the ADA model generality problem.

1 https://www.redcross.org.uk/stories/disasters-andemergencies/world/turkey-syria-earthquake

Recent progress in AI and deep learning have been made to improve the performance of ADA applications [Imran et al., 2022; Li et al., 2019; Mouzannar et al., 2018; Zhang et al., 2021]. Current solutions often focus on designing tailored AI models that can accurately identify the event-specific damage visual characteristics for a specific disaster event to ensure accurate ADA performance [Li et al., 2019]. We refer to the optimized performance of a customized ADA model for a specific disaster event as the model's specificity to that event. However, we observe that current ADA solutions often lack model generality, which leads to poor performance when the model is applied to a disaster event that is different from the one on which the model was trained [Zhang et al., 2021]. For example, in Figure 1, we observe that (A) and (B) share similar visual features of grey sky but end up with completely different damage severity levels. An ADA model trained for the wildfire event mistakenly identifies the image in (B) as severe damage, and an ADA model trained for the hurricane event incorrectly classifies the image in (A) as no damage due to the lack of generality of the AI models. A possible solution to address the ADA model generality problem is to simultaneously train multiple ADA models, one for each specific disaster event [Li et al., 2019]. However, a critical problem is that such a solution requires a good amount of high-quality training data from each event, which is not always available [Kumar et al., 2020]. The lack of event-specific training data often leads to an overfitting issue where the trained ADA models fail to learn the event-specific damage visual features for the studied event, leading to an undesirable performance loss of the events that lack training data [Saunders, 2022].

In this paper, we jointly explore the different yet complementary AI and human intelligence from crowdsourcing systems such as Amazon Mechanical Turk 2 (i.e., crowd intelligence) to address the ADA model generality problem. Unlike the AI models that often rely on the training data available in each disaster event to generate the estimated ADA labels, humans can often reasonably estimate the ADA labels across different disaster events without the need for training data from such events [Fuchs, 2022; Zhang et al., 2022a]. For example, in Figure 1, we can clearly understand that the grey sky in (A) indicates large smokes caused by a wildfire but reflects a heavy cumulonimbus in (B). As a result, crowd intelligence is often more generalizable in identifying disaster damages across different disaster events. However, unlike the AI models that can maintain a certain level of consistency and accuracy once they are trained for a specific disaster event, crowd workers often make inadvertent mistakes on the imagery data from a disaster event [Draws et al., 2021]. Motivated by the above observations, this paper develops a hybrid crowd-AI collaborative learning framework that jointly leverages the specificity of AI and the generality of human intelligence to address the ADA model generality problem. However, two technical challenges exist in designing our framework.

The first challenge is how to effectively optimize the ADA model generality without sacrificing its specificity on an individual disaster event. A possible solution to tackle the ADA model generality problem is to train an ADA model using the training data from all studied disaster events so that the ADA model instance can learn the disaster-related visual features from all trained events. However, such a one-size-fits-all solution can lose the sensitivity on the event-specific visual features and lead to undesirable performance loss on specific events of interest [Ghifary et al., 2016]. On the other hand, recent efforts have been made to tackle the AI model generality problem [Zhang et al., 2020; Sankaranarayanan et al., 2018]. Those solutions often leverage the divergencebased or adversarial-based neural network designs to transfer or adapt the ADA model learned from a source event (with sufficient training data) to a target event (with little training data) that shares similar damage visual characteristics with the source event. However, the actual ADA performance largely depends on the level of similarity between the source and target events, and an appropriate source event is not guaranteed to exist [Zhang et al., 2021].

2 https://www.mturk.com/

The second challenge is how to effectively integrate the complementary yet different AI and crowd intelligence to address the ADA model generality problem. In particular, a few recent crowd-AI collaborative systems have been developed to leverage crowd intelligence to troubleshoot the AI failure cases or retrain the AI models to boost the application performance [Sener and Savarese, 2018; Zhang et al., 2019]. However, those solutions are not designed to address the AI model generality problem and can lead to suboptimal performance when they are directly applied to different disaster events [Risi and Togelius, 2020]. In addition, we observe that the imperfect crowd labels could confuse the AI models to identify incorrect visual features that might further impair the model generality across different events [Rolnick et al., 2017; Zhang et al., 2022b]. Moreover, there also exist efforts in active learning and label aggregation that can be applied to fuse the inputs from both AI and crowd intelligence [Gemalmaz and Yin, 2021; Hube et al., 2019]. Those solutions often leverage analytical approaches (e.g., Bayesian optimization, maximum likelihood estimation) to boost the aggregated label accuracy. However, those approaches do not jointly model the specificity of AI and the generality of crowd intelligence to achieve the optimal ADA performance on individual disaster events.

To address the above challenges, this paper develops CollabGeneral, a subjective logic-driven crowd-AI collaborative learning framework that exploits AI and crowd intelligence to address the ADA model generality problem. To address the first challenge, we develop a novel deep model optimization framework that designs a generality-aware network optimization function design to optimize the trade-off between the ADA model's generality and specificity. To address the second challenge, we model the crowd intelligence and AI through a novel subjective logic framework to fuse the intelligence from both humans and AI. We work closely with domain experts and local community stakeholders in disaster response to provide the in-the-field know-how to validate our framework. To the best of our knowledge, CollabGeneral is the first crowd-AI hybrid approach to tackle the AI model generality problem in ADA applications. We also envision that our framework can be applied to address the AI model generality problem in a much broader set of AIdriven applications beyond ADA (e.g., misinformation detection, intelligent transportation, smart health). We evaluate CollabGeneral through four real-world ADA datasets and the results demonstrate that CollabGeneral consistently outperforms state-of-the-art deep neural networks, crowd-AI models, and AI model generality frameworks by improving ADA model generality under a rich set of evaluation scenarios.

2 RelatedWork

AI-based Disaster Informatics: AI-based disaster informatics received a good amount of attention in recent years due to its efficiency, scalability, and effectiveness in providing accurate and timely information during the disaster events [Sun et al., 2020; Zhang et al., 2023]. From disaster risk and damage identification [Metaxa-Kakavouli et al., 2018] to emergency response and recovery [Soden and Owen, 2021], AI-based disaster informatics plays a vital role in reducing the negative impacts of natural disasters [Soden and Palen, 2018]. Disaster damage assessment using social media data is an important application in AI-based disaster informatics, where timely observations of the disaster damages from social media posts are leveraged to obtain the situational awareness (e.g., damage severity, casualties and injuries) during devastating natural disasters [Zade et al., 2018]. For example, Ning et al. designed a deep convolutional network based context information extraction framework that explores the realtime Twitter posts to identify highly impacted areas and track the dynamic damage severity during flood disasters [Ning et al., 2020]. Mangalathu et al. developed a recurrent neural network model that analyzes earthquake-related social media posts to estimate building damage severity in earthquake events [Mangalathu and Burton, 2019]. Barmpoutis et al. designed a convolutional neural network framework that leverages the multimodal social media posts to detect fire regions [Barmpoutis et al., 2019]. However, current ADA models often focus on optimizing the performance for a single disaster event and lack model generality when performing ADA tasks on different events. In contrast, we design a crowd-AI hybrid learning solution to tackle the ADA model generality problem in AI-based disaster informatics.

AI Model Generality: The lack of generality is a fundamental issue in AI applications, and recent efforts have been made to improve the generality for AI models [Li et al., 2020; Sankaranarayanan et al., 2018; Kini et al., 2021; Wang et al., 2019b; Chan et al., 2018; Zong et al., 2023]. For instance, Li et al. proposed an unsupervised domain adaptation solution that incorporates a cluster-based regularization to improve the image classification performance across different domains [Li et al., 2020]. Kini et al. designed a vector-scaling optimization framework that leverages a multiplicative logit adjustment mechanism to improve the crossdomain model generality in domain-sensitive image classification [Kini et al., 2021]. Wang et al. developed a symmetric cross-entropy optimization framework that leverages a counterpart reverse optimization design to minimize the domainwise overfitting problem and improve the model generality in natural scene classification [Wang et al., 2019b]. However, current AI model generalization solutions often sacrifice the AI model's specificity when optimizing the generality of the model. To the best of our knowledge, CollabGeneral is the first crowd-AI hybrid learning framework that explicitly leverages the complementary nature of AI and crowd intelligence to optimize the ADA model generality without sacrificing the ADA performance in each studied disaster event.

3 ProblemDescription

Definition 1. Disaster Event (D): We define D = {D1, D2, ..., DT } to be a set of studied disaster events where Dt represent tth studied disaster event, and T is the total number of disaster events in the studied ADA application.

Definition 2. Disaster-related Imagery Data (X): We define X = {X1,X2, ...,XI} to be a set of disaster-related social media imagery data posted during different disaster events for the ADA application (e.g., Figure 1). In particular, Xi indicates the ith image sample, and I represents the total number of studied image samples. In addition, we define XDt to be the subset of image samples in X collected from the tth studied disaster event Dt.

Definition 3. Class Label Estimated by AI (̂ Y A): This work focus on the physical status based disaster scene classification in ADA applications. For instance, in a prior ADA study [Nguyen et al., 2017], the disaster damage severity is categorized into three different classes: no/minor damage, medium damage, and severe damage. In particular, we define ̂ Y A as the set of class labels estimated by the AI model for all imagery data X, where ̂ Y A i represents the estimated class label for Xi.

In our paper, we focus on exploring both AI and crowd intelligence to tackle the ADA model generality problem. Therefore, we further define a few key definitions on acquiring the crowdsourcing-based human intelligence.

Definition 4. Crowd Intelligence Query (Q): We define Q as a crowdsourcing task to acquire human intelligence from crowd workers. In particular, our CollabGeneral framework focuses on identifying a subset of image samples in X that the AI models fail to provide accurate estimation results due to their lack of generality [Ren et al., 2021]. The identified image samples are forwarded to a crowdsourcing platform where each image in Q is annotated by a set of B crowd workers. We define W = {W1,W2, ...,WB} as the set of crowd workers participating in Q. Wb indicates the bth crowd worker. We present the details of crowd intelligence query tasks in Section 4.

Definition 5. Class Label Annotated by CrowdWorkers ( ̂ Y W ): We define ̂ Y W as the set of class labels annotated by crowd workers for the imagery data that are selected in Q. In addition, we define ̂ Y Wb as the set of class labels contributed by a crowd worker Wb where ̂ Y Wb i is the class label contributed byWb for image sample Xi in Q.

Definition 6. Class Label Identified by CollabGeneral ( ̂ Y ): We define ̂ Y as the final outputs of the CollabGeneral framework by leveraging the class labels returned by AI (i.e., ̂ Y A) and crowd workers (i.e., ̂ Y W ). Specifically, ̂ Yi represents the final identified class label for image sample Xi.

The goal of our ADA model generality problem is to utilize the collaborative strengths of AI and crowd intelligence to achieve the optimal ADA performance in each studied disaster event as follows:

where ̂ YDt and YDt indicate the estimated and ground-truth class labels for imagery data XDt from disaster event Dt, respectively. Note that, instead of learning an individually tailored ADA model for each studied disaster event, our CollabGeneral framework leverages the image samples X from all events to learn a holistic and accurate ADA model that generates the optimized results for each event.

4 Solution

CollabGeneral is a crowd-AI hybrid learning framework that integrates AI and crowd intelligence to optimize model generality in AI-based disaster damage assessment applications.

1) Generality-aware Deep Optimization (GDO): it designs a novel deep model optimization scheme that effectively learns a set of ADA model instances to achieve a good tradeoff between AI model generality and specificity through a novel generality-aware network optimization design. The learned ADA model instances are then used to identify the subset of image samples for crowd intelligence query.

2) Subjective logic-driven Crowd-AI Fusion (SCF): it develops a principled subjective logic-driven crowd-AI fusion framework to effectively integrate the class labels generated by the ADA model instances from GDO module and the crowd labels returned by crowd intelligence query to derive accurate ADA results for each studied disaster event.

4.1 Generality-aware Deep Optimization

We first present the generality-aware deep network optimization design to learn a set of ADA model instances that have a high likelihood of achieving an optimized trade-off between the model generality and specificity in ADA applications. We first introduce a key definition for our GDO module.

Definition 7. Deep Estimation Network (Φ): We define Φ to be the deep estimation network (i.e., AI model) in the GDO module that estimates the class labels from the input image samples. Rather than reinventing the wheel, we set Φ to be a representative convolutional neural network (e.g., ResNet, VGG, DenseNet) that is designed to perform the image-based multi-class classification tasks.

Given the deep estimation network Φ, our next step is to learn the optimal network instance of Φ for accurate eventwise ADA performance. To that end, our GDO module introduces two sets of loss functions to explicitly supervise the network optimization process and derive the optimal network instance that can achieve a good trade-off between ADA model generality and specificity. We first define the accuracy-aware loss function for Φ as:

where L1 denotes the accuracy-aware loss function for Φ. Dt is a disaster event from the set of studied events D. K denotes the number of unique classes in the ADA application of interest. ̂ Y Φ Dt and YDt indicate the estimated class labels from Φ and ground-truth class labels for all imagery data from disaster event Dt, respectively. || · ||2 is the L2-norm of a matrix. The objective of the accuracy-aware loss is to supervise Φ to accurately estimate the class labels from all input imagery data. However, a limitation of L1 loss function is that L1 only focuses on the overall ADA performance but may not supervise Φ to achieve optimized ADA performance on each individual disaster event. Therefore, we further define the generality-aware loss function for Φ to address such a limitation as:

where L2 is the generality-aware loss function for Φ. Dt1 ,Dt2 represent any two different disaster events from the set of studied disaster events D. ̂ Y Φ Dt1 and ̂ Y Φ Dt2 indicate the estimated class labels for all imagery data from event Dt1 and Dt2 , respectively. YDt1 and YDt2 indicate the ground-truth class labels for all imagery data from event Dt1 and Dt2 , respectively. We then combine the two loss functions to derive the overall loss function for Φ to learn the optimal network instance of Φ as:

Using the overall loss function above, the optimal network instances of Φ can be learned by investigating the trade-off between the exploitation and exploration during the network optimization process through a budget-constrained multiarmed bandit learning process [Feurer and Hutter, 2019]. On the one hand, we keep tuning the same network instance that achieves the low value for LOverall. On the other hand, we take action to attempt new network instances to prevent the model from being trapped into a local optimum. Such a optimization strategy could jointly explore the large network instance space while finding the optimal network instance for Φ.

After performing the budget-constrained multi-armed bandit learning process, one possible solution to obtain the optimal network instance is to use the network instance with the lowest value of LOverall as the optimal network instance. However, the optimized network instance could be overfitted to the training/validation data and lead to non-negligible performance degradation when it is applied to the testing data due to the potential feature discrepancy between the training/validation and testing sets [Saunders, 2022]. To address such an issue, our GDO module not only exploits the network instances with the lowest value of LOverall but also explores other candidate network instances with low values of LOverall. We formally define the network instances generated by our GDO module as follows.

Definition 8. Optimized Network Instance Set (M): We define M = {M1,M2, ...,MJ} as a set of network instances learned by the GDO module, which includes network instances with top J lowest values in LOverall. In addition, Mj indicate the j th learned network instance.

Note that all network instances in M are the instances of the deep estimation network Φ (Definition 7). To generate different network instances in M, our GDO module keeps tracking the LOverall of different network instances generated during one budget-constrained multi-armed bandit learning process. Our GDO module then adds the network instances with top J lowest values in LOverall to M. The above design avoids the low computational efficiency of performing the budget-constrained multi-armed bandit learning process J times to generate different network instances inM.

Our CollabGeneral then jointly leverages the identified network instances and crowd intelligence to derive accurate ADA results for all studied disaster events, which will be discussed in the next subsection.

4.2 Subjective Logic-driven Crowd-AI Fusion

In this module, we design a novel subjective logic-driven crowd-AI fusion framework to fuse the AI and crowd intelligence to derive the accurate ADA results for all studied disaster events to address the ADA model generality problem.

We first discuss how to perform crowd intelligence query Q to collect crowd intelligence for the SCF module. We observe that it is impractical to query the crowd intelligence for all studied image samples due to the budget and resource constraints, which is especially challenging in ADA applications with massive social media data inputs [Li et al., 2019]. Therefore, our SCF module samples a subset of image samples for Q in which different network instances in M (Definition 8) cannot reach a consensus on. We first measure the divergence of the class labels estimated by all network instances inM for each image sample Xi using Shannon entropy [Lin, 1991]. The divergence indicates the degree of disagreement between different network instances in M on the estimated class label for Xi. We then select the image samples with top δ × I highest divergence for Q. Here, δ indicates the percentage of studied disaster-related imagery data that are sampled for Q. δ is determined by the trade-off between the ADA model performance and the crowdsourcing cost in the ADA application of interest. I is the total number of studied images.

Our next step is to effectively fuse the crowd labels returned by Q with the estimated labels generated by different network instances inM. In particular, we define:

Definition 9. Crowd-AI Fusion Committee (S): We define S = {S1, S2, ..., SC} as a crowd-AI fusion committee, which contains all J different optimized network instances M learned by the GDO module and all B different crowd workers W in an ADA application. In particular, we have S =M ∪W, where C = J +B. C is the size of committee S, and Sc is a committee member in S (i.e., either an AI network instance or a crowd worker.

The goal of our SCF module is to effectively fuse the inputs from all members in S to derive the accurate ADA labels for the studied disaster events. To that end, we first define the "opinion" of each committee member towards the class label of each image sample through subjective logic, a probabilistic logic that models the epistemic uncertainty and source trust when combining the opinions from different sources [Jøsang, 2016]. In our paper, we leverage the subjective logic to explicitly model each committee member's uncertainty and reliability in estimating the ADA labels for all disaster events.

Definition 10. Committee Member Opinion Entity (E): We define EkSc = {T k Sc , F k Sc , U k Sc } to represent the opinion of a member Sc on whether an image sample belongs to a particular class k or not. In particular, we have:

where T k Sc and F k Sc indicates Sc's belief and disbelief in the class label of an image sample to be k, respectively. U k Sc indicates Sc's uncertainty in determining if the class label of an image sample to be k or not.

Given the opinion entity of each committee member, we can utilize the consensus operation from subjective logic to combine the opinions from different committee members. Consensus operation is a key operation in subjective logic that is used to determine the shared belief and uncertainty of two sources by considering the individual belief and uncertainty of each source. In particular, we can use the consensus operation ⊕ to combine the opinions from any two committee member Sp and Sq as follows:

where EK Sp,Sq indicates the opinion entity after combining the opinions from both Sp and Sq, which indicates their collective opinions on whether an image sample belongs to a particular class k or not.

Then, we can recursively adopt the consensus operation ⊕ to combine the opinions from all committee members in the crowd-AI fusion committee as follows:

Given the combined opinion EkS from all committee member in the crowd-AI fusion committee S, we can leverage it to derive the accurate class label for each image sample. In particular, we set the class label estimated by our CollabGeneral framework to be the one that has the highest belief value T k Si,k among all possible class labels k for each studied image sample Xi as follows:

where Si,k indicates the set of committee members who estimates the class label for Xi as k.

However, EkSc for each committee member Sc in S is unknown a priori and we need to infer the value for each EkSc before estimating the accurate class label for each image sample. To that end, we further design an iterative learning framework in our SCF module to obtain the accurate value for each EkSc . In particular, we first introduce two important concepts in our iterative learning framework.

Definition 11. Committee Member Reliability (R): We define Rk c to be the probability of a committee member Sc in correctly estimating the class label of an image from class k.

Definition 12. Image Sample Discriminative Score (Z): We define Z k i as the discriminative score of an image sample Xi in terms of identifying the reliable committee member that can correctly estimate the label for image samples of class k.

Given the above two definitions, we note that the values of both committee member reliability R and the image sample discriminative score Z are unknown and depend on each other. Therefore, we optimize R and Z alternately as follows.

First, we optimize the image sample discriminative score Z given the committee member reliability R as follows:

where Si,k is the set of crowd-AI committee members who estimate the class label of Xi to be k. Sp and Sq are any two committee members in Si,k . Rk p and Rk q are the reliability of Sp and Sq, respectively. N k Sp,Sq is the number of image samples where both Sp and Sq estimate the class label to be k. N k Sq is the number of image samples where Sq estimates the class label to be k. In addition, Z k i is set to be 0 if there is only 1 or no committee member label Xi to be k. Intuitively, a high value of Z k i indicates a high likelihood that the estimated class label for Xi is to be k, and vice versa.

Then, we compute the committee member reliability R using the updated image sample discriminative score Z as:

where ∆Sc k indicates the set of all image samples where Sc estimates the class label to be k. Intuitively, a high value of Rk c indicates that the class labels estimated by Sc are more likely to be correct.

Given the above two definitions, we can obtain the optimal value for all Z k i ∗ and Rk c ∗ by iteratively updating all Z k i and Rk c until their values convergence (e.g., the values of Z k i and Rk c remains unchanged between two consecutive iterations). We then leverage Z k i ∗ and Rk c ∗ to derive the optimal opinion entity EkSc ∗ for each committee member as follows:

where Ω(·) is a normalization function to normalize the input between 0 and 1. Z = ∑∀i∈∆Sc k Z k i ∗ indicates the likelihood that Sc is certain about estimated labels for images of class k.

The learned opinion entity EkSc ∗ is then plugged in Equation (7) to derive the accurate class labels for all imagery data in each studied disaster event.

5 Evaluation

5.1 Datasets and Crowdsourcing Settings

Disaster Damage Assessment Datasets: In the experiments, we use four publicly available real-world ADA datasets. 3 The datasets consist of social media images collected from four different disaster events: Hurricane Irma (2017), Ecuador Earthquake (2016), Nepal Earthquake (2015), and Sri Lanka Flooding (2017). Images in each dataset reflect disasterspecific visual characteristics of a disaster (e.g., structure damage vs. flooding damage, urban layout vs. rural layout, plateau landscape vs. coastal landscape). Following the standard practice in ADA applications [Nguyen et al., 2017], we classify the disaster damages into three classes including severe damage, medium damage, and no/minor damage. Each image is annotated by three independent annotators, with the majority voting as the aggregated label. We invited our domain expert to cross-validate the aggregated label to obtain the final ground-truth annotation. A summary of all datasets is presented in Table 1. Additionally, we split the training and testing sets with a ratio of 7:3 and use the training sets to train all compared schemes for ADA tasks and evaluate their performance on the testing sets.

3 https://crisisnlp.qcri.org/

Crowdsourcing Settings: We leverage the widely-used Amazon Mechanical Turk (AMT) to acquire crowd intelligence in our experiments. AMT is one of the largest crowdsourcing platforms offering 24/7 crowdsourcing services from a massive amount of crowd workers around the world. For each task on AMT, we recruit crowd workers who have finished at least 1000 approved tasks with an overall task approval rate of 95% or above to ensure the crowdsourcing label quality. We pay $0.05 per image to the crowd workers and follow the IRB protocol approved for this project.

5.2 Baseline Methods and Experiment Settings

In our evaluation, we compare CollabGeneral with a set of state-of-the-art baselines, including (1) Deep Neural Network (DNN): ResNet [Targ et al., 2016], DenseNet [Huang et al., 2017], and VGG [Li et al., 2018]; (2) AI Model Generalization: GTA [Sankaranarayanan et al., 2018], VS [Kini et al., 2021], SL [Wang et al., 2019b]; (3) Crowd-AI Collaboration: Deep Active [Sener and Savarese, 2018], CrowdLearn [Zhang et al., 2019], SL [Wang et al., 2019b].

In the experiments, to ensure the fairness of comparison, we use the same inputs for all compared methods. In particular, the inputs to each scheme include: 1) the social media images for all studied disaster events in both training and testing datasets; 2) the ground-truth labels for social media images in the training dataset, where the number of training images from each disaster event is proportional to the total number of images from that event as shown in Table 1; and 3) the labeled social media images returned by the crowd workers. In particular, we use the crowd labels to retrain the DNN and AI model generalization baselines to ensure all baselines have the same inputs and the performance of compared baselines is optimized. For the DNN baselines, we consider two different training settings: 1) training a single DNN model for all studied disaster events, which is referred as DNN-A (e.g., ResNet-A for ResNet); 2) training four DNN models, one for each specific disaster event, which is referred as DNN-S (e.g., ResNet-S for ResNet).

We use three evaluation metrics that are commonly used to quantify the performance of multi-class text classification: 1) F1-score, and 2) Matthews Correlation Coefficient (MCC), 3) kappa score (K-Score). We use MCC and K-Score in our evaluation because our datasets are imbalanced, and these two metrics are known to be reliable on imbalanced data [Chicco and Jurman, 2020]. The higher values of the above metrics demonstrate better ADA performance.

5.3 Evaluation Results

Model Generality on Different Types of Disaster Events

Firstly, we study the ADA model generality with a challenging evaluation setting, where the studied disaster events are of completely different types: Ecuador Earthquake and Sri Lanka Flooding. We summarize the evaluation results in Table 2. We observe that CollabGeneral consistently outperforms all compared baselines in terms of the ADA performance on each individual event and the overall performance across two different types of events. For example, the performance gains of CollabGeneral compared to the bestperforming baseline (i.e., VS) on the Sri Lanka Flooding event on F1-Score, MCC, and K-Score are 5.22%, 6.52%, and 8.41%, respectively.

Model Generality on Different Number of Events

Secondly, we evaluate the ADA performance of CollabGeneral when there exist more than two disaster events. In particular, we evaluate CollabGeneral up to four different disaster events by leveraging all possible disaster events available in our datasets. In our experiments, we evaluate the ADA performance by comparing CollabGeneral with the best-performing baselines in each category. The results are presented in Figure 2. Note that we only show the evaluation results on the F1-Score due to the page limit. The evaluation results on other metrics are similar. We observe that our CollabGeneral continuously outperforms all compared baselines on both individual events and overall performance when the number of studied disaster events increases. This is because our subjective logic-based crowd-AI framework design effectively improves the ADA model generality without sacrificing the model's specificity on each studied disaster event.

6 Conclusion

The paper presents a CollabGeneral framework to address the ADA model generality problem. In particular, we design a generality-aware crowd-AI collaborative framework that integrates the complementary AI and crowd intelligence to achieve an optimized trade-off between generality and specificity of ADA models. Our CollabGeneral is shown to significantly improve the ADA model generality by achieving the highest ADA accuracy in each studied disaster event compared to a rich set of deep neural networks, AI model generality, and crowd-AI baselines in four different real-world ADA datasets. We believe CollabGeneral provides useful insights to address the AI model generality problem in many real-world AI-driven applications (e.g., intelligent transportation, smart health, AIoT) for future research in this domain.

Acknowledgments

This research is supported in part by the National Science Foundation under Grant No. IIS-2202481, CHE-2105005, IIS-2008228, CNS-1845639, CNS-1831669. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.