Trustworthy Machine Learning under Imperfect Data

1 Introduction

Machine learning (ML) has demonstrated strong power in numerous real-world applications. In general, it assumes that there are n independent and identically distributed training data {(x1, y1), ..., (xn, yn)} ∼ Ptr(X, Y ), where xi ∈ Rd is the i-th sample, yi ∈ R is the corresponding label, and Ptr denotes the joint probability distribution of the feature X and the label Y . Traditional supervised learning typically assumes that the training distribution can adequately reflect the mapping relationship between test samples and their labels, i.e., Ptr(X, Y ) = Pte(X, Y ), where Pte denotes the joint probability distribution of test data. Therefore, we can model Ptr(X, Y ) based on the principle of risk minimization, which can serve as an estimate for Pte(Y |X).

However, in real-world scenarios, data is not always perfect, which makes it difficult for Ptr(X, Y ) = Pte(X, Y ) to hold true. Specifically, there are mainly three types of imperfect data along with their challenges for ML:

• Noisy labels. That is, the biased label ˜ Y in Ptr(X, ˜ Y ). When the training labels are incorrect, it poses a risk to the application of ML models. Moreover, traditional noise modeling methods are highly hypothetical and cannot cope with label noise in big data.
• Adversarial examples. Namely, the biased feature ˜X in Ptr( ˜X, Y ). In ML, natural examples and adversarial examples often come from different distributions. However, traditional methods mainly assume that samples are identically distributed, which cannot be directly applied to adversarial examples.
• Out-of-distribution data. That is, the biased feature ˜X and the biased label ˜ Y in Pte(X, ˜ Y ) and Pte( ˜X, Y ). In ML, the inconsistent distribution of training and test data can lead to unpredictable and incorrect outputs, which poses a risk to deploying models in practice.

In summary, the aforementioned challenges lead to a significant difference between the training distribution Ptr and the test distribution Pte. This finding makes it impossible to guarantee the performance of ML algorithms designed based on the principle of risk minimization on test data. Therefore, we aim to build trustworthy machine learning (TML) algorithms and models under imperfect data. In this article, we will review the recent advance on TML, and share our contributions on three aspects of TML under imperfect data:

• For TML with noisy labels, we leveraged the memorization effects of deep neural networks to achieve robust training under extremely noisy circumstances. By drawing on human cognition, we incorporated a structure prior of the noise transition matrix, enabling us to accurately estimate the mathematical and statistical information of the label-noise model.
• For TML against adversarial examples, we proposed a causality viewpoint to contribute adversarial learning. By employing the Maximum Mean Discrepancy, we endowed the robust model with detection capabilities. Furthermore, we achieved an expansion of robust training under adversarial examples through gradually informative attacks from weak to strong.
• For TML under out-of-distribution data, we enhanced the model's ability to detect out-of-distribution data by utilizing its reprogrammable features and intrinsic learning capability. Through an informative extrapolation strategy, we made full use of limited informative auxiliary outliers, effectively avoiding the high costs associated with model retraining.

We will also discuss some real-world challenges that deserve more attention.

2 TML with Noisy Labels

Learning with noisy labels pertains to supervision scenarios in which some of the data are incorrectly labeled, and the corresponding goal of trustworthiness is to enhance the robustness of models against such noise. Trustworthy machine learning is a critical area of research, underscored by numerous studies that have shown the prevalence of noisy labels in real-world situations [Xiao et al., 2015; Li et al., 2017; Wei et al., 2022] and their notable adverse effects on model performance [Liu and Tao, 2015; Arpit et al., 2017]. Here, we advance the research on TML with noisy labels from the perspectives of data, regularization, and optimization.

Data perspective. The properties of noisy labels within training datasets are typically characterized by the noise transition matrix [Van Rooyen and Williamson, 2018]. Generally, the transition matrix can be written as follows:

which denotes the probability that the noisy label ˜ Y will be j for the feature X with the true label i. Early works have suggested a series of theoretical frameworks to correct the biases raised by label noise [Liu and Tao, 2015; Patrini et al., 2017; Goldberger and Ben-Reuven, 2022]. However, the noise transition matrix is often unknown a priori, and its inaccurate estimation can degrade the performance of the resulting models.

We discerned between class pairs that are more prone to confusion through the lens of human cognition, suggesting an effective strategy for transition matrix estimation named "Masking" [Han et al., 2018a]. This method involves manually constraining the matrix structure by masking out transitions between categories with potentially low likelihood and focusing on estimating the more probable ones, thereby reducing the parameterized searching space of noise estimation. To integrate this structural prior during learning with noisy labels, we suggested a probabilistic model that involves the true label Y ∼ P(Y |X), the noise transition matrix T ∼ P(T), and the matrix structure variable T0 ∼ P(T0), and the noisy label ˜ Y ∼ P(˜ Y |Y, T). From this generative process, we can derive the evidence lower bound to approximate the loglikelihood of the noisy data. Therein, the inclusion of the structural prior T0 helps manage unknown noisy data and addresses the limitations of end-to-end matrix estimations, which are susceptible to setting at local minima.

Regularization perspective. Researchers have explored various regularization techniques during training to alleviate overfitting to noisy labels. Trustworthy learning methods incorporating regularization involve constraints on parameters norms and smoothness [Azadi et al., 2016; Foret et al., 2020; Berthelot et al., 2019], constraints on entropy [Berthelot et al., 2019; Reed et al., 2015], data augmentation [Zhang et al., 2018], among many others. However, we argue that these conventional regularization methods do not discern between noisy and clean data. While they are effective in penalizing the model overfitting to noisy data, they also inadvertently hinder the model ability to learn from clean data.

To this end, we proposed a novel regularization strategy that treats data in an adaptive manner, named stochastic integrated gradient underweighted ascent "SIGUA" [Han et al., 2020]. Therein, we distinctly handle potential clean and noisy data, where we perform conventional gradient descent for the former, yet for the latter, we apply gradient ascent. Overall, SIGUA is designed to pull back the learned knowledge for potentially noisy data, intending to mitigate the conflicting model memorization raised by them. Unlike previous regularization strategies like weight decay, which apply uniformly, SIGUA selectively reduces optimization on noisy data while reinforcing the useful knowledge characterized by clean data.

Optimization perspective. Trustworthy methods dealing with noisy labels typically explored the memorization effects of deep models [Arpit et al., 2017]. It refers to a general observation where models tend to initially learn from correct data and memorize incorrect data later. This phenomenon suggests that learning models possess an inherent ability, to some extent, to distinguish correct from wrong data, motivating a series of self-training techniques such as small-loss selection [Ren et al., 2018; Jiang et al., 2018] and self relabeling [Wang et al., 2021; Yi and Wu, 2019]. However, by employing memorization effects, models may still make mistakes in correcting label noise and such errors will accumulate during existing learning procedures, limiting the robustness of trained models.

Due to the stochasticity of deep model optimization, different models can yield varying predictions on the same data, especially for noisy ones. Consequently, we proposed a "Coteaching" training paradigm [Han et al., 2018b], where two pseudo-Siamese models are encouraged to exchange smallloss samples recognized by each other. It means that each model should update its gradients based on the filtered data selected by its counterpart. This collaborative process facilitates mutual learning to mitigate the previous deficiencies of the memorization effects in error accumulation, notably enhancing the robustness of models to noisy labels.

3 TML against Adversarial Examples

Adversarial examples are important in the field of trustworthy machine learning [Goodfellow et al., 2015; Szegedy et al., 2014], as they highlight the vulnerability of machine learning models to carefully crafted inputs that can cause erroneous predictions [Carlini and Wagner, 2017; Croce and Hein, 2020]. They are often imperceptible to human recognition but can potentially undermine the reliability of machine learning systems in critical applications. Therefore, it necessitates the need for robust learning methods [Madry et al., 2018; Cohen et al., 2019; Zhang et al., 2019; Zhang et al., 2021a; Chen et al., 2022; Zhou et al., 2023]. To deepen the understanding of the adversarial vulnerability, we constructed the first causal formulation, which inspires an adversarial distribution alignment method. To the challenge of detecting adversarial samples, we proposed a novel semantic-aware deep kernel to enhance the power of the maximum mean discrepancy test. To adapt the adversarial training to the weak supervision setting, we designed a novel training framework for complementary labels that are more accessible.

Causal formulation on adversarial attacks. To identify the origin of vulnerability, we constructed the first theoretical formulation on the data-generating process of natural and adversarial samples [Zhang et al., 2021b]. The natural sample X is decomposed into the context-related cause C, which solely determines the label Y , and context-independent style cause S with a structure Y ← C → X ← S. The adversarial sample ˜X is then formulated to be a composition of the natural sample X and a special perturbation noise E depending on natural sample X and target Y . With the insight from the above causal formulation, the key is eliminating the spurious correlation path S → X ← C → Y opened by the observation (e.g., the given variable X) of X. We therefore proposed an adversarial distribution alignment objective:

Here, d measures the alignment and the second term is the penalty for the misalignment averaged on different styles.

Statistical tests on adversarial samples. In principle, samples from the natural and adversarial distribution could be distinguished. The maximum mean discrepancy (MMD) [Borgwardt et al., 2006] is a measurement of the closeness between two distributions P and Q. Let k be a bounded kernel with the reproducing kernel Hilbert space (RKHS) Hk, and let F be a unit ball in that space, then the kernel-based MMD MMD(P,Q,Hk) is defined as

To mitigate the empirical failure of the MMD test on adversarial samples, we developed a more effective method, called Sematic-Aware MMD "SAMMD" [Gao et al., 2021], with a semantic-aware deep kernel kω(x, y) defined as

where sf(x,y) = κ(ϕp(x), ϕp(y)) is a deep kernel function for the similarity between x and y; Here f is a deep model; we take ϕp as its second and last layer. κ is the Gaussian kernel with σϕp bandwidth. ϵ0 ∈ (0, 1) and q(x, y) (the Gaussian kernel with σq bandwidth) are key components to ensure kω is a characteristic kernel. The parameters of the deep kernel are optimized to maximize the approximate test power of SAMMD. Equipped with a well-optimized semantic-aware deep kernel, as well as using the wild bootstrap to tackle possible dependency among samples, the natural and adversarial samples can be distinguished.

Adversarial training with complementary labels. The complementary labels (CLs) indicate a class to which a sample does not belong. It is featured by its low acquisition cost and privacy preservability. We presented the first work to identify the issues of adversarial training in the setting of complementary labels [Zhou et al., 2022]. In fact, directly applying such labels to adversarial training leads to a consistent empirical failure. Theoretically, with a limited number of CLs, the adversarial training on the complementary risk is no longer guaranteed to be statistically consistent with that with the ordinary risk. Empirically, the training dynamics suffer from the gradient vanishing, and the quality of adversarial samples suffers from the less information nature of CLs.

To mitigate the above challenges, we proposed a new learning strategy consisting of two key components: Warm-up Attack and Pseudo-Label Attack. The former component gradually enhances the degree of adversarial generation and controls the variance introduced into the optimization. The later component utilizes the simple pattern prevalent in the adversarial date at the early optimization stage to predict ordinary labels and then construct high-quality adversarial samples.

4 TML under Out-of-Distribution Data

In open-world scenarios, models often encounter out-ofdistribution (OOD) samples during deployment, i.e., the samples fall outside the training data label space, known as indistribution (ID) data. We aspire for the model to classify ID data while effectively detecting OOD data. This capability is crucial to mitigate the critical risks associated with misclassifying OOD data [Fang et al., 2022; Sehwag et al., 2021; Yang et al., 2022], especially in high-risk scenarios such as autonomous driving. For TML under OOD data, we enhanced the model's ability to detect OOD data from two perspectives: the model and data. Furthermore, we explored performing OOD detection with foundation models like CLIP.

Model perspective. Most existing OOD detection methods design scoring functions to detect OOD samples, which can be considered a means of harnessing the model's intrinsic capabilities. Specifically, the scoring function can be formulated using logit outputs [Hendrycks and Gimpel, 2017], gradients [Huang et al., 2021], and embedding features [Lee et al., 2018]. By utilizing the trained model and test data, input samples with lower scores are identified as OOD samples, whereas those with higher scores are classified as ID samples.

Previous works largely ignored deep models' reprogramming property [Elsayed et al., 2019], which may help detect OOD samples. The reprogramming property of deep models refers to repurposing a well-trained model for a new task without modifying its parameters by transforming the original input, such as through universal feature perturbations. We proposed "watermarking" [Wang et al., 2022] to reprogram a well-trained model by adding a watermark to the original inputs, thereby enhancing the model's ability to detect OOD samples. Specifically, the watermark shares the same shape as the inputs and is a static pattern that can be added to inputs during testing. We proposed a learning framework for effective watermarking by superimposing a unified pattern onto input features. Our watermarking complements and can be integrated with existing methods.

Data perspective. Someworks [Hendrycks et al., 2019; Du et al., 2022] have been proposed from the data perspective to enhance OOD detection. Outlier Exposure (OE) [Hendrycks et al., 2019] uses an auxiliary outlier dataset to train OOD detectors for detecting unseen OOD samples, demonstrating promising performance. Different from OE, Virtual Outlier Synthesis (VOS) [Du et al., 2022] synthesizes virtual outliers from the low-likelihood region of the class-conditional distribution estimated in the feature space to regularize the model's decision boundary rather than using real outlier datasets.

Given that OOD samples encountered during model deployment are unpredictable, OE necessitates an auxiliary outlier dataset that is sufficiently large and representative to encompass various OOD categories, which is impractical and challenging. What's more, relying solely on the generated virtual outliers is less effective than utilizing real collected outliers. We thus proposed an informative extrapolation strategy based on the finite auxiliary outliers given for effective OOD detection, termed Diversified Outlier Exposure (DivOE) [Zhu et al., 2023]. Technically, we introduced a new learning objective to diversify the auxiliary distribution during training. This learning objective can learn from Ds out and generalize to the different OOD distributions via the maximization part. Extensive experiments show that our strategy is effective and compatible with other OE methods.

OOD detection with FMs. Since the pre-trained visual language models (VLMs) learn robust feature representations, OOD detection can be performed in a zero-shot manner [Ming et al., 2022; Esmaeilpour et al., 2022]. Here ID classes are defined with respect to the classification task of interest, rather than the classes used during pre-training. Maximum Concept Matching (MCM) [Ming et al., 2022] constructs a textual classifier by calculating the cosine similarity between image and text features, then uses maximum softmax probability to detect OOD data.

However, MCM relies solely on ID class names while neglecting VLMs' text interpretation capabilities. We introduced "NegLabel" [Jiang et al., 2024] to harness the knowledge embedded in VLMs by leveraging a vast array of negative labels for OOD detection effectively. Technically, we designed a NegMining algorithm to select high-quality negative labels from extensive corpus databases. We developed a novel scoring function that integrates knowledge from ID and negative class names. This scoring function is positively correlated with the affinities between images and ID labels, and negatively correlated with those of the negative labels. To better exploit VLMs' inherent capability to identify samples in open-world settings, we proposed EOE [Cao et al., 2024] by employing the expert knowledge of large language models (LLMs) to envision potential outlier exposure. We designed LLM prompts based on the visual similarity rule to generate potential outlier class labels, and experiments show that our EOE outperforms existing methods.

5 Challenges and Opportunities

Although significant progress has been made in TML under three types of imperfect data with traditional approaches, several challenges and opportunities remain.

Trustworthy foundation models. Foundation Models (FMs) have shown impressive ability in real-life scenarios, including general problem-solving and multi-modal data processing. However, the concerns regarding data safety under open-world settings pose great challenges to FMs:

• Adversarial prompts of FMs. The success of FMs can be credited to the large amounts of corpus. As
• such, harmful and offensive content will inevitably be included. Additionally, preventing FMs from generating harmful information is still challenging, e.g., jailbreak attack [Zou et al., 2023; Wei et al., 2023; Li et al., 2023].
• Multi-modal noise of FMs. Compared to previous methods, FMs are able to process multi-modal data, such as images and texts. Due to the various kinds of data sources and types, noise and semantic inconsistency inevitably exist in the multi-modal data.
• Variations of data distribution in foundation model deployment. Although the data for FM pre-training have considered multiple potential distributions, FMs tend to obtain inconsistent performances across different scenarios, hindering the model's reliability in the openworld setting.

Trustworthy federated learning. Federated learning is a typical paradigm for privacy protection (e.g. medical health and social networks) by updating the central model with the client models trained on local data. The central model can not observe the data throughout the process. However, federated learning is facing with the following challenges:

• Unreliability of data. The assumption that the labels of the available data are complete and clear is not appropriate for practical applications. This poses a challenge to existing methods and we have to consider extra techniques to infer the labels of data for better performance.
• Competition of clients. Current methods also implicitly assume that there is no malicious competition among clients. However, clients may be trained with malicious samples (e.g. poisoning samples), which may further hinder the performance of the central model.
• Heterogeneity of clients. Another assumption is that all clients are equally important. However, current model aggregation strategies tend to ignore data heterogeneity, which may further result in information decay. Thus, adaptive strategies are required to avoid such a case.

Trustworthy causal learning. Foundation Models can effectively handle complex wild data in reality; on the other hand, causality offers a rigorous way to investigate datagenerating processes behind data. However, it is still an open problem how causality and foundation models can reliably benefit each other. Here are some potential frontiers:

• FMs for causal identification. Leveraging the capabilities of FMs to enhance causality in representation abstraction [Liu et al., 2024], structure identification, treatment effect estimation, and counterfactual predictions.
• Mechanistic understanding of FMs. Utilizing rigorous methods like causal abstraction and identification to investigate the inner mechanisms of FMs, including the reasoning ability [Prystawski et al., 2024], in-context learning ability, and trustworthy properties like explainability, fairness, and safety.
• Counterfactual generation. Counterfactual generation gives things about What If questions that are unverifiable. Causal methods can potentially make the generated narratives or images more controllable and reliable.

Acknowledgments

This work was partially supported by NSFC (62376235, 62006202), RGC (22200720), GDST (2024A1515012399, 2022A1515011652), RIKEN AIP and HKBU CSD. The author would like to thank TMLR Group members for improving the paper.