Trusted Multi-view Learning with Label Noise

1 Introduction

Multi-view data is widely present in various real-world scenarios. For instance, in the field of healthcare, a patient's comprehensive condition can be reflected through multiple types of examinations; social media applications often include multi-modal contents such as textual and visual reviews [Liu et al., 2024]. Multi-view learning synthesizes both consistency and complementary information to obtain a more comprehensive understanding of the data. It has generated significant and wide-ranging influence across multiple research areas, including classification [Chen et al., 2024; Wang , 2022 ], clustering [Xu et al., 2023; Huang et al., 2023; Wen et al., 2022], recommendation systems [Lin et al., 2023; Nikzad-Khasmakhi et al., 2021], and large language models [Min et al., 2023].

∗Corresponding author

Most existing multi-view learning methods focus on improving decision accuracy while neglecting the decision uncertainty. This significantly limits the application of multiview learning in safety-critical scenes, such as healthcare. Recently, Han et al. propose a pioneering work [Han et al., 2020], Trusted Multi-view Classification (TMC), to solve this problem. TMC calculates and aggregates the evidences of all views from the original data features. It then utilizes these evidences to parameterize the class distribution, which could be used to estimate the class probabilities and uncertainty. To train the entire model, TMC requires the estimated class probabilities to be consistent with the ground-truth labels. Following this line, researchers propose novel evidence aggregation methods, aiming to enhance the reliability and robustness in the presence of feature noise [Gan et al., 2021; Qin et al., 2022], conflictive views [Xu et al., 2024] and incomplete views [Xie et al., 2023].

Regretfully, these trusted multi-view learning methods consistently rely on high-quality ground-truth labels. The labeling task is time-consuming and expensive especially when dealing with large scale datasets, such as user generated multi-modal contents in social media applications. This motivates us to delve into a new Generalized Trusted Multi-view Learning (GTML) problem: how to develop a reliable multiview learning model under the guidance of noisy labels? This problem encompasses two key objectives: 1) detecting and refining the noisy labels during the training stage; 2) recognizing the model's uncertainty caused by noisy labels. For example, instances belonging to classes like "dog" and "wolf" might exhibit similarities and are prone to being mislabelled. Consequently, the model should exhibit higher decision uncertainty in such cases. An intuitive analogy is an intern animal researcher (model) may not make high-confidence decisions for all animals (instances), but is aware of the cases where a definitive decision is challenging.

In this paper, we propose an Trusted Multi-view Noise Refining (TMNR) method for the GTML problem. We consider the label noises arising from two sources: low-quality data features, such as blurred and incomplete features, and easily-confused classes, such as classes "dog" and "wolf". Our objective is to leverage multi-view consistent information for noise detection. To achieve this, we first construct the view-specific evidential Deep Neural Networks (DNNs) to learn view-specific evidence, which could be termed as the amount of support to each category collected from data. We then model the view-specific distributions of class probabilities using the Dirichlet distribution, parameterized with view-specific evidence. These distributions allow us to construct opinions, which consist of belief mass vectors and uncertainty estimates. We design view-specific noise correlation matrices to transform the original opinions to the noisy opinions, which aligns with the noisy labels. Considering that low-quality data features are prone to mislabeling, we require the diagonal elements of the noise correlation matrices to be inversely proportional to the uncertainty. Additionally, we incorporate class relations into the off-diagonal elements. For instance, the elements corresponding to "dog" and "wolf" should have larger values since these two classes are easily mislabelled. Next, we aggregate the noisy opinions to obtain the common evidence. Finally, we employ a generalized maximum likelihood loss on the common evidence, guided by the noisy labels, for model training.

The main contributions of this work are summarized as follows: 1) we propose the generalized trusted multi-view learning problem, which necessitates the model's ability to make reliable decisions despite the presence of noisy guidance; 2) we propose the TMNR method to tackle this problem. TMNR mitigates the negative impact of noisy labels through two key strategies: leveraging multi-view consistent information for detecting and refining noisy labels, and assigning higher decision uncertainty to instances belonging to easily mislabelled classes; 3) we empirically compare TMNR with stateof-the-art trusted multi-view learning and label noise learning baselines on 5 publicly available datasets. Experiment results show that TMNR outperforms baseline methods on accuracy, reliability and robustness.

2 RelatedWork

2.1 DeepMulti-view Fusion

Multi-view fusion has demonstrated its superior performance in various tasks by effectively combining information from multiple sources or modalities [Liang et al., 2021; Zhou et al., 2023]. According to the fusion strategy, existing deep multi-view fusion methods can be roughly classified into feature fusion [Hu et al., 2024; Xu et al., 2022; Liu et al., 2023] and decision fusion [Jillani et al., 2020; Liu et al., 2022]. A major challenge of feature fusion methods is each view might exhibit different types and different levels of noise at different points in time. Trust decision fusion methods solve this making view-specific trust decisions to obtain the view-specific reliabilities, then assigning large weights to these views with high reliability in the multi-view fusion stage. Following this line, Xie et al. [Xie et al., 2023] tackle the challenge of incomplete multi-view classification through a two-stage approach, involving completion and evidential fusion. Xu et al. [Xu et al., 2024] focus on making trust decisions for instances that exhibit conflicting information across multiple views. They propose an effective strategy for aggregating conflicting opinions and theoretically prove this strategy can exactly model the relation of multi-view common and view-specific reliabilities. However, it should be noted that these trusted multi-view learning methods heavily rely on high-quality ground-truth labels, which may not always be available or reliable in real-world scenarios. This limitation motivates for us to delve into the problem of GTML, which aims to learn a reliable multi-view learning model under the guidance of noisy labels.

2.2 Label-Noise Learning

In real-world scenarios, the process of labeling data can be error-prone, subjective, or expensive, leading to noisy labels. Label-noise learning refers to the problem of learning from training data that contains noisy labels. In multi-classification tasks, label-noise can be categorized as Class-Conditional Noise (CCN) and Instance-Dependent Noise (IDN). CCN occurs when the label corruption process is independent of the data features, and instances in a class are assigned to other classes with a fixed probability. Dealing with CCN noise often involves correcting losses by estimating an overall category transfer probability matrix [Patrini et al., 2017; Hendrycks et al., 2018]. IDN refers to instances being mislabeled based on their class and features. In this work, we focus on IDN as it closely resembles real-world noise. The main challenge lies in approximating the complex and highdimensional instance-dependent transfer matrix. Several approaches have been proposed to address this challenge. For instance, Cheng et al. [Cheng et al., 2020] proposes an instance-dependent sample sieve method that enables model to process clean and corrupted samples individually. Cheng et al. [Cheng et al., 2022] effectively reduce the complexity of the instance-dependent matrix by streaming embedding. Berthon et al. [Berthon et al., 2021] approximate the transfer distributions of each instance using confidence scores. However, the confidence depends on the pre-trained model and may not reliable. The proposed TMNR reliably bootstraps the correlation matrix based on multi-view opinions, leading to superior performance. In addition, TMNR not only refines the noise in the labels but also recognizes the model's uncertainty caused by noisy labels.

3 The Method

In this section, we first define the generalized trusted multiview learning problem, then present Trusted Multi-view Noise Refining (TMNR) in detail, together with its implementation.

3.1 Notations and Problem Statement

We use {xvn ∈ Rdv }Vv=1 to denote the feature of the n-th instance, which contains V views, dv denotes the dimension of the v-th view. yn ∈ {1, ...,K} denotes the ground-truth category, where K is the number of all categories. In the generalized trusted multi-view classification problem, the labels of some data instances contain noise as shown in Figure 2. Therefore, we utilize {˜ yn ∈ {1, ...,K}}Nn=1 as the set of noisy labels that may have been corrupted

The objective is to learn a trusted classification model according to noisy training instances {{xvn }Vv=1 , ˜ yn}Ntrain n=1 . For the instances of the test sets, the model should predict the category {yn} and uncertainty {un}, which can quantify the uncertainty caused by low-quality feature and noisy labels.

3.2 TrustedMulti-viewNoise Refining Pipeline

As shown in Figure 3, We first construct view-specific opinions using evidential DNNs {f v (·)}Vv=1 . To account for the presence of label noise, the view-specific noise correlation matrices {T v }Vv=1 transform the original opinions into noisy opinions aligned with the noisy labels. Finally, we aggregates the noisy opinions and trains the whole model by the noisy labels. Details regarding each component will be elaborated as below.

View-specific Evidence Learning

In this subsection, we introduce the evidence theory to quantify uncertainty. Traditional multi-classification neural networks usually use a Softmax activation function to obtain the probability distribution of the categories. However, this provides only a single-point estimate of the predictive distribution, which can lead to overconfident results even if the predictions are incorrect. This limitation affects the reliability of the results. To address this problem, EDL [Sensoy et al., 2018] introduces the evidential framework of subjective logic [Jøsang, 2016]. It converts traditional DNNs into evidential neural networks by making a small change in the activation function to get a non-negative output (e.g.,ReLU) to extract the amount of support (called evidence) for each class.

In this framework, the parameter α of the Dirichlet distribution Dir(p|α) is associated with the belief distribution in the framework of evidence theory, where p is a simplex representing the probability of class assignment. We collect evidence, {e vn } by view-specific evidential DNNs {f v (·)}Vv=1 . The corresponding Dirichlet distribution parameter is αv = ev + 1 = [αv 1 , · · · , αvK ]T . After obtaining the distribution parameter, we can calculate the subjective opinion Ov = (bv , uv ) of the view including the quality of beliefs bv and the quality of uncertainty u, where bv = (αv -1)/Sv = ev /Sv , uv = K/Sv , and Sv = ∑Kk=1 αvk is the Dirichlet intensity.

Evidential Noise Forward Correction

In the GTML problem, we expect to train the evidence network so that its output is a clean evidence distribution about the input. To minimise the negative impact of IDN in the training dataset, we modify the outputs of the DNNs with an additional structure to adjust the loss of each training sample before updating the parameters of the DNNs. This makes the optimisation process immune to label noise, called evidential noise forward correction. This structure should be removed when predicting test data.

For each view {xv }Vv=1 of a specific instance, we construct view-specific noise correlation matrix to model the noise process:

To predict probability distributions, the noise class posterior probability is obtained by calculating the correlation matrix:

Based on the evidence theory described in the previous subsection and considering the constraints within T v itself, we convert the transfer of predicted probabilities of instances into a transfer of the extracted support at the evidence level using the following equation:

where ˜ evj /evk denotes the j-/k-th element in the noise/clean class-posterior evidence quantities ˜ ev /ev .

Therefore, the clean posterior ev obtained from the prediction of each view is transferred to the noisy posterior ˜ ev . The parameter ˜α in the noisy Dirichlet distribution is computed in order to align the supervised labels ˜ y that contain the noise. The entire evidence vector could be calculabed by:

Trust Evidential Multi-view Fusion

After obtaining opinions from multiple views, we consider dynamically integrating them based on uncertainty to produce a combined opinion. We achieve this via the Dempster's combination rule [Jøsang, 2016]. We take the clean opinion aggregation as example. Given clean opinions of two views of the same instance, i.e., O1 = (b1 , u1 ) and O2 = (b2 , u2 ), the computation to obtain the aggregated opinion O = O1 ⋄ O2 = (b, u) is defined as follows:

For a set of opinions from multiple views {Ov }Vv=1 , the joint multi-view subjective opinion is obtained by O = O1 ⋄ O2 ⋄ · · · OV . The corresponding parameters of the Dir(p|α) are obtained with S = K/u, αk = bk × S + 1 and the final probability could be estimated via pk = αk/S.

3.3 Loss Function

In this section, we explore the optimization of the parameter set {θ, ω} in the evidence extraction network f(·; θ) and the correlation matrices {{T v n }Vv=1 }Ntrain n=1 .

Classification Loss

We capture view-specific evidence from a single view xv of a sample. The vector ev = f v (xv ) denotes the clean classposterior evidence obtained from the corresponding view network prediction. This evidence undergoes correction through Eq. (4) to yield the noisy class-posterior evidence, denoted as ˜ ev , along with the associated Dirichlet parameter ˜αv . Then in the constructed inference framework, the evidence multiclassification loss L containing the classification loss Lace and the Kullback-Leibler (KL) divergence term LKL is defined, where the classification loss Lace is obtained by adjusting for the conventional cross-entropy loss, i.e., the generalized maximum likelihood loss, as follows:

where ψ(·) is the digamma function, ˜αv j denote the j-th element in ˜αv .

Eq. (6) does not ensure that the wrong classes in each sample produce lower evidence, and we would like it to be reduced to zero. Thus the KL divergence term is expressed as:

where ¯αv = ˜y + (1 - ˜y) ⊙ ˜αv is the Dirichlet parameter adjusted to remove non-misleading evidence. ˜y denotes the noisy label ˜ y in the form of a one-hot vector, and Γ(·) is the gamma function. Thus for a given view's Dirichlet parameter ˜αv , the view-specific loss is

where λ ∈ [0, 1] is a changeable parameter, and we gradually increase its value during training to avoid premature convergence of misclassified instances to a uniform distribution.

Uncertainty-Guided Correlation Loss

Optimizing the view-specific correlation matrix, being a function of the high-dimensional input space, presents challenges without any underlying assumptions. In the context of the IDN problem, it is important to recognize that the probability of a sample being mislabeled depends not only on its category but also on its features. When the features contain noise or are difficult to discern, the likelihood of mislabeling increases significantly. As highlighted earlier, the uncertainty provided by the evidence theory has proven effective in assessing the quality of sample features. Therefore, it is natural for us to combine the uncertainty estimation with the IDN problem, leveraging its potential to enhance the overall performance.

In our work, we do not directly reduce the complexity of correlation matrix by simplifying it. Instead, we propose an assumption that "the higher the uncertainty of the model on the decision, the higher the probability that the sample label is noisy". Based on this assumption, a mild constraint is imposed on the correlation matrix to effectively reduce the degrees of freedom of its linear system. Specifically, based on the obtained Dirichlet parameters ˜αv with its corresponding opinion uncertainty uv . We impose different constraints on various parts of the correlation matrix T v , aiming to encourage it to transfer evidence for instances with higher uncertainty and uncover potential labeling-related patterns.

Diagonal elements. Since the diagonal element {tvkk }Kk=1 in the T v corresponds to the probability that the labelled category is equal to its true category. Meanwhile the confidence we obtain from subjective opinions is only relevant for the diagonal elements corresponding to their labelled category ˜ y, for {tvkk }Kk=1,k̸=˜ y , uv no longer provides any direct information. Therefore, we simply make the other diagonal elements close to the confidence mean of the corresponding class of samples from the current batch. It can be expressed as:

where uv = K/ ∑Kk=1 ˜αvk , ¯uvk is the average of the uv of all samples with label ˜ y = i in the current batch.

Non-diagonal elements. The constraints on the diagonal elements could be regarded as guiding the probability of the sample being mislabeled. The probability of being mislabeled as another category is influenced by the inherent relationship between the different categories. For example, "dog" is more likely to be labeled as "wolf" than as "plane". Considering that samples in the same class can be easily labeled as the same error class, the transfer probabilities of their nondiagonal elements should be close. In addition, since the labeled information may contain noise, we aim to eliminate the misleading of the error samples in the same class. To solve this problem, we construct the affinity matrix {Sv }Vv=1 for each viewpoint and calculate this loss with only the k most similar samples in the same class, and for the n-th sample:

Algorithm 1 TMNR algorithm

/*Training*/

Input: Noisy training dataset, hyperparameter β, γ

Output: Parameters of model

• 1: Initialize the parameters of the evidence neural network.
• 2: Initialise all correlation matrices T as unit matrices.
• 3: while not converged do
• 4: for v = 1 : V do
• 5: Obtain clean evidence evn with f v (xvn ; θ);
• 6: Obtain ˜ evn and ˜αvn through Eq. (4);
• 7: end for
• 8: Aggregation to obtain ˜αn by Eq. (5);
• 9: Calculate overall loss with Eq. (14);
• 10: Update the parameters;
• 11: Correct T to satisfy Eq. (1).
• 12: end while

/*Test*/

Calculate the clean joint Dir(p|α) and the corresponding uncertainty u by f (·; θ)

where svnm denotes the (n,m)-th element in the affinity matrix Sv for the v-th view, which measures the similarity between xvn and xvm. N(xvn , k) indicates the k-nearest neighbours of the view xvn . ¯ T v i is the matrix after zeroing the diagonal elements of T v i . Thus, the overall regularization term for the inter-sample uncertainty bootstrap is expressed as:

Inter-view consistency. In multi-view learning, each view represents different dimensional features of the same instance. In our approach, we leverage the consistency principle of these views to ensure the overall coherence of the correlation matrix across all views. The consistency loss is denoted as:

Overall Loss

To sum up, for a multi-view instance {xv }Vv=1 , to ensure that view-speicfic and aggregated opinions receive supervised guidance. We use a multitasking strategy and bootstrap the correlation matrix according to the designed regularization term:

where β and γ are hyperparameters that balances the adjusted cross-entropy loss with the uncertainty bootstrap regularisation and the inter-view consistency loss. ˜α is obtained by aggregating multiple noise class-posterior parameters { ˜αv }Vv=1 .

The overall procedure is summarized in Algorithm 1.

4 Experiments

In this section, we test the effectiveness of the proposed method on 5 real-world multi-view datasets with different proportions of labelling noise added. In addition, we also verify the ability of the model to handle low-quality features and noisy labels.

4.1 Experimental Setup

Datasets. UCI 1 contains features for handwritten numerals ('0'-'9'). The average of pixels in 240 windows, 47 Zernike moments, and 6 morphological features are used as 3 views. PIE 2 consists of 680 face images from 68 experimenters. We extracted 3 views from it: intensity, LBP and Gabor. BBC 3 includes 685 documents from BBC News that can be categorised into 5 categories and are depicted by 4 views. Caltech101 4 contains 8677 images from 101 categories, extracting features as different views with 6 different methods: Gabor, Wavelet Moments, CENTRIST, HOG, GIST, and LBP. we chose the first 20 categories. Leaves100 5 consists of 1600 leaf samples from 100 plant species. We extracted shape descriptors, fine-scale edges, and texture histograms as 3 views.

Compared methods. (1) Sing-view uncertainty aware methods contain MCDO [Gal and Ghahramani, 2016] measuring uncertainty by using dropout sampling in both training and inference phases. IEDL [Deng et al., 2023] is the SOTA method that involving evidential deep learning and Fisher's information matrix. (2) Label noise refining methods contain: FC [Patrini et al., 2017] corrects the loss function by a CCN transition matrix. ILFC [Berthon et al., 2021] explored IDN transition matrix by training a naive model on a subset. (3) Multi-view feature fusion methods contain: DCCAE [Wang et al., 2015] train the autoencoder to obtain a common representation between the two views. DCP [Lin et al., 2022] is the SOTA method that obtain a consistent representation through dual contrastive loss and dual prediction loss. (4) Multi-view decision fusion methods contain: ETMC [Han et al., 2022] estimates uncertainty based on EDL and dynamically fuses the views accordingly to obtain reliable results. ECML [Xu et al., 2024] is the SOTA method that propose a new opinion aggregation strategy. We summarize baseline methods in Table 1. For the single-view baselines, we concatenate feature vectors of different views.

Implementation details. We implement all methods on PyTorch 1.13 framework. In our model, the view-specific evidence extracted by fully connected networks with a ReLU layer. The correlation matrices are initially set as unit matrix. We utilize the Adam optimizer with a learning rate of 1e-3 and l2-norm regularization set to 1e-5 . In all datasets, 20% of the instances are split as the test set. We run 5 times for each method to report the mean values and standard deviations. We follow [Cheng et al., 2020] to generate the instancedependent label noise training sets.

1 http://archive.ics.uci.edu/dataset/72/multiple+features

2 https://www.cs.cmu.edu/afs/cs/project/PIE/MultiPie/Multi-

Pie/Home.html

3 http://mlg.ucd.ie/datasets/segment.html

4 https://github.com/yeqinglee/mvdata

5 https://archive.ics.uci.edu/dataset/241/one+hundred+plant+ species+leaves+data+set

4.2 Experimental Results

Performance comparison. The comparison between TMNR and baselines on clean and noisy datasets are shown in Table 2. We can observe the following points: (1) On the clean training dataset, TMNR achieves performance comparable to state-of-the-art methods. This finding indicates that the noise forward correction module has minimal negtive impact on the model's performance. (2) The performance of multi-view feature fusion methods degrade clearly with the noise ratio increase. The reason is the feature fusion would badly affected by noisy labels. (3) On the noisy training dataset, especially with high noise ratio, TMNR significantly outperforms all baseline. Such performance is a powerful evidence that our proposed method effectively reduces the effect of noisy labels through forward correction. We would further verify this in ablation study and uncertainty evaluation experiments.

Model uncertainty evaluation. In real-world datasets, various categories have varying probabilities of being labeled incorrectly. If we can identify the classes that are more likely to be labeled incorrectly during the labeling process, we can apply specialized processing to address these classes, such as involving experts in secondary labeling. As incorrect labeling leads to increased model uncertainty, our model can effectively identify classes that contain noise by assessing their predicted uncertainty.

To observe significant results, we intentionally flipped the labels of samples belonging to classes '0' and '1', as well as classes '8' and '9', within the UCI dataset during training. Subsequently, predictions were made on the test samples, and the average uncertainty for each category was calculated. The results, depicted in Figure 4(a), demonstrate a notable increase in uncertainty for the categories where the labels were corrupted. Figure 4(b) presents a heat map displaying the mean values of all trained correlation matrix parameters. The results clearly illustrate that the model's structure captures the probability of changes in inter-class evidence.

Correlation matrix evaluation. We analyze the sensitivity of hyperparameter β on all datasets containing 30% noise. The results is shown in Figure 5. It is evident that the sensitivity of the parameter β varies across different datasets, yet optimal performance is consistently achieved within the range of 0.05 to 0.1. This observation validates the effectiveness of the regularization applied to the diagonal elements of the correlation matrices. Taking this into consideration, we have determined the appropriate value of β for the remaining evaluation experiments.

5 Conclusion

In this paper, we introduced a TMNR method for addressing the generalized trusted multi-view learning problem. TMNR

Figure 5: Correlation matrix evaluation. Classification accuracy when adjusting β on all datasets with 30% noise rate.

leverages evidential deep neural networks to learn viewspecific belief mass vectors and uncertainty estimates. We further designed view-specific noise correlation matrices to effectively correlate the original opinions with the noisy opinions. By aggregating the noisy opinions and training the entire model using the noisy labels, we achieved robust model training. Experimental results on five real-world datasets validated the effectiveness of TMNR, demonstrating its superiority compared to state-of-the-art baseline methods.

Acknowledgments

This research was supported by the National Natural Science Foundation of China (Grant Nos. 62133012, 61936006, 62103314, 62073255, 62303366), the Key Research and Development Program of Shanxi (Program No. 2020ZDLGY04-07), Innovation Capability Support Program of Shanxi (Program No. 2021TD-05) and Natural Science Basic Research Program of Shaanxi under Grant No.2023JC-QN-0648.