Dual Robust Unbiased Multi-View Clustering for Incomplete and Unpaired Information

1 Introduction

Multi-view clustering (MVC) aims to learn more accurate common representations of multi-view data by exploiting both the consistency and complementarity of multi-view information. However, the above methods heavily rely on the assumptions of view consistency that is, the correspondences between multi-view data for the same object are complete and instance completeness that is, the multi-view data itself is complete without missing. In real-world scenarios, however, the correspondences between multi-view data are often incomplete. PVC [Huang et al., 2020] provides a classic example: due to street surveillance cameras being distributed at different locations along a street, the target of interest may be captured at different times and locations by different cameras. Due to such spatial or temporal complexities, it is difficult to obtain cross-view correspondences, leading to the issue of PVP. Similarly, due to issues like unstable data transmission or storage device failures, data itself may be lost, resulting in the issue of PSP. When multi-view data is affected by PVP or PSP, existing MVC methods tend to produce suboptimal results.

For PVP, most existing studies rely on contrastive learning to reconstruct the alignment relationships of unaligned data by learning the alignment relationships of known aligned data. However, unaligned data only lacks alignment relationships, while the data itself still exists. Therefore, focusing solely on the aligned data while ignoring the information contained in the unaligned data is unreasonable. Such an approach may inadvertently introduce PSP issues into multiview data that originally only had PVP issues, as illustrated in Figure 1(a).

In recent years, various studies have proposed different methods to address PSP, such as matrix factorizationbased methods[Li et al., 2014; Zhao et al., 2016; Shao et al., 2015; Hu and Chen, 2019a], incomplete multi-kernel learning[Bach and Jordan, 2002; Liu et al., 2020], and graphbased approaches[Wang et al., 2019; Liu et al., 2019]. However, due to the incompleteness of multi-view data, the distributions learned from incoherent data tend to be biased. Specifically, because of missing data, the cluster centers or prototypes of different classes within each view may shift. When extended to the multi-view, this results in incorrect correspondences between the cluster centers or prototypes of the same class across different views, as illustrated in Figure 1(b).

Moreover, most current methods addressing PSP or PVP, such as Deep Incomplete Multi-View Clustering (DIMVC) [Lin et al., 2021a; Zhu et al., 2019; Wang et al., 2020; Wen et al., 2021; Zhao et al., 2024a; Zhao et al., 2025] and Deep Partial View-Alignment Clustering (DPVC) [Yang et al., 2021a; Wang et al., 2024a; Zhao et al., 2024b], adopt a contrastive learning approach. They treat the representations of different views of the same sample as positive pairs, while treating the remaining samples across different views as negative pairs. However, among these negative pairs, there may exist sample pairs that, while not belonging to the same sample, belong to the same class. For clustering tasks, the model needs to learn more about cluster-level separability. So from the perspective of clustering tasks and clusters, this negative pair construction strategy may introduce noisy information, i.e., false negatives, thereby leading to suboptimal clustering performance.

To address the aforementioned challenges, we propose a novel method called Dual Robust Unbiased Multi-View Clustering for Incomplete and Unpaired Information (DRUMVC), which simultaneously addresses PSP and PVP within a unified framework. Specifically, to tackle the issue of incorrect PSP introduction in PVP and the clustering center shift caused by missing data in PSP, we leverage complete and aligned data within PSP or PVP, rather than relying on clustering centers, as an anchor set to explore similarity relationships with the complete dataset. This approach avoids the learning of erroneous correspondence information caused by clustering center shifts.

Furthermore, we propagate anchor graph information across views based on the assumption of semantic consistency [Wang et al., 2022a], thereby constructing highquality cross-view correspondences for incomplete and unpaired data. This enables the integration of incomplete and unpaired data into the model's learning process, while also addressing clustering center shift issues by compensating for the missing cross-view correspondences in incomplete data.

Moreover, to mitigate the impact of noise in sample construction, we also design a dual noise-robust contrastive learning loss. This loss aims to reduce the introduction of noise and alleviate the effect of false negatives on model training, both during sample pair construction and throughout the contrastive learning process. Leveraging these modules, DRUMVC is capable of learning more complete and robust consistency information for multi-view data, thereby achieving class-level cross-view correspondence reconstruction. The contributions of this work can be summarized as follows:

• We propose an unbiased multi-view representation learning module that constructs an anchor graph for the
• entire dataset by leveraging aligned and complete data as the anchor set. Then through cross-view propagation, it establishes high-quality cross-view correspondences for unaligned and incomplete data, thereby incorporating data with PVP and PSP into the model's learning process. This approach effectively avoids issues such as the erroneous introduction of PSP in PVP methods and clustering center shifts.
• From the perspectives of sample pair construction and model learning, we design a dual noise-robust contrastive learning loss. By exploiting the neighborhood information of complete and aligned data, the method filters out most of the noise introduced by random negative sample selection. Additionally, based on the similarity assumption for positive and negative samples (i.e., samples of the same class should be close in highdimensional space, while samples of different classes should be far apart), we construct a novel contrastive loss to mitigate or even eliminate the impact of false negatives during model training.
• We conduct extensive experiments on several commonly used real-world datasets under various scenarios, including PVP, PSP, and their simultaneous presence. The proposed method consistently achieves state-of-the-art performance. Moreover, the model demonstrates exceptional clustering performance even when the data is severely corrupted.

2 Method

In this section, we provide a detailed introduction to the proposed DRUMVC, designed to simultaneously address both incompleteness and unalignment issues. First, we describe PSP and PVP that needs to be addressed. Then, the implementation process of the model is elaborated in the subsequent three sections. The overall model architecture is illustrated in Figure 2.

2.1 ProblemFormulation

For {X(v) }Vv=1 = {x(v) 1 , x(v) 2 , . . . , x(v) N }Vv=1 a given multiview dataset, where V represents the number of views, N denotes the number of samples in each view, and X(v) ∈ RN×D .Due to the presence of PVP and PSP, the original data can be divided into {S(v) }Vv=1 = {s(v) 1 , s(v) 2 , . . . , s(v) NAC }Vv=1 and {W(v) }Vv=1 = {w(v) 1 ,w(v) 2 , . . . ,w(v) NUI }Vv=1 , where NAC +NUI = N. The sets {S(v) }Vv=1 and {W (v) }Vv=1 represent complete and aligned data, and data affected by PSP, PVP, or both, respectively. In this paper, we reorganize the multi-view dataset such that the complete and aligned data is placed in the first part, while the data with PVP, PSP, or both is placed in the latter part. Thus, we have {X(v) }Vv=1 = {S(v) , W (v) }Vv=1 .

2.2 Mutual Information-like Representation Learning

Data samples from different views often differ in dimensionality due to variations in the inherent structure of the data. Additionally, real-world multi-view data typically exhibit high dimensionality. As a result, directly processing the raw data not only suffers from the impact of redundant information but also incurs high computational complexity in terms of both time and space. To address these, We will derive the latent representation Z (v) for each view using a set of view-specific encoders:

where θv is the network layer parameters of encoder.

Since the original data {X(v) }Vv=1 can be divided into two parts, {S(v) }Vv=1 and {W(v) }Vv=1 , the corresponding {Z (v) }Vv=1 can similarly be divided into two parts, {Z (v) s }Vv=1 and {Z (v) w }Vv=1 . It is worth noting that, in recent years, reconstruction loss based on information theory [Oord et al., 2018; He et al., 2020] has garnered increasing attention:

where I represents mutual information, and NAC refers to the number of aligned and complete samples. By maximizing the mutual information between data from different views, the consistency information across views can be effectively explored.

Leveraging the concept of mutual information, we no longer directly feed the latent representations of multiple views into their respective decoders. Instead, we concatenate the latent representations of multiple views, denoted as {Z (v) s }Vv=1 , and feed the concatenated representation into the decoders. Consequently, the overall loss for this module is:

where [...] represents the concatenation operation for multiview data, and θg denotes the learnable parameters of the decoder.

2.3 UnbiasedMulti-ViewRepresentation Learning

When multi-view data simultaneously suffers from both incompleteness and unalignment, the situation becomes even more complex. The corrupted data is difficult to explore for multi-view consistency information because it not only has missing data but also lacks cross-view correspondences. Furthermore, when the proportion of incomplete and unaligned data exceeds 50%, it becomes unreasonable to infer overall data information using only a small portion of the data. Therefore, we propose an unbiased multi-view representation learning approach to complete the missing and misaligned data, enabling the model to go beyond learning only from the aligned and complete portions of the data.

We use complete and aligned data {S(v) }Vv=1 as the anchor set {A(v) }Vv=1 to measure the similarity relationship between the data within each view and the anchors. The anchor graph C (v) ∈ RN×NAC is defined as:

where ⟨i⟩v denotes the index set of the m (m << n) nearest anchors to x(v) i , determined using a distance function such as the l2 distance. Although the anchor graph C (v) has NAC columns, each row contains only m non-zero values that sum to 1. To conserve space, the zero values can be omitted, reducing the graph to the formRN×m. However, for the sake of clarity and to facilitate understanding in subsequent descriptions, we choose to retain the zero values in the paper. Furthermore, Kδ (x(v) i , a(v) j ) = exp(-D2 (x(v) i , a(v) j )/σ2 ) is a commonly used Gaussian kernel function with a bandwidth parameter δ.

We achieve learning from incomplete and unaligned data by leveraging the information contained in the anchor graph C (v) . Specifically, As AMCP [Zhao et al., 2024c] says, if the corresponding samples in different views of multi-view data describe the same object in different forms, these corresponding samples across multiple views are not only similar themselves but also exhibit similar neighborhood information. When extended to incomplete and unaligned data, even if a sample in one view cannot find its corresponding sample due to PVP or PSP, we can reasonably infer the spatial neighborhood information of its corresponding sample based on the neighborhood information of the given sample. This enables the cross-view transfer of neighborhood structures.

Moreover, due to our anchor selection strategy and the consistency assumption of multi-view data [Luo et al., 2018], aligned and complete data across different views are expected to be similar. Consequently, by leveraging the alignment relationships of anchors and the cross-view transfer of neighborhood structures, we can construct high-quality cross-view correspondences for misaligned and incomplete data.

where [NAC :, :] refers to the rows of anchor graph information corresponding to incomplete and unaligned data in the i-th view. Z (j) s represents the aligned and complete samples and anchor set in the j-th view. Meanwhile, Z (j) c is constructed in the j-th view based on the neighborhood information of incomplete and unaligned data from the i-th view, forming cross-view correspondences.

Thus, incomplete and unaligned data can also be incorporated into the scope of model learning. However, as crossview correspondences are constructed through the transfer of cross-view neighborhood structures, minimizing the spatial distance between them is not appropriate. Instead, we propose optimizing it to a boundary using a bounded contrastive learning loss.

where a is the boundary of similarity. Extending this to the entire view, the overall loss of this module is:

2.4 Dual Noise Robust Contrastive Learning

To mitigate or even eliminate the impact of false negatives, we designed a Dual Noise-Robust Contrastive Learning (DRC) loss from two perspectives: sample pair construction and the contrastive learning process.

where P = 1/0 for positive/negative pairs, L(v) pi will take effects when the pairs are positive, and L(v) ni acts on negative pairs.

For the anchor graph C (v) computed in the above module, since aligned and complete data are used as the anchor set, the similarity relationships corresponding to this part of the data in C (v) can essentially be transformed into the spatial structural information of the aligned and complete data within the view. In other words, each sample can derive its similarity relationships with its top m closest samples through C (v) .

To further process the aligned and complete samples in each view, we define an index set ⟨k⟩vi , to store the index information of the top K (K ≤ m) nearest neighbors for each sample. Additionally, based on the assumption of semantic consistency [Wang et al., 2022b], neighborhood information within a view can be transferred across views and converted into adjacency relationships with cross-view samples.

Thus, for aligned and complete data, besides the explicit cross-view corresponding samples that can serve as positive sample pairs, the cross-view neighborhood information also contains implicit positive sample pair information. Finally, we mine the consistency information of multi-view data by minimizing the distance between these relationships:

where z(i) st represents the t-th aligned and complete sample in the i-th view, while z(j) pk denotes the k-th indirect positive sample identified in the j-th view based on the cross-view index set ⟨k⟩vi . β is a hyperparameter.

Additionally, by learning positive sample pairs, the model can further leverage the consistency information across multiple views, improving the accuracy and reliability of crossview information transfer in unbiased multi-view representation learning.

Moreover, the cross-view index set ⟨k⟩vi is not only utilized to mine indirect positive sample pair information across views but also serves to reduce the introduction of noisy information during sample pair construction. Specifically, for each aligned and complete sample z(i) st as the anchor, M negative samples z(j) n are randomly selected from the aligned and complete data across views, with the constraint that the crossview negative samples n /∈ ⟨k⟩it . By preliminarily filtering neighborhood information, this approach effectively reduces the introduction of false negative pair information to some extent. Furthermore, we designed a noise-robust contrastive learning loss to mitigate or even eliminate the impact of false negative pairs during the model training process:

where d refers to the Euclidean distance between the negative sample pair z(i) st and z(j) n , c is a constant and md is a distance margin that is calculated only once during the initial stage using the following formula:

where Np refers to the total number of explicit positive sample pairs, while Nn represents the total number of negative sample pairs.

Next, we will demonstrate the noise robustness of the loss function using the visualizations in Figure 3. When the distance between negative sample pairs lies within the range [0,md], the gradient of our loss function differs from that of conventional contrastive learning loss functions. The latter simply aims to increase the distance between negative pairs, failing to mitigate the impact of noisy negative pairs. For false negative pairs (FNP) within [0,md/2], they are considered to belong to the same class but not the same sample, with relatively small distances. While the objective of reconstructing class-level relationships may require reducing these distances as much as possible, it is still necessary to preserve the discriminative information within the same class cluster to extract more accurate spatial structural information. Therefore, for FNP within [0,md/2], we moderately increase their distance.For FNP within the range [md/2,md], instead of simply pushing them apart as in conventional contrastive learning loss functions, we reverse their gradients and gradually pull them closer. This approach enhances the model's robustness to noise introduced by random sampling. For negative pairs within the range [md, 2md], we directly push their distances apart.

Considering that a small number of TNP may exist within [0,md], the loss functions designed for explicit and indirect positive pairs, along with the unbiased multi-view representation learning module, enable the model to learn sufficient consistency information between samples. For TNP with relatively large distances, the model has already learned the discriminative information between negative sample pairs, giving it the ability to distinguish true negative pairs. Additionally, compared to L(v) pi , which focuses on the quadratic term of the sample pairs, L(v) ni focuses only on the linear termof the distance for negative pairs. Furthermore, the gradient of L(v) ni within [0, md] has a maximum value of just 1. As a result, the small number of TNP within [0,md] will not be optimized in the wrong direction due to the small reverse gradient.

Overall, the loss function is as follows:

Once the model converges, the cross-view correspondence for incomplete data is determined by calculating the distance matrix D between views. The mean of the neighboring data points is used for imputation. For PVP, the nearest sample across views is selected as the correspondence.

3 Experiments

3.1 Experimental Settings

In this study, we utilized four multi-view datasets: Scene15 [Fei-Fei and Perona, 2005], Reuters [Amini et al., 2009], NoisyMNIST [Wang et al., 2015], and MNIST-USPS [Peng et al., 2019]. A brief description of each dataset is provided in Table 1.

In this paper, we conduct experiments on multi-view datasets with an alignment rate of 0.5, a completeness rate of 0.5, and various scenarios of unaligned incomplete multiview datasets. The primary evaluation metrics include Accuracy (ACC), Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI). Higher values for these metrics indicate better clustering performance.

3.2 Comparisons with State of the Arts

Based on various settings, we compared DRUMVC with 18 latest state-of-the-art multi-view clustering baselines, including BMVC [Zhang et al., 2018], AE2-Nets [Zhang et al., 2019], PVC [Huang et al., 2020], MvCLN [Yang et al., 2021b], EGPVC [Zhao et al., 2023], GCFAgg [Yan et al., 2023], CMK [Liu et al., 2023], SURE [Yang et al., 2022], OTGM [Wang et al., 2024b], NC3L [Qian et al., 2024], PMVC [Li et al., 2014], UEAF [Wen et al., 2019], DAIMC [Hu and Chen, 2019b], EERIMVC [Liu et al., 2020], COMPLETER [Lin et al., 2021b], DSIMVC [Tang and Liu, 2022], ProImp [Li et al., 2023], and DIVIDE [Lu et al., 2024]. The best experimental results are in bold, with the second-best results marked using an underscore ' '. A dash '-' denotes impractical methods due to excessive time or memory consumption.

Since some of the comparison algorithms were not specifically designed for the PVP or PSP problems, we adopted the same approach as SURE. For partially aligned multi-view datasets, we used the Hungarian algorithm to realign the data before inputting it into the relevant models. In the case of incomplete multi-view datasets, we employed the mean value within each view for data imputation.

As shown in Table 2, our model achieves satisfactory experimental results in both settings compared to existing stateof-the-art algorithms for PVP or PSP. Although the performance improvement in unpaired setting over the NC3L model is not particularly significant, the lightweight nature of DRUMVC stands in contrast to the complex mathematical derivation process utilized by NC3L. Furthermore, in the challenging context of the incomplete multi-view dataset Reuters, DRUMVC exhibited a minimum of 19% improvement across all three metrics. These experimental findings validate the capability of DRUMVC to effectively mitigate data missing or unpaired challenges in multi-view datasets.

As of now, SURE is the only effective method that can address both PVP and PSP. Therefore, we follow up with experimental comparisons with it in a variety of PVP and PSP ratio scenarios. Table 3 reveals that, in comparison to SURE, DRUMVC yielded consistently optimal experimental outcomes across all datasets, accompanied by significant improvements. Notably, when data corruption was more severe, such as alignment rates of only 0.1 with completeness levels of 0.5 or 0.7, owing to its unbiased multi-view representation learning module (as detailed in Section 2.3), DRUMVC demonstrated an enhanced capacity to learn more comprehensive data information compared to other models. In particular, on the Noisymnist and NNIST-USPS datasets, when the alignment rate was 0.1 and the completeness level was 0.5, DRUMVC achieved a remarkable improvement of 81.15% in ACC, 92.30% in NMI, and an astonishing 156.67% in ARI on the Noisymnist dataset. Similarly, on the NNISTUSPS dataset, it achieves a 57.15% improvement in ACC, a 72.02% improvement in NMI, and a dramatic 110.66% improvement in ARI. These results not only demonstrate the effectiveness of DRUMVC in addressing the simultaneous occurrences of incompleteness and misalignment but also highlight our model's ability to maintain commendable clustering performance even in the presence of severe data degradation.

3.3 Ablation Study and Parameters Analysis

To validate the effectiveness of the designed modules, we conducted ablation experiments on three loss functions: LMIL, LUMR, and LDRC. The LMIL loss function leverages the concept of class mutual information to extract the consistency information of multi-view data; LUMR is employed to ensure that the model learns unbiased spatial structural information and comprehensive consistency information; while LDRC aims to reduce, or even eliminate, the impact of false negatives on noise. Table 4 indicates that the model's performance reaches its optimal level when all three losses are applied in conjunction.

Similarly, we evaluated the model's performance under various combinations of hyperparameters with 0.5 paired and complete rate. As shown in Figure 4, our model exhibits robust performance concerning the chosen hyperparameters.

3.4 Visualization Results

We present a visualization of the clustering results for SURE and our model DRUMVC on the Noisymnist datasets, both characterized by alignment and completeness rates of 0.5. Figure 5 shows that DRUMVC effectively learns more comprehensive spatial structural information and unbiased consistency information through its unbiased multi-view representation learning module. It also shows the dual noise-robust contrastive learning module significantly mitigates the impact of false negatives. Consequently, the clustering outcomes produced by DRUMVC do not exhibit instances of data partitioning within the same class, in contrast to SURE. Furthermore, due to the model's ability to learn complete data information, the cluster centroids within each class Cluster are more accurate.

4 Conclusion

To mitigate the erroneous introduction of PSP in PVP datasets, address the issue of cluster centroid displacement caused by data incompleteness, and tackle the presence of noise in sample pair construction, we presented a novel algorithm, DRUMVC, capable of simultaneously addressing both PSP and PVP challenges. By incorporating problematic data into the model's learning process, DRUMVC enables the model to acquire a more comprehensive understanding of the data information. Furthermore, by utilizing neighborhood information and spatial distance to implement a dual noiserobust contrastive loss, the algorithm effectively reduces the impact of noise on data information extraction. Through the synergistic interaction of multiple modules, DRUMVC achieves satisfactory clustering performance across various datasets.

Acknowledgments

This work is supported by the Science and Technol ogy Project of Liaoning Province (2024JH2/102600027, 2023JH2/101700363) and the Science and Technology Project of Dalian City (2024JJ12GX025, 2023JJ12SN029 and 2023JJ11CG005).