LRGR: Self-Supervised Incomplete Multi-View Clustering via Local Refinement and Global Realignment

1 Introduction

Deep Multi-view Learning methods have demonstrated unprecedented success in domains such as pattern recognition, computer vision [He et al., 2023], and bioinformatics

∗Corresponding author.

[He et al., 2024; Zhu et al., 2024], owing to their superior feature extraction capabilities and multi-task adaptability. Among the downstream tasks, Incomplete Multi-View Clustering (IMVC) has emerged as a critical challenge, as it aims to uncover cluster structures in scenarios where data is partially missing across multiple views.

Existing deep IMVC methods can be broadly categorized into imputation-based and imputation-free approaches. Imputation-free methods mainly focus on learning complementary fused representations or aligning different components of representations across views, such as DIMVC [Xu et al., 2022] and APADC [Xu et al., 2023]. On the other hand, imputation-based methods concentrate on completing missing multi-view data by utilizing cross-view similarity relationships or prior knowledge. A critical challenge for these methods is to achieve accurate imputation, which is essential for restoring the underlying data distribution. Existing approaches, such as Completer [Lin et al., 2021], DCP [Lin et al., 2022], SURE [Yang et al., 2022], RPCIC [Yuan et al., 2024], CPSPAN [Jin et al., 2023] and ProImp [Li et al., 2023a], focus on exploring the relationships between views, samples, or prototypes to achieve accurate imputation. However, these methods often face challenges due to insufficient exploration of prior information, resulting in imprecise imputations and blurred cluster boundaries. To address this challenge, several GCN-based IMVC [Feng et al., 2024; Pu et al., 2024] methods have been developed. Due to the inherent message passing mechanisms of GCNs, these methods provide an automatic and efficient approach to handle missing data by aggregating the representations of neighborhood nodes, using the structure information. As a representative GCN-based method, ICMVC [Chao et al., 2024] constructs the adjacency graph based on the multi-view consistency assumption and addresses missing values using GCN.

Although the aforementioned approaches have made significant progress, their reliance on cross-view similarity leads to distribution shifts, particularly when the identified similar neighbors belong to different clusters. This issue becomes challenging at high missing rates, as maintaining cross-view consistency becomes increasingly difficult. Additionally, the message passing mechanism of GCN aggregates neighbors of each node through the constructed adjacency matrix. As the number of propagation steps increases, the node embeddings become progressively similar to those of their neighbors. To better illustrate above two issues, we empirically analyze and illustrate them on the CUB dataset under the 50% and 70% missing rates. As shown in Fig. 1, the first column displays a subgraph of CUB, highlighting the imputed sample and its neighboring nodes selected from the adjacency matrix. Nodes from the same cluster are annotated in the same color. We visualize the similarity matrix of the imputed node and its corresponding neighbors, which is obtained using different GCN-based methods.

To address the distribution shift caused by inaccurate neighbors and the homogeneous representation inherent in the GCN, we propose a novel self-supervised local refinement and global realignment (LRGR) model for incomplete multiview clustering. Specifically, we introduce the global semantic realignment module to generate consistent pseudo-labels and compute confidence scores, enabling semantic-level contrastive learning to mitigate the impact of incorrect neighbor aggregation for missing samples. To tackle the second issue, we design a local imputed refinement module that preserves the view versatility of the imputed representations, guided by view-specific prototypes. Overall, the contributions of this paper can be summarized as follows:

• From the aspect of data imputation in GCN, we observe the potential distribution shift issue caused by the
• inaccurate neighbors and propose a self-supervised realignment module to alleviate this shift problem, named global realignment;
• From the aspect of representation learning in GCN, we observe the underlying similar imputed representations and propose a local refined module to emphasize the versatility, named local refinement;
• Extensive experimental results verify the effectiveness of LRGR and show considerable improvement over ten state-of-the-art methods on five multi-view datasets.

2 RelatedWork

IMVC is an important branch of the MVC field that has attracted widespread research. Classic IMVC methods can be roughly categorized into three types: kernel-learningbased methods [Liu et al., 2019a; Liu et al., 2019b; Li et al., 2023b], matrix-factorization-based methods [Hu and Chen, 2018], and graph-based methods [Wang et al., 2022b; Wen et al., 2023; Li et al., 2022]. However, their shallow architectures often struggle to effectively handle complex datasets. In response, DL-based IMVC methods have emerged, and can be further subdivided into Autoencoderbased methods [Lin et al., 2021; Lin et al., 2022; Xu et al., 2023], Graph Convolutional Network (GCN)-based methods [Chao et al., 2024; Pu et al., 2024] and Generative Adversarial Networks (GANs)-based methods [Wang et al., 2022a; Zhang et al., 2020]. In addition, to address the semantic alignment problem, several methods such as SPICE [Niu et al., 2022] and PLP-L/G [Feng et al., 2024] have been proposed. These approaches treat pseudo-labels as reliable supervision signals and align network predictions using KL divergence or cross-entropy loss. In contrast, our proposed LRGR framework focuses on resolving the distribution inconsistency of imputed samples across views. Among existing GCN-based IMVC methods, ICMVC is the most closely related to our work. While both approaches utilize GCNs for view imputation, their focuses differ significantly. ICMVC primarily exploits complementary information from high-confidence samples, aiming to enhance representation by aggregating reliable neighborhood information. In contrast, our proposed method focuses on mitigating the homogeneous representation issue inherent in GCN by leveraging view-specific information and addressing the distribution shift caused by pseudo-label guided.

3 Method

3.1 Notations and Overview

We define the incomplete multi-view data including N instances with v views as X = {X(0) , ...,X(v) }, where X(v) ∈ RN v ×dv . The latent representations obtained by GCN are defined as Z(v) = {Z(v) c , Z(v) p }, where Z(v) c ∈ RN v c ×d and Z(v) p ∈ RN v p ×d denote the complete and imputed feature matrix of the v-th view, N v c and N v p are the number of complete, imputed samples of the v-th view, d is the latent dimension. And then we construct the indicator vector P (v) , where p(v) i = 1 if the i-th sample is missing in v-th view, otherwise p(v) i = 0.

In this section, we propose a novel self-supervised local refinement and global realignment (LRGR) model for IMVC. As illustrated in Fig. 2, LRGR mainly consists of two components, i.e. a local imputed refined module to prevent homogeneous representations and a global self-supervised representation module to alleviate the distribution shift issue. Next, we will introduce the two modules and the final loss function in detail.

3.2 Missing View Imputation

To fully leverage structural information for missing data imputation by GCN, we first construct the view-specific adjacency matrix based on the view consistency assumption.

Missing Adjacency Matrix Handling

To better capture the data structure, we compute the similarity matrix of each view, denoted as {S(v) }Vv=1 by normalized cosine function. For the missing samples, their corresponding similarity relationships cannot be directly computed. To address this, following [Chao et al., 2024], we transfer the cross-view similarity relationships to the missing samples based on the consistency assumption. As shown in Fig. 3, the column corresponding to the missing sample in the similarity matrix is set to zero, and the missing row is imputed using the cross-view similarity. Mathematically, we first constructed the similarity matrix S(v) by cosine similarities.

where S(v) ∈ RN×N denotes the similarity matrix for each view. Next, we use the indicator vector P (v) to select the existing cross-view similarity values for the similarity transfer operation:

where S(v) [P (v) , :] selects the missing rows of the v-th similaritiesmatrix, S(u) [P (v) , :] selects the existing rows of crossview similarities matrix. After the missing similarity transfer, we construct the adjacency matrix A = {A(1) , ..., A(v) }, where a(v) i,j = A(v) [i, j] = 1 if there is an edge between x(v) i and x(v) j , otherwise a(v) i,j = 0, based on k-nearest neighbor. Then, to stabilize the training of the GCN, the adjacency matrix is normalized through calculating ˜D- 12 ˜A ˜D- 1 2 , where ˜A=A+ I is the self-connection, ˜D = ∑j ˜aij is the degree matrix.

GCN-based Instance Imputation

The core idea of GCN imputation is to leverage local structural information to iteratively update the representation of each node by aggregating features from its neighboring nodes. Specifically, after obtaining the complete adjacency matrix for each view, we feed both the adjacency matrix A(v) and the feature matrix X(v) into a three-layer GCN encoder. The node representations of each view Z(v) can be updated by the graph convolution operation of the m-th layer, as described below:

where Z(v) (m) is the output embeddings of the m-th layer, W (v) (m) indicates the trainable parameters in the m-th layer of v-th graph encoder. After that, we adopt a shared MLP layer H(v) = MLP(Z(v) ; θshared). to capture consistent patterns. Then, we utilize following reconstruction loss:

where N denotes the number of the sample, ˜X(i) = D(v) (H(v) , θ(v) ) is the reconstructed representation obtained by view-specific decoders.

3.3 Local Imputed Refinement Module

Existing imputation methods rely on the view consistency assumption, which is typically satisfied at the cost of view versatility. Furthermore, the inherent message-passing mechanism in GCNs often increases the similarity between neighboring nodes [Xia et al., 2022]. For missing samples, this exacerbates the issue of reduced versatility, resulting in a decreased discriminability of the imputed samples. To preserve the diversity of imputed samples, we leverage viewspecific prototypes C(v) ∈ Rk×d and contrastive learning to promote greater dissimilarity between imputed samples. Specifically, we select the imputed samples from each view Z(i) p ∈ RN ip×d and their corresponding counterparts in crossview Z(j) p ∈ RN ip×d to form positive and negative sample pairs. We then adopt the following contrastive loss, which maximizes the similarity of assignment probabilities between the same samples across views, while simultaneously maximizing the difference in assignment probabilities between different instances.

where z(v) i denotes the representation of the i-th sample in the v-th view, c(v) k represents the k-th prototypes in the v-th view, p(v) ik denotes the probability of the i-th sample assigned to k-th cluster in v-th view, N denotes the number of samples, s(·, ·) is the cosine similarity, τ is the temperature parameter.

3.4 Global Semantic Realignment Module Adaptive Fusion Block

To better extract the common pattern and evaluate the performance, we adopt an adaptive fusion block consisting of two convolutional layers. First, we concatenate the latent representations H(v) from each view along the channel dimension:

where Hcon ∈ R[N,view,d] . Then we pass the concatenated feature matrix Hcon through a two-layer convolutional network. The first convolutional layer applies a 1D convolution operation with n kernels along the channel dimension, expanding the number of channels from the number of views to n. In this paper, n is set as 32.

The second convolutional layer also uses a 1D convolution operation, a single convolutional kernel is applied across the feature channel dimension, which reduces the number of feature channels from n to 1.

where H ∈ R[N,1,d] means the fused common representation, W(.) and b(.) are trainable parameters.

Semantic-wise Realignment

The local Refinement module enhances the diversity of imputed representations. However, this also means that it pushes samples away without considering their semantic categories, potentially causing a distribution shift. To alleviate this issue, we achieve the global realignment by preserving semantic consistency. Inspired by [Liu et al., 2024], we compute the semantic pseudo-labels l ∈ RN and global prototypes C ∈ R[k,d] by performing k-means clustering on the fused representation H. To evaluate the confidence of the pseudolabels, we calculate the t-student distribution between the view-specific embeddings H(v) and the global prototypes C, using the result as the confidence score for the pseudo-labels.

where q(v) ik denotes the probability of the i-th view assigned to the k-th cluster in v-th view. Finally, the semantic alignment loss is computed using semantic pseudo-labels and corresponding confidence score. For samples with high confidence, the loss is designed to reinforce the model's certainty in its pseudo-labels, encouraging samples with the same labels to be grouped together while pushing those with different labels apart.

where q(v) ic is the probability of the i-th sample belongs to the c-th cluster in v-th view, s(·, ·) denotes the cosine similarity and li represents the pseudo-label of the i-th sample.

3.5 Loss Function

Finally, we combine the loss functions of the reconstruction loss, local refinement loss and global realignment loss into an overall loss function, which is expressed as follows:

where Lrec, Lloc and Lglob are defined in Eq. (4), Eq. (5) and Eq. (11), α and β are the trade-off parameters, which balance the importance between the local imputed refinement loss and the global semantic realignment loss.

4 Experiment

In this section, we evaluate the proposed LRGR method on five widely-used multi-view datasets compared with 10 stateof-the-art baselines. First, we present the basic information of experimental settings, like dataset and evaluation metrics.

In Sec. 4.2, we give a detailed analysis between our LRGR and comparison methods. Then in Sec. 4.3, we conduct comprehensive ablation studies to demonstrate the effectiveness of LRGR. Finally, we give visualization results.

4.1 Experimental Settings

We evaluate our proposed model on five widely used datasets. Specifically, CUB [Wah et al., 2011] with 600 samples, 2 views. DHA [Lin et al., 2012] with 383 samples, 2 views. COIL20 [Wan et al., 2021] with 1440 samples, 3 views. YTF10 [Wolf et al., 2011] with 38654 samples, 4 views, and UCI [Asuncion et al., 2007] with 2000 samples, 3 views. To assess performance in an incomplete data setting, we manually mask data at various missing rates. Specifically, for a dataset with v views, we randomly select m samples as missing and remove 1 ∼ v - 1 views from each. The missing rate η is defined as η = mn , where n is the number of samples. We implement our method using PyTorch 1.12 and conduct experiments on a standard Ubuntu 20.04.6 OS with an NVIDIA GTX 1060 GPU. The activation function is the RELU [Glorot et al., 2011]. Adam [Kingma, 2014] is chosen as the optimizer with the 0.0005 and 0.0001 learning rate in the pre and align training process. In pretrain stage, we update the network only with reconstruction loss in Eq.(4). And in the second stage, we update the network with Eq.(12). For comparison methods, we use the recommended parameters and network structures to ensure the best performance. For evaluation metrics, we utilize three metrics including clustering accuracy (ACC) [Cai et al., 2005], normalized mutual information (NMI) [Knops et al., 2006] and F-score[Powers, 2008] to evaluate performance.

4.2 Comparisons with State of the Arts

To verify the effectiveness of our model, we choose ten stateof-the-art IMVC baselines, including DPML [Zhang et al., 2020], UIMC [Fang et al., 2021], DSIMVC [Tang and Liu, 2022], DIMVC [Xu et al., 2022], IMVC CBG [Wang et al., 2022b], ProImp [Lin et al., 2022], DCP [Lin et al., 2022], DSIMVC++ [Yan et al., 2023], RPCIC [Sun et al., 2024] and ICMVC [Chao et al., 2024].

We first evaluate LRGR and comparisons under the Complete situation (with the missing rate of 0%) and Incomplete (with the missing rate of 50%). Table 1 demonstrates the clustering performance on ACC, NMI and F-Score. It can be observed that our LRGR significantly outperforms the state-of-the-art methods on all datasets with the missing rate of 0% and on CUB, YouTubeFace10, DHA and UCI datasets with the missing rate of 50%. Specifically, LRGR achieves a 1.42% (82.08% v.s. 80.66%) and 5.89%(84.38% v.s. 78.49%) improvement on NMI under the Complete and Incomplete scenarios on the YouTubeFace10 dataset, compared with the second best result. This increasing performance shows the effectiveness of the proposed local imputation refinement module and global semantic realignment module. To better prove the robustness of proposed LRGR, we illustrate the clustering results by increasing the missing rate from 10% to 70% with a gap of 20% on the COIL20 dataset. As shown in the Fig. 4, our LRGR substantially achieves the comparable results on three metrics on the COIL20 dataset. On the other hand, the robustness demonstrate the effectiveness of our imputation mechanism and also the global semantic realignment module.

4.3 Ablation Studies

In this section, to better understand the LRGR, we conduct a series of ablation studies. We first study and visualize the effectiveness of local imputed refinement module. Then we study the effectiveness of shared and adaptive fusion module. Finally, we conduct experiments with different combination of loss items in LRGR.

Effectiveness of Local Refinement

To demonstrate the model's ability to capture versatile information guided by view-specific prototypes, we visualize the similarity between an imputed node and its selected neighboring nodes on the CUB dataset, under a 50% missing rate, without the guidance of prototypes in the local refinement module. As shown in Fig. 6 (a), we observe that the learned imputed representation retains some discriminative power, compared with Fig. 6 (b), but does not exhibit the same degree of distinction as in Fig. 6 (b). Such results demonstrate the effectiveness and superiority of the proposed local refinement module, particularly its capability to capture versatile information and avoid homogeneous representations.

Effectiveness of Adaptive Fusion and MLP

To verify the effectiveness of the proposed adaptive fusion module, we perform ablation studies on four datasets with a 50% missing rate, where the fusion module is removed and the representations from all views are fused via an average weighted sum. To verify the effectiveness of the shared MLP layer, we conduct the ablation studies on four datasets with a 50%missing rate, where the MLP layer is removed.

Effectiveness of Loss Term

To further verify the effectiveness of each module in our proposed model, we conduct ablation experiments with different loss combinations on the UCI, COIL20 and CUB datasets under the 50%missing rate. The result of ablation study is presented in Table 2. Concretely speaking, while the reconstruction loss term Lrec can achieve comparable results on several datasets, it fails to learn discriminative representations and cannot prevent the emergence of homogeneous representations. In contrast, the proposed local imputed refinement loss effectively maintains view versatility, guided by view-specific prototypes.

4.4 Hyperparameters Sensitivity Analysis

In this study, we investigate the trade-off coefficients α and β in the loss function. We follow a common analysis strategy between local imputed refinement loss and global realignment loss. Specifically, we set the α and β varies from [0.0001, 0.001, 0.01, 0.1, 1]. As shown in Fig. 7, when α is too large, the global realignment loss becomes dominant, leading to a homogeneous representation. Conversely, when β is too large, the model tends to overly prioritize versatility at the cost of learningmeaningful representations, thus failing to maintain semantic consistency. For proposed LRGR, the trade-off coefficient α exhibits robustness in [0.0001 0.01], the trade-off coefficient β exhibits robustness in [0.0001, 1].

4.5 Visualizations

To more effectively assess the performance of the proposed LRGR, we visualize the clustering results using t-SNE. The results for our method, along with four recent baseline methods, are presented on the YouTubeFace10 dataset at a 50% missing rate, as shown in Fig. 5. From the figure, it is clear that IMVC-CBG struggles to distinguish between clusters, with blurred inter-cluster boundaries and undefined intracluster structures. For ICMVC, and RPCIC, although they show relatively clear inter-cluster boundaries, there is some overlap between clusters. In contrast, LRGR demonstrates clear cluster boundaries and compact intra-cluster structures, indicating its ability to learn a well-defined clustering structure even in the presence of missing data.

5 Conclusion

In this paper, we propose a novel Local Imputed Refinement and Global Semantic Realignment (LRGR) model to address the challenges of similar imputed representations and distribution shifts. Unlike existing methods that primarily focus on shared patterns across views, our Local Imputed Refinement module explicitly preserves the diversity information in the imputed representations, guided by view-specific prototypes. Furthermore, to alleviate the distribution shift caused by inaccurate neighbors, we ensure semantic consistency through the pseudo-labels and their corresponding confidence scores. Extensive experiments demonstrate the effectiveness and superiority of the proposed LRGR model.

Acknowledgements

The work was supported in part by the National Natural Science Foundation of China under grants 62476258, and in part by the Natural Science Foundation of Hubei Province under grant 2025AFA113, and in part by the Key Laboratory of Target Cognition and Application Technology (No. 2023-CXPTLC-005), and in part by the Fundamental Research Funds for National Universities, China University of Geosciences (Wuhan) under grant 2024XLB6.