Imputation-free Incomplete Multi-view Clustering via Knowledge Distillation

1 Introduction

Multi-view clustering (MVC) [Bickel and Scheffer, 2004; Yao et al., 2019; Wei et al., 2021; Yu et al., 2024] is an unsupervised learning paradigm specifically devised for the analysis of multi-view data (MVD), which clusters data by exploiting the feature similarities across different views. Multi-view data, which describes objects from multiple perspectives, is ubiquitous in real-world scenarios. For example, in medical diagnosis, diseases can be examined using computed tomography, magnetic resonance imaging, and ultrasound imaging. By harnessing the consistency and complementarity of information across multiple views, more precise diagnostic results can be obtained.

∗Corresponding authors

However, multi-view data is not invariably complete. For instance, some diseases can be diagnosed without conducting all possible examinations, leading to missing views in the dataset. Such data is known as incomplete MVD. As illustrated in Figure 1 (a), a canonical approach to handling incomplete MVD involves converting it into complete MVD, followed by applying MVC methods. This conversion typically includes three strategies [Wen et al., 2022]: (1) excluding data with missing views, (2) filling missing views with zero or the average of existing views [Li et al., 2014; Zhao et al., 2016], and (3) utilizing deep learning techniques to generate missing data [Wei et al., 2020a; Bu et al., 2024].

Recently, incomplete MVC solutions based on deep learning for data recovery have been put forward. For instance, COMPLETER [Lin et al., 2021] recovers missing views by minimizing the conditional entropy between different views through dual prediction. ICMVC [Chao et al., 2024] regards the kNN graph of existing views as a potential graph structure for the missing views and utilizes the message-passing capability of graph neural networks to fill in the missing data. The consistency and complementarity of multi-view data can be beneficial for feature learning and downstream tasks [Tan et al., 2021; Wei et al., 2020b]. Nevertheless, as the data scales, complex imputation becomes progressively challenging. On the one hand, interpolation of large-scale data demands a substantial amount of computing resources. On the other hand, imputation is not always effective when there is an excessive number of missing views.

Partial views can provide enough information to conduct downstream tasks [Tan et al., 2018; Ren et al., 2023a]. For example, the diagnosis of chronic kidney disease necessitates a comprehensive analysis of diverse data sources, including blood tests, urine tests, and kidney ultrasound. In contrast, Helicobacter pylori infection can be diagnosed solely through an antigen test. Moreover, experienced doctors can rapidly diagnose a patient's illness based on their accumulated medical knowledge without conducting additional examinations. Such rapid recognition is feasible, particularly when a certain margin of error is acceptable. Nevertheless, in the realm of incomplete multi-view clustering, using the available views to infer data categories rather than filling in the missing data through imputation is extremely challenging and remains an unsolved problem.

Based on the motivation mentioned above, this article proposes an Imputation-free Incomplete MVC (I2MVC) algorithm. Specifically, as depicted in Figure 1 (b), the incomplete MVC problem is decomposed into a clustering problem of MVD and a classification problem of fully incomplete MVD. We train an MVC Teacher model using contrastive learning and then train a lightweight Student model for fully incomplete MVD classification through pseudo-supervised knowledge distillation. I2MVC brings two key benefits: (1) It fully exploits the information from MVD without requiring imputation for incomplete MVD, thereby reducing processing complexity and avoiding the accumulation of errors due to imputation. (2) Through pseudo-supervised training, the Student model can adjust the number of views in the input data to a single view, performing incomplete MVD clustering without the need of all views. Then the clustering of fully incomplete data is conducted by the Student model. Extensive experiments demonstrate the effectiveness of I2MVC.

The main contributions of our work are listed as follows: (i) We propose a divide-and-conquer strategy for incomplete MVC. It converts the task into clustering complete data and pseudo-supervised classification of fully incomplete data, effectively using complete data to guide the clustering of incomplete data.

(ii) We propose an imputation-free incomplete MVC (I2MVC) based on pseudo-supervised knowledge distillation. I2MVC transforms the requirement of multiple views into a single view, reducing the required number of views when clustering incomplete data and avoiding the imputation operation.

(iii) Experiments on benchmark datasets demonstrate the effectiveness of I2MVC on both complete and incomplete MVD, and the pseudo-supervised knowledge distillation effectively guides the clustering of incomplete data.

2 RelatedWork

2.1 Multi-viewClustering

Multi-view clustering refers to methods designed to cluster complete MVD. These approaches typically include matrix factorization, multi-kernel learning, graph learning, and deep learning methods. Matrix factorization-based methods [Yao et al., 2019; Wei et al., 2020b; Peng et al., 2023] decompose data matrices into low-rank matrices constrained by the consistency of multiple views and clustering on the low-rank matrices. Multi-kernel learning-based methods [Zhang et al., 2021; Liu et al., 2023; Su et al., 2024] combine predefined kernels from different views, either linearly or non-linearly, to enhance clustering performance. Graph learning-based methods [Yang et al., 2022] aim to learn a consistent affinity graph across all views and perform clustering algorithms like spectral clustering on the consensus graph. Deep learning-based methods [Xu et al., 2022; Ren et al., 2023b] leverage deep networks to learn latent representations and then use learned representations to partition the data into distinct groups.

Learning cohesive and discriminative representations from different views is crucial for deep learning-based MVC methods. Contrastive learning has been extensively employed to learn consistent representations frommultiple views, presenting significant advantages in MVC. For example, MCGC [Pan and Kang, 2021] regularizes the consensus graph with a graph contrastive loss. CMHHC [Lin et al., 2022] uses contrastive learning to align sample-level representations across views and performs hierarchical clustering in the hyperbolic space. DealMVC [Yang et al., 2023] uses global and local contrastive calibration loss to jointly optimize interacted cross-view features at two levels for multi-view clustering. Simple Contrastive MVC (SCM) [Luo et al., 2024] fuses data at the data level instead of the feature level, and constructs augmented views via masking and adding noise, training an auto-encoder with contrastive learning. Despite their strong clustering performance, these methods do not address missing views, leading to suboptimal results when applied to incomplete data.

2.2 Incomplete Multi-viewClustering

Most incomplete MVC solutions transform incomplete data into complete data through imputation and then apply MVC for clustering. Common imputation strategies include simple methods (such as zero imputation and mean imputation) [Zhou et al., 2024] and generative methods (such as diffusion imputation and graph neural networks) [Wen et al., 2024; Chao et al., 2024]. COMPLETER [Lin et al., 2021] proposes recovering missing views at sample-level, rather than missing similarity information, and achieves data recovery and consistency learning within a unified informationtheoretic framework, improving the interpretability of incomplete MVC algorithms. However, sample-level recovery will lose instance commonality. ProImp [Li et al., 2023] constructs a sample-prototype relationship and performs data recovery using prototypes from the missing views and the sample-prototype relationships inherited from observed views. Graph convolutional networks (GCN) [Kipf and Welling, 2017] have an advantage in learning community consistency, providing a new paradigm in imputation. ICMVC [Chao et al., 2024] utilizes the message-passing mechanism of GCN to complete missing latent representations by leveraging neighbors of missing instances. DVIMC [Xu et al., 2024] parametrizes the approximate posterior of each view using Variational auto-encoders (VAEs) and integrates information through Product-of-Experts, thereby avoiding the need for imputation. Although DVIMC learns the data distribution of different views, it ignores the consistent information across views. Compared with the existing incomplete MVC methods, our I2MVC uses multi-view data for pseudo-supervised distillation and trains a Student model that can cluster incomplete data with just one view.

3 Methodology

This section introduces the proposed I2MVC, which mainly contains three components: self-supervised simple contrastive MVC, pseudo-supervised knowledge distillation, and imputation-free clustering for fully incomplete MVD. Figure 2 shows the schematic diagram of I2MVC.

3.1 Preliminary

Given an MVD dataset X = {X1 , . . . ,Xv , · · ·XV } with N samples and V (V ≥ 2) views, Xv = {xv1 , · · · , xvN } ∈ RN×dv is the data matrix of the v-th view, where dv is its feature dimension. When there are no missing views in the data, it is referred as complete MVD. When all samples have [1, V - 1] missing views, it is referred as fully incomplete MVD. ˜X denotes the feature matrix formed by concatenating all views in complete MVD. For problem analysis, suppose ˜Xm denotes randomly masked data, and ˜Xn denotes the matrix with Gaussian noise. ˜Xs denotes mixing all views of complete MVD. For fully incomplete MVD, existing views are averaged to generate ˜Xf . The encoder embeddings are denoted as ˜Z, the clustering distribution as Q, and the auxiliary distribution as P. The clustering partition for complete MVD is C c , while that for fully incomplete MVD is C f .

3.2 Self-supervised Simple Contrastive MVC

I2MVC divides the incomplete MVD into complete MVD and fully incomplete MVD for clustering. For complete MVD, any deep MVC algorithm can be used. Simple Contrastive MVC [Luo et al., 2024] efficiently transforms view-specific encoders into a single encoder via data-level fusion, thereby reducing the model size. However, its feature learning and clustering processes are isolated, causing a lack of clusteringoriented guidance during feature learning. Given that, we propose Self-supervised Simple Contrastive MVC (S3CM) to make the joint optimization of feature learning and clustering.

Before concatenation, Principal Component Analysis (PCA) [Hotelling, 1933] is applied to X to obtain dimensionaligned features. Consistent with Simple Contrastive MVC, data-level fusion on complete MVD is performed to obtain ˜Xt . Then random masking and Gaussian noise are applied to generate augmented data ˜Xm and ˜Xn , which are used to construct sample pairs of augmented views for contrastive learning.

Auto-encoder (AE) efficiently conducts unsupervised representation learning by minimizing the reconstruction error. Given its simplicity and effectiveness, we apply AE as a backbone network to learn the embedding of the concatenated view and augmented views:

where f t θ is the encoder and θ is the learnable weight.

The reconstruction loss optimizes the feature learning of the AE:

where ˆXt , ˆXm, ˆXn are reconstructed data by decoder gtθ :

To learn the consistent information between different augmented views, a contrastive loss is constructed:

where Ln,m i and Lm,n i are the mutual contrastive InfoNCE loss [Oord et al., 2018] of the i-th instance as follows:

where Sn,m i , Sn,t are the cosine similarity between two sample features. S - = {˜zvj }v=n,m j̸=i is the negative samples set of ˜zni . τ1 is a temperature coefficient, which is set to 0.5 according to Simple Contrastive MVC [Luo et al., 2024]. We consider different augmented views of the same sample as positive pairs and other samples as negative ones.

To ensure that representations learned by AE are aligned with the clustering objective, we use a self-supervised clustering loss to jointly optimize clustering and feature learning:

where µj is the cluster centers vector of the j-th cluster, and Qij is the probability of the i-th instance assigned to the j-th cluster. P is a target distribution for promoting highconfident probabilities to be larger and low-confident ones to be smaller, which is controlled by the KL divergence [Kullback and Leibler, 1951]. To obtain reliable initial clustering centers for calculating clustering distribution Q, we pre-train AE f t θ , gtθ using the loss function as follows:

The initial clustering centers are initialized by conducting k-means [Hartigan and Wong, 1979] on the embedding learned by AE. The S3CM is optimized by Lt :

where α is a hyper-parameter controlling self-supervised clustering.

Finally, the clustering labels of complete MVD are obtained by selecting the index with maximum probability:

where k is the pre-defined number of clusters.

3.3 Pseudo-supervised Knowledge Distillation

S3CM uses contrastive loss, reconstruction loss, and selfsupervised clustering loss to fully learn information from complete MVD, which can be applied for feature learning and clustering in fully incomplete MVD. However, due to the missing views, its formulation cannot directly fit the input requirement of S3CM.

To enable clustering using only the available views in fully incomplete MVD, we transfer the knowledge from S3CM to a simpler AE through knowledge distillation [Hinton, 2015]. This involves training a model capable of learning singleview features. Specifically, we first adapt the AE's input by mixing up the views in complete MVD to generate ˜Xs , where each view represents a separate sample. The AE then learns data features through the following process:

It is also optimized by reconstruction loss Lsrec . The soft clustering assignment Qs is calculated according to the welltrained center vectors. Additionally, to enable the Student model to learn features without ground-truth supervision, we design the following offline distillation process:

where τ2 is the distillation temperature and β is also a balanced hyper-parameter. Since no ground-truth labels exist, the one-hot vector of clustering pseudo-labels is substituted for the classification pseudo-supervision information. This ensures that the Student AE can generalize the knowledge learned by S3CM, enabling feature extraction and clustering for fully incomplete MVD with its observed views.

3.4 Imputation-free Clustering for Fully Incomplete MVD

Considering that the number of available views in fully incomplete MVD varies across samples, we compute the feature ˜Xf for each sample by averaging the features of its available views. The Student model then clusters these samples, producing the clustering result C f for fully incomplete MVD:

Finally, the clustering partitions C c , C f from complete MVD and fully incomplete MVD are simply merged to obtain the final clustering partition C, since C c , C f are induced from the same cluster centers.

3.5 Analysis of Computational Complexity

Assume the incomplete dataset contains N samples, V views, and k clusters, with a view missing rate of η. I2MVC adopts mini-batch training, with the batch size denoted as |B|. The computational complexity of each stage is analyzed as follows: In the data division stage, the algorithm determines whether each sample has complete views and partitions the data accordingly. This operation has a complexity of O(NV ). The complexity of the S3CM phase primarily arises from the contrastive loss computation, resulting in a complexity of O(((1 - η)N)2 ). During the pseudosupervised knowledge distillation phase, the complexity is O((1 - η)Nk), as it involves assigning pseudo-labels and training the lightweight classifier on (1-η)N complete MVD samples. The imputation-free clustering phase, responsible for clustering ηN fully incomplete MVD samples, has a complexity of O(ηNk). For incomplete data clustering, overall complexity is O(|B|2 +NV + |B|k); for complete data clustering, it is O(|B|2 + |B|k), both dominated by S3CM.

4 Experiments

4.1 Experimental Setup

Datasets. The datasets used in this study are benchmark datasets widely used in MVC, including NGs [Hussain et al., 2010], BDGP [Cai et al., 2012], WebKB [Craven et al., 1998], and MNIST-USPS [Peng et al., 2019]. These datasets not only contain different modalities and different amounts of samples, but also different numbers of views. Table 1 provides the statistics information of these datasets.

Since our focus is on incomplete MVD, we randomly selected data from each category to construct two subsets: complete MVD and fully incomplete MVD, with a missing rate of η, where 0 ≤ η < 1. To create the fully incomplete MVD, we applied randommasking to set the number of views for each sample to a range of [1, V - 1], where V is the total number of views. This setup ensures that the constructed datasets simulate scenarios involving randomly missing views.

Metrics. The performance of all methods was assessed using three primary evaluation metrics: clustering accuracy (ACC), normalized mutual information (NMI), and adjusted Rand index (ARI). To enable comprehensive algorithm comparison, we additionally introduced four computational efficiency indicators: peak GPU memory utilization (GPU MEM), total trainable parameters (PARAM), computational time per epoch (TPE), and total execution time (TIME). These metrics collectively serve to quantify both the effectiveness and efficiency of the evaluated approaches.

Baselines. Baselines span three categories: complete MVC (DIVIDE [Zhang et al., 2024] and SCM [Luo et al., 2024]), imputation-based incomplete MVC (COMPELTER [Lin et al., 2021], ProImp [Li et al., 2023], ICMVC [Chao et al., 2024]), and imputation-free incomplete MVC (APADC [Xu et al., 2023] and GIMVC [Bai et al., 2024]). All baselines conduct clustering using k-means [Hartigan and Wong, 1979] on the latent representation of data.

Implementation Details. We used the publicly available code for the compared methods and followed the implementation details provided in their original papers. For I2MVC, its backbone network structure was kept consistent with SCM [Luo et al., 2024] to ensure a fair comparison. The code of I2MVC was developed using Python 3.8 and PyTorch 1.13, and all experiments were conducted on a Tesla T4 GPU with 16GB memory. The distillation temperature τ2 was set to 2, and the view missing rate varied within {0, 0.1, 0.3, 0.5, 0.7}. The learning rate for the Teacher model was set to 3e-4 during both the pre-training and training phases, while the Student model in the distillation phase used a learning rate of 3e-5. Each phase was trained for at least 50 iterations. The hyper-parameters α, β were searched within {0.001, 0.01, 0.1, 1, 10} and ranged 0.1 ∼ 0.9 with step 0.1, respectively. All algorithms were performed five times, and the average value and standard deviation were measured.

4.2 Comparison with Baselines on Complete and Incomplete MVD

We compared the clustering performance of eight methods across four datasets, including both complete MVD and incomplete MVD (with a missing rate of η=50%). The results are shown in Table 2. Based on the results, we make the following observations: (i) Our proposed I2MVC achieves competitive results on both complete and incomplete MVD, especially on incomplete data. (ii) Although complete MVC algorithms are impacted by view missing, applying simple filling techniques can still yield reasonable clustering performance, which suggests it is possible to achieve MVC without specific imputation. (iii) Imputation-based clustering methods generally outperform other imputation-free methods except for our I2MVC. Intuitively, employing imputation to complement missing views will introduce additional information. However, when the view missing rate is high, the quality of the supplemented views is compromised, leading to suboptimal clustering outcomes. (iv) Since our I2MVC uses the clustering partitions of the complete view data as supervision information to guide the clustering of incomplete data, I2MVC stands out among the imputation-free methods. These results demonstrate the effectiveness of I2MVC.

4.3 Performance with Different Missing Rates

Figure 3 presents the clustering performance of eight methods applied to the BDGP with varying missing rates within {0, 0.1, 0.3, 0.5, 0.7}. I2MVC exhibits a more pronounced clustering effect on data with low missing rates than other methods. Moreover, the proposed approach retains its ability to effectively cluster data despite the increase of the missing rate. These findings demonstrate the effectiveness of the proposed clustering framework.

4.4 Ablation Study

To validate the effectiveness of the key components (i.e. S3CM and pseudo-supervised knowledge distillation) in our proposed method, we designed several variant models: (i) SCM is the simple contrastive multi-view clustering network [Luo et al., 2024]; (ii) I2MVC w/o S3CM removes the proposed S3CM module and directly uses the MVD information learned by original SCM to guide the training of the Student network; (iii) I2MVC w/o PSKD removes the pseudosupervised knowledge distillation (PSKD) module from the I2MVC framework. In this case, the Student model is trained solely using the reconstruction loss of the auto-encoder, without guidance from the well-trained Teacher model on the MVD. Besides, we recorded the clustering performance at each clustering stage.

The ablation experiments were conducted using the NGs and BDGP datasets, with a missing rate η = 0.5. The results about different components are presented in Figure 4. Overall, removing any single component from I2MVC leads to a performance decrease, which indicates that each component of the proposed method is effective. Specifically, after removing the self-supervised clustering loss, the clustering performance on both datasets significantly deteriorates, demonstrating the important role of the proposed S3CM in learning clustering-friendly representations. Additionally, after removing the pseudo-supervised knowledge distillation, the clustering performance on both datasets decreases sharply, suggesting that the hard clustering pseudo-labels and soft clustering logits generated by S3CM play a guiding role when training the Student model. As illustrated in Figure 5, the results across distinct clustering stages reveal that S3CM not only significantly enhances the clustering performance of the Teacher module but also exerts a notable influence on clustering outcomes in subsequent phases. In contrast, PSKD primarily affects the clustering efficacy of fully incomplete MVD (FIMVD).

4.5 Model Analyses

Hyper-parameters. I2MVC involves hyper-parameters during training: α that controls the self-supervised clustering in S3CM, and β that controls pseudo-supervised knowledge distillation. We conducted a hyper-parameter search on the NGs and BDGP datasets, with the results visualized in a 3D bar chart in Figure 6. The results indicate that I2MVC is not very sensitive to these hyper-parameters. Generally, the value of α can be set to 1, and β can be set to 0.5.

Comprehensive Performance. Figure 7 presents a comparative analysis of the comprehensive performance across competing methods. The area occupied by each method within the radar chart is proportional to its aggregated performance. Evidently, I2MVC gains significant superiority in integrated performance compared to existing MVC and IMVC baselines.

Convergence. Figure 8 plots the loss function values across distinct clustering stages on the BDGP dataset. The results demonstrate that the proposed I2MVC exhibits a consistent loss decline across all stages, ultimately achieving robust convergence.

Visualization. We used t-SNE [van derMaaten andHinton, 2008] to visualize the clustering results and features with 50% missing views to intuitively demonstrate the clustering performance of I2MVC on both complete MVD and incomplete MVD. As shown in Figure 9, I2MVC not only clusters complete data effectively but also learns information from complete data to guide the clustering of incomplete data. Moreover, the clustering results of incomplete data indicate that it is feasible to achieve effective clustering of incomplete data using only the available views without imputation.

5 Conclusion

We propose an imputation-free incomplete MVC framework (I2MVC) via pseudo-supervised knowledge distillation, to address the error accumulation issue inherent in existing imputation-based MVC methods. Experimental results show that I2MVC not only achieves superior clustering performance on incomplete MVD but also performs well on complete MVD. Future work will focus on exploring more flexible mix-up strategies.

Acknowledgements

This work is supported by National Key Research and Development Program of China (No. 2024YFF1206604), NSFC

• (62272276 and 62432006), Shandong Provincial Natural Science Foundation (No. ZR2024JQ001), Taishan Scholars Program (No. tsqn202306007 and tsqn202408317).