Active Deep Multi-view Clustering

1 Introduction

Multi-view data are ubiquitous in real-world applications. For example, web pages on the Internet may contain multiple views such as images, texts, and videos. To handle these multi-view data, multi-view clustering is one of the challenging tasks and attracts much attention [Zhou et al., 2019; Zhang et al., 2019; Xie et al., 2020; Wang et al., 2020; Zhou and Du, 2023].

Since the deep neural network (DNN) has been widely used in many fields and achieves promising performance on many tasks, deep multi-view clustering attracts increasingly more attention [Xue et al., 2021; Xu et al., 2021; Lin et al., 2021a; Xia et al., 2021; Lin et al., 2021b; Wang et al., 2022; Xu et al., 2022a; Du et al., 2022; Yan et al., 2023]. These methods apply the DNN to extract more semantic representations of data and do the clustering on the learned representations. For example, [Du et al., 2021] used auto-encoder to extract features, and designed cross-entropy-based regularization and local regularization to ensure the consistency between any two views of a sample; [Xiao et al., 2023] utilized view-specific graph convolution networks to learn the representation of each view and proposed a fusion method on the attribute-level and structurelevel; [Pan and Kang, 2021] learned a new consensus graph by considering the relationship between nodes and the initial graph, and designed a novel graph-level contrastive loss.

∗ Peng Zhou is the corresponding author.

However, since multi-view clustering is still an unsupervised task, where there is no supervised information in the learning, deep multi-view clustering may still obtain unreliable results, especially when handling complicated real-world data. To tackle this problem, some semi-supervised multiview clustering methods are proposed [Chen et al., 2022; Tang et al., 2022; Qin et al., 2021; Whang et al., 2020; Zhu and Gao, 2022]. These methods apply some supervised information, such as the labels of data or the pairwise constraints, to guide the multi-view clustering. For example, [Tang et al., 2022] applied the cannot-link and must-link constraints to guide the clustering and proposed constrained tensor representation learning model based on the unified constraint to learn the representations; [Qin et al., 2021] utilized the labels of data to build the affinity matrix and applied the semi-supervised learning methods on the affinity matrix to obtain the clustering results. Although semi-supervised methods can alleviate the problem of lacking supervised information to some extent, all these methods ignore how to obtain the supervised information. The supervised information used in these methods is all pre-given or randomly generated. As we know, the performance of semi-supervised learning highly depends on the quality of the supervised information, and poor supervised information may deteriorate semi-supervised learning seriously. All these semi-supervised multi-view clustering methods ignore this supervised information selection problem.

To tackle the supervised information selection problem, in this paper, we propose a novel Active Deep Multi-view Clustering (ADMC), which introduces active learning into the multi-view clustering task. Active learning often involves multiple interactive batches. In each batch, the algorithm selects several data with a pre-given budget and queries the human annotations on these selected data [Hoi et al., 2006; Chattopadhyay et al., 2013; Wang et al., 2015a; Yan and Huang, 2018; Wang et al., 2015b; Wang et al., 2019; Halder et al., 2023; Sun et al., 2022; Zhou et al., 2023]. In our method, we design an active selection module to automatically select some important data for annotation by considering the representative and diversity properties of data. Besides the active selection module, we propose a fusion module to integrate the information in different views. Then, we propose a supervised module to apply the human annotations to train the whole model. Notice that in our active clustering setting, we wish to obtain a reliable result with as few annotations as possible. Therefore, we can only train the network with a very small number of data in the supervised module. To address this issue, we also carefully design an unsupervised module, which can make full use of the remaining unlabeled data to aid the network training. We seamlessly integrate these modules into a unified framework for multiview clustering. The extensive experiments on benchmark data sets show that our method can outperform other stateof-the-art unsupervised and semi-supervised deep multi-view clustering methods, which well demonstrates the superiority of our method.

The main contributions of this paper are summarized as follows:

• To the best of our knowledge, we are the first to tackle the active deep clustering problem on multi-view data.
• We propose a novel active deep multi-view clustering method that can automatically select important data for annotations. Due to the carefully designed active selection module and the unsupervised module, the proposed method can learn better representations with as few annotations as possible.
• Extensive experiments on benchmark data sets demonstrate the superiority of the proposed method.

2 Active DeepMulti-viewClustering

In this section, we introduce our proposed method ADMC. First, we introduce some notations in this paper for a better understanding of our method. Given a multi-view data set X = {Xv ∈ RN×dv }Vv=1 that can be divided into C clusters, where N denotes the number of samples, V represents the number of views, dv denotes the sample dimension of the v-th view, and xvi denotes the v-th view of the i-th sample, we denote L and U as the sets of the labeled samples and unlabeled samples, respectively. Therefore, L ∪ U = ∅ and L ∩ U = X. In the beginning, all samples are unlabeled, which means L = ∅ and U = X. Since our ADMC is a batch-mode active learning method, we suppose that there are overall T batches, and the budget of each batch is K, which means in each batch, we can select at most K samples for querying the human annotations.

Figure 1 shows the architecture of our method. In our method, since there are V views, we adopt V same-structured auto-encoders [Hinton and Salakhutdinov, 2006] as the backbones to extract the latent features for each view. In the auto-encoder, we use the multi-layer perceptron (MLP) as the encoder which contains four hidden layers. The activation function used in each layer is nonlinear REctified Linear Unit (ReLU). Let Ev denote the used encoder for the v-th view, and Dv denote the decoder for the v-th view. The reconstructed ˜xvi can be represented as follows

Then, we use the following reconstruction loss lrecon for pretraining:

Our method contains four modules: a fusion module, an active selection module, a supervised module, and an unsupervised module. After obtaining the features of each view, the fusion module ensembles the features with adaptive weights. Then, we select some important samples for annotation according to the fusion features in the active selection module. Afterward, the data set is divided into the labeled set and the unlabeled set, which are used in the supervised module and the unsupervised module respectively to train the backbone networks. The details of the four modules will be introduced in the following subsections.

2.1 FusionModule

The fusion module utilizes multiple views to obtain comprehensive information for more accurate and robust results. The traditional feature fusion approaches, such as the summation or concatenation of each view, may be too simple to obtain the intrinsic representations for the complicated real-world multi-view data. In real-world applications, the views may contain various noises and the quality of each view may also differ. Therefore, the traditional methods, which directly and equally concatenate or sum them, may be inappropriate. Additionally, it is often hard to tell which view is better and which view is worse in advance.

To tackle this problem, we propose an adaptive feature fusion module, which can automatically learn the weight of each view, and then fuse all views according to the learned weights. Specifically, for a sample xi, after obtaining the representations for each view F = {f 1 i , · · · , f V i } where f v i = Ev (xvi ), we feed f v i into an MLP to obtain the weight for the v-th view wv′ i =MLP(f v i ) ∈ R1×1 . Then we use the Softmax function to get the final weight wv i for the v-th view of xi as follows.

At last, the fused representation of xi is:

2.2 Active SelectionModule

After obtaining the fused representation of each sample, we actively select several important samples for querying human annotations. We select the samples based on the following two properties:

• Representativeness. The representative data can capture the intrinsic clustering structure of data. If we obtain their annotations, we can easily characterize the clustering structure of all data.
• Diversity. If two samples are similar, we only need to annotate one of them. Therefore, to avoid the waste of the annotation budget, we should select diverse samples for querying.

To simultaneously achieve both two properties, we design a simple yet effective active selection module. Notice that the budget of each batch is K, which means in each batch, we can only query at most K samples for annotations. We first run k-means on the fused representations ˜ fi (obtained by Eq.(4)) of all samples in the unlabeled set U to partition them into K clusters. In each cluster, the closer the sample is to the cluster center, the more representative this sample is. Therefore, for each cluster, we select the sample which is the closest to the cluster center for querying. Since in each cluster, we only select one sample which lies in the center of the cluster, and according to the property of k-means algorithm, all selected samples will be far away from each other and achieve the diversity property.

Denote S as the set of K selected samples. We query humans to annotate the samples in S. Then, we move the samples in S from the unlabeled set U to the labeled set L, which means we update U = U - S and L = L ∪ S.

2.3 SupervisedModule

After obtaining the annotations of several samples, we can design a supervised module to apply the labeled data L to train the network. Given a labeled sample xi = {x1i , . . . , xVi } ∈ L whose label is yi, its high-level representation from the autoencoder is {f 1 i , . . . , f V i }, and its fused representation is ˜ fi (obtained from Eq.(4)). Notice that ˜ fi contains the consensus information among all views, and {f 1 i , . . . , f V i } contains some individual information of each view. To obtain a more comprehensive representation, we first concatenate them as [f 1 i ∥ . . . ∥f V i ∥ ˜ fi], and then feed the concatenated representation into a classification layer, which is a Fully Connected (FC) layer to obtain the final embedding ˆ yi ∈ R1×C , where C is the number of classes. Then, we feed the final embedding ˆ yi into a Softmax function to obtain the probability vector of sample xi as ˜ yi ∈ R1×C .

To train the network, we use the following cross-entropy loss between the final representation ˜ yi and the human annotation yi:

where H(·, ·) denotes the cross-entropy loss.

Besides this, the reconstruction loss lrecon is also used in the supervised module to prevent the model collapse. Therefore, the whole supervised loss lS of labeled data is as follows

where γ is a balanced parameter.

2.4 UnsupervisedModule

In our active clustering setting, most of the samples are unlabeled. It is hard to directly learn a reliable representation because we do not have enough labeled data to guide the learning. To address this issue, we design an unsupervised module to enhance the representation learning.

In the unsupervised module, since we do not use the label information, we apply contrastive learning to guide the representation learning. Contrastive learning is to learn the common features between similar samples and distinguish the differences between dissimilar samples. In single-view data, contrastive learning often needs some data augmentation techniques to generate positive and negative pairs of data for learning. In our multi-view setting, we can construct positive and negative pairs of data more naturally.

Notice that in multi-view data, each data contains multiple views and thus we can make the different views of the same sample as the positive pairs and the views of different samples as the negative pairs. Specifically, we obtain the representations of each view of xi and xj as f 1 i , · · · , f V i and f 1 j , · · · , f V j , respectively. We use (f p i , f q i ) and (f p j , f q j ) (p, q ∈ {1, 2, · · · , V }) as the positive pairs and (f p i , f q j ) as the negative pair. Then, we use the cosine similarity to measure the similarity between two representations as follows:

where ⟨·, ·⟩ represents the inner production.

Based on this similarity, we can design the following contrastive loss:

where τ denotes the temperature parameter.

Eq.(8) is a sample-level loss that does not consider the cluster structure. To characterize the cluster structure, we also introduce a center loss. The basic idea of center loss is that a clear cluster structure often needs the samples to be close to their cluster centers. To achieve this, we utilize a k-means based center loss function to characterize the cluster structure. In more detail, after obtaining the fused representation ˜ f1, · · · , ˜ fN by Eq.(4), we run k-means on them to obtain C clusters. Then, we design the following center loss lcen:

where ci denotes the center of the cluster which xi belongs to. Similar to the supervised module, we also add the reconstruction loss lrecon to the unsupervised module, which aims at preventing model collapse. As a result, the whole unsupervised loss lU is as follows

where α and β are two balanced parameters.

Algorithm 1 Active Deep Multi-view Clustering

Input: Data set X, cluster number C, the budget of each selection batch K, number of selection batches T .

Output: Clustering result

• 1: Initialize L = ∅, U = X.
• 2: Pretrain the auto-encoder by minimizing the reconstruction loss Eq.(2).
• 3: for i = 1, · · · , T do
• 4: Obtain the representations of each view by the autoencoder.
• 5: Obtain the fusion representation ˜ F by Eq.(3) and Eq.(4).
• 6: Select samples to annotate with the active selection module, and then update L and U.
• 7: Use the supervised loss Eq.(6) to train auto-encoder on L.
• 8: Use the unsupervised loss Eq.(10) to train the autoencoder on U.

9: end for

• 10: Obtain the final clustering result from the representation ˜ y1, · · · , ˜ yN .

2.5 Algorithm

At last, we feed all data into the backbones and the FC layer with Softmax trained in the supervised module to obtain the probability ˜ y1, · · · , ˜ yN. Then, given xi, we put it into the cluster with the largest probability in ˜ yi. The whole algorithm of our method is shown in Algorithm 1.

3 Experiment

In this section, we compare our method with the state-of-theart unsupervised and semi-supervised methods on benchmark data sets.

3.1 Data sets

To validate the effectiveness of ADMC, we conduct experiments on eight benchmark data sets, including BBCSport 1 , Caltech-2V [Fei-Fei et al., 2004], Caltech-5V [Fei-Fei et al., 2004], CCV [Jiang et al., 2011], Reuters1200 [Amini et al., 2009], Cora [Wen et al., 2020], Scene [Fei-Fei and Perona, 2005], and Noisy-Mnist 2 . The details of these data sets are shown in Table 1.

1 http://mlg.ucd.ie/datasets/

2 https://github.com/nineleven/NoisyMNISTDetection

3.2 Experimental Setup

We compare ADMC with eight state-of-the-art multi-view clustering algorithms including five unsupervised algorithms and three semi-supervised algorithms. The five unsupervised multi-view algorithms are CVCL [Chen et al., 2023], DealMVC [Yang et al., 2023], MFLVC [Xu et al., 2022b], DSMVC [Tang and Liu, 2022] and SDSNE [Liu et al., 2022]. The three semi-supervised multi-view algorithms are IMvGCN [Wu et al., 2023], DSRL [Wang et al., 2021], and MVAR [Tao et al., 2017].

For our proposed ADMC, the sizes of the four hidden layers in the auto-encoder are set to 500, 500, 2000, and 128, respectively. The FC in the fusion module is a four-layer MLP with RELU as an active function whose sizes are set to 128, 128, 256, and 1, respectively. The FC in the supervised module is a single layer whose size is the number of clusters. α is fixed as 10-5 , and β and γ are chosen from [10-3 , 103 ]. τ is fixed as 0.6. We use AdmW as the optimizer, The experiments contain 5 selection batches and the budget of each batch is 10 samples. For other compared methods, we use the public codes and pre-trained models provided by their authors to conduct experiments. All experiments are conducted on the PC with AMD Ryzen 7 7840H CPU, NVIDIA GeForce RTX 4060 GPU, and 16GB RAM.

We apply four wildly-used metrics to evaluate the final clustering performance, including clustering ACCuracy (ACC), Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and PURity (PUR).

3.3 Experimental Results

Table 2 shows the results of ADMC and state-of-the-art unsupervised methods. The best results are highlighted in bold. The results of ADMC in Table 2 are the results after 5 selection batches. Among all data sets on all metrics, ADMC

achieves the best results. The results of ADMC are significantly better than other methods. It shows that the supervised information is helpful in the clustering task as claimed in the Introduction.

Tables 3 and 4 show the ACC and NMI results compared with semi-supervised methods on all data sets with different numbers of annotated data. The results w.r.t. ARI and PUR are shown in the Appendix. We can see that the performance of ADMC increases with the increase of human annotations on most data sets. When compared with other semisupervised methods, our ADMC achieves better performance on all data sets at last. Moreover, we can also find that our ADMC achieves better performance with fewer annotations compared with other methods. For example, on Caltech-2V data set, we only need 30 annotations to outperform other methods with 50 annotations. It well demonstrates the effectiveness of our active clustering method.

3.4 Ablation Study

To further verify the effectiveness of our active selection module, we design an ablation study by comparing ADMC with its unsupervised and semi-supervised versions. In more detail, we denote ADMC-U as the unsupervised version without any annotations, which means it removes the active selection module and the supervised module. We denote ADMCR as the semi-supervised version, which means in the active selection module, we randomly select K samples for querying annotations.

Table 5 shows the ACC and NMI results of our method and these two degenerated versions. The results w.r.t. ARI and PUR are shown in the Appendix. We can find that ADMC-U is much worse than ADMC-R and ADMC, even though we only have very few annotations in ADMC-R and ADMC. It shows that the supervised information is quite important for the clustering. Moreover, compared with ADMC-R, ADMC often achieves better performance, even when there are only 10 annotations. It shows that our designed active selection module performs much better than the random selection no matter how many samples are annotated, demonstrating the effectiveness of the active selection module.

3.5 Hyper-parameter Study

To investigate the sensitivity of two hyper-parameters β and γ in ADMC, we show the ACC and NMI results with different values of β and γ in the range [10-3 , 103 ]. Figure 2 shows the results after 50 annotations on Cora and Caltech-2v data sets. The results on other data sets are similar. From Figure 2, we can see that the performance is stable in a relatively wide range. It means that we can easily select hyper-parameters. For example, we can select them in the range [10-3 , 10-1 ] to easily achieve a relatively good performance.

3.6 Conclusion

In this paper, we proposed a novel active deep multi-view clustering algorithm that can automatically select important samples for querying human annotations to guide the clustering. In this method, we provided an adaptive multi-view fusion module to ensemble multiple views by learning appropriate weights. Then, we designed an active selection module to identify representative and diverse samples for human annotations in a simple yet effective way. We used the annotations to train the network in a supervised module and we also carefully designed an unsupervised module with contrastive learning to enhance the multi-view representation learning. Experimental results on eight widely-used multi-view data sets by comparing it with state-of-the-art supervised and semi-supervised multi-view clustering methods show the effectiveness and superiority of our proposed ADMC. We also conducted the ablation study to compare it with its unsupervised and semi-supervised versions, which shows the effectiveness of our designed active selection module, and well demonstrates our motivation of active clustering.

In the future, we will further study other selection criteria, such as uncertainty and informative, and design a new active selection module for multi-view clustering.

Acknowledgments

This paper is supported by the National Natural Science Foundation of China grants 62176001, and the Natural Science Project of Anhui Provincial Education Department under grant 2023AH030004.