Self-supervised Weighted Information Bottleneck for Multi-view Clustering

1 Introduction

Multi-view clustering (MVC) [Bickel and Scheffer, 2004] is a hot topic in machine learning and computer vision community, and it focuses on investigating and utilizing the close relationships among different views for learning the final consistent data cluster structure. And it has been successfully applied into many computer vision and pattern recognition fields, such as medical image analysis and image retrieval. Of all the MVC methods, short for MVCs, weighted MVC is one of the effective methods working by learning and applying the view weight or importance on multiple views for fully exploring the complementary information across views.

∗Corresponding Author

Recently, many weighted MVCs have been proposed and a few typical ones are listed in the followings. In the early period, a k-means based multi-view clustering method [Cai et al., 2013] is designed with automatic weight learning, and also the model robustness is ensured. Additionally, Wang et al. [Wang et al., 2020] proposed a multi-view graph-based method for data clustering problem, which produces an effective unified fused graph matrix with rank constrained parameter-free weight learning strategy, and further this learned fused matrix benefits the graph matrix learning of each view. More recently, Xia et al. [Xia et al., 2023] introduces an effective tensorized bipartite graph learning method for MVC problem, worked by utilizing the between-view similarity and within-view similarity. It also incorporates the anchor point learning mechanism for efficient model learning. However, most existing weighted MVCs only consider the quality of each view, ignoring the vital role of pseudo label self-supervision information in weight learning.

In this work, we propose a novel self-supervised weighted information bottleneck (SWIB) method for solving the multiview clustering problem, as shown in Figure 1. It combines the weighted information from different views based on information bottleneck theory, and the view weight learning mechanism is newly designed by simultaneously taking into accounting both the quality of view-contained information and the self-supervised information on the data partition of each view. Specifically, the view-contained part focuses on the quality of the relevant feature information of each view. The self-supervised part fully employs the pseudo-labels in each iteration to guide the learning of view weights. Both of the two parts are combined together with a proper balance between them to learn an effective view weights reflecting the importance of each view. The challenges of incorporating the self-supervised information into weight learning lie in two aspects. First, data samples from different views are always heterogeneous, thus directly using the self-supervised information for computing the importance of each view is very difficult. Second, lots of existing methods still need many parameters to regularize the weight distribution during the weight learning process, which makes the parameter tuning difficult. Thus, incorporating the self-supervised information into weight learning without bringing or only bringing one additional parameter is still challenging. Experimental re-

"Pseudo-label"Self-supervision

sults on various multi-view datasets, i.e., multi-view 20NGs text dataset, multi-feature COIL20 image dataset, multi-angle WVU video dataset, and multi-modal text-image PASCAL dataset as well as large-scale datasets (IAPR and ALOI) show the superiority of the SWIB method.

• To our knowledge, this is the first work incorporating the self-supervised learning into weighted multi-view clustering.
• A new view weight learning mechanism is designed by integrating both view-contained information and selfsupervised information within and across views respectively.
• A novel SWIB method is proposed to comprehensively quantify the complementary information among views to enhance the MVC performance with only one parameter.
• Experimental results on various multi-view text, multifeature image, multi-angle video, and multi-modal textimage dataset as well as large-scale datasets show the superiority of the SWIB method.

2 The Proposed SWIBMethod

2.1 Prior Knowledge: Mutual Information and Information Bottleneck Theory

Information bottleneck (IB) [Tishby et al., 1999] is an information-theoretic principle, and has been successfully applied to many real-world applications, such as unsupervised image segmentation [Bardera et al., 2009] and classification [Lou et al., 2013]. A more comprehensive survey on IB can be reached here [Hu et al., 2024].

Recently, many multi-view learning methods via IB [Hu et al., 2022b; Lou et al., 2013; Hu et al., 2022a; Zhang et al., 2023] have been proposed and exhibited promising performance. Motivated by this, in this paper we tackle the multiview clustering problem based on the IB theory.

IB regards the data clustering process as data compression of the input variable X into a compressed variable T, and meanwhile it preserves the relevant information about the feature variable Y as much as possible. The IB function is defined by

where I(T;X) denotes the mutual information between T and X, focusing on the data compactness quantification, I(T; Y ) denotes the mutual information between compact variable T and feature variable Y , focusing on the useful information preservation, β ∈ (0,+∞) is a balance parameter trading off both of them, and p(t|x) denotes the data assignment probability.

To simplify the optimization, we give the max-version of the IB function

which is optimized by

where p(t) = ∑x,y p(x, y, t) = ∑x p(x)p(t|x), p(y|t) = 1 p(t) ∑x p(x, y)p(t|x), Z(x, β) is a normalization function and DKL is the Kullback-Leibler divergence [Cover and Thomas, 2006].

Note that we define the frequently-used concept of mutual information (MI) [Cover and Thomas, 2006] between two variables as

where p(x, y) denotes the joint probability distribution, p(x) = ∑y p(x, y) and p(y) = ∑x p(x, y).

2.2 ProblemFormulation

Assume that we have data samples X = {xi}ni=1 , denoted by a random variable X. These samples contain m different views, leading to m variables {Y k }mk=1 , where Y k ∈ Rdk characterizes the samples from one view with dk feature dimensionality. Based on that, we have m joint distributions {p(X, Y k )mk=1 } reached by the classical Bag-of-Words model [Fei-Fei and Perona, 2005]. Our newly-designed SWIB method aims to explore and exploit the weighted complementary information among different views to obtain a good compact representations p(t|x) of X to T. Note that we use c to indicate the number of clusters in each view.

2.3 The Overall Objective Function

In this part, motivated from the popular self-supervised learning, we propose a novel self-supervised weighted information bottleneck method for addressing the multi-view clustering problem, where the overall objective function is defined as follows

where wi denotes the weight value of the i-th view, β denotes the balance parameter trading off the information compression and preservation. Generally, the compact variable is a highly compressed from the input data, leading to |T ∗ | ≪ |X|. Thus, we usually set the parameter β as +∞ in the practical usage [Hu et al., 2021]. And the specific view weight learning mechanism is given in the following parts.

2.4 ViewWeight LearningMechanism

In this part, we design a new view weight learning mechanism for the above SWIB method, where the weight learning mainly contains two parts, including view-contained and selfsupervised parts. The whole weight learning is defined as

where W iVC denotes the view-contained part, W iSS denotes the self-supervised part, and λ is the trade-off parameter balancing the two parts.

View-contained Part

In view-contained part, the view quality is evaluated by mutual information between the clustering partition variable T ∗ and relevant feature variable Y i for weight learning, as shown in Figure 2. The detailed formulation is given as

where H(T ∗ ) and H(T i ) are the entropy values of T ∗ and Y i , and H(T ∗ ; Y i ) is the cross entropy of both variables.

Based on the above, we have the view-contained part for viewweight learning as

Self-supervised Part

In self-supervised part, we incorporating the pseudo-labels as self-supervised signal for guiding the weight learning process, as shown in Figure 3. It works by using the metric of MI between the data partition variable T ∗ and the partition of each view T i , shown as follows.

where I(T ∗ ; T i ) indicates the MI between the final data partition T ∗ and the data partition of the i-th view, H(T ∗ ) and H(T i ) are their entropy values.

Based on the above, we have the self-supervised part for viewweight learning as

2.5 Analysis on the Model

Observing from the proposed model, there are mainly two advantages compared with existing weighted multi-view clustering methods.

• Unlike existing models, the proposed method incorporates the self-supervised learning into view weight learning mechanism. To our knowledge, this is the first work doing so.
• Our weighted multi-view clustering method is one of the few models containing only one parameter in the objective function, which benefits for its usage in practical applications.

Algorithm 1 Algorithm for Optimizing the SWIB

• 1: Input: m joint probability distributions {p(X, Y i )}mi=1 , cluster number c, and the parameter λ. 2: Output: Final partition T ∗ . 3: Local Clustering: 4: {T 1 , T 2 , . . . , T m} ← Applying information bottleneck on X into c clusters on different views; 5: repeat 6: i ← 1; 7: while i ≤mdo 8: for all x ∈ X do 9: Draw x from its original cluster; 10: Reassign x into clusters from the current i-th view, and calculate the merger cost; 11: Merge x into a new cluster with minimal merger cost; 12: end for 13: i ← i + 1; 14: end while
• 15: Update the viewweight values with the Eq. (6);
• 16: until Data partition unchanged or a fixed number of iterations finished

2.6 Optimization

In this part, we adopt the draw-and-merge sequential optimization method for solving the proposed method. It is a kmeans like method by sequentially drawing each data sample from its original cluster and iteratively merging to a new cluster with minimal merger cost until the final clustering results unchanged or a fix number of iterations. The readers may refer to the related works [Lou et al., 2013] for details, and the algorithm framework is given in Algorithm 1.

3 Experimental Validation

3.1 Datasets

We adopt multi-view text, multi-feature image, multi-angle video, and multi-modal text-image dataset as well as largescale datasets to show the superiority of the SWIB method and the details are shown in Table 1.

Multi-viewText. 20NGs dataset 1 has 500 documents from the popular 20 Newsgroups dataset. Every document is processed by three various methods, corresponding to each view of the dataset.

Multi-feature Image. COIL20 dataset 2 contains images of 20 different objects on a motorized turntable. These images are captured by a fixed camera with a turntable rotated in 360◦ at 5◦ change to make different object posture. We use 3 kinds of features, i.e., shape, color and texture representation, namely SIFT [Lowe, 2004], Color Attention [Khan et al., 2009], and TPLBP [Wolf et al., 2008] respectively to characterize each view.

1 http://lig-membres.imag.fr/grimal/data.html

2 https://www.cs.columbia.edu/CAVE/software/softlib/coil-20. php

Multi-angle Video. WVU dataset 3 contains human action videos of 10 types captured from different angles. The nonadjacent views 1, 3, 5 and 7 are used for experimental validation, where the setting is quite similar to real-world scenario, e.g., road video surveillance. We adopt space-time interest points technique [Laptev, 2005] for feature extraction with 1000 dimensionality.

Multi-modal Text-Image. PASCAL dataset 4 has 20 clusters with 1000 data samples, and they are captured from image and text aspect, where the images are extracted with SIFT representation and the documents are extracted with BoW model.

3.2 ComparedMethods

We have the following compared methods classified into two types.

Classical Single/All-view clustering. k-Means (KM) and Information Bottleneck (IB) [Tishby et al., 1999], and allview version of them, namely, All-view KM and IB, short for AVKM and AVIB respectively, where multiple views are combined and the two methods are then applied.

State-of-the-art MVCs. We select 10 state-of-the-art MVCs for experimental comparison, in which RMKMC, MfIB, DEKM, MLAN, GMC and SMVSC are all based on learning weights.

1. MVIB [Gao et al., 2007]: A typical multi-view IB method proposed in early years by imposing compatible constraints on individual clusterings.
2. RMKMC [Cai et al., 2013]: Amulti-view k-means clustering method with robustness guaranteed.
3. MfIB [Lou et al., 2013]: A multi-feature IB method based on weight learning designed for unsupervised image clustering.
4. DEKM [Xu et al., 2016]: A discriminatively embedded multi-view k-means method for addressing clustering problem.
5. MLAN [Nie et al., 2018]: A weighted multi-view learning method without penalty parameters for alleviating outliers and noise.
6. GMC [Wang et al., 2020]: A multi-view graph clustering method by learning a fused unified matrix regularized by a rank constraint.
7. SMVSC [Sun et al., 2021]: A scalable MVC method with anchor learning for large-scale data clustering.
8. FPMVS-CAG [Wang et al., 2022]: A fast MVC method by simultaneous graph learning and consensus anchor learning for large-scale setting.
9. OMSC [Chen et al., 2022]: An efficient orthogonal MVC method with joint learning of representation and clustering.
10. TBGL [Xia et al., 2023]: A tensorized bipartite graph learning for MVC by utilizing the between-view similarity and within-view similarity.

3 https://community.wvu.edu/vkkulathumani/wvu-action.html

4 https://aclanthology.org/W10-0721.pdf

method. The lower values of both reveal poorer clustering performance. For all the methods, we take 10 runs for all the multi-view text, image, video and text-image datasets, and then reveal them in Table 2 with average deviation, where λ = 0.2 for 20NGs, WVU and PASCAL datasets, and λ = 0.4 for COIL20 dataset.

From the table, we have the following conclusions. Our SWIB method consistently outperform the clustering results reached by single/all-view KM and IB methods on all the datasets with a significant improvement, e.g., WVU and PASCAL dataset. This clearly reflects the advantages of the proposed method over single/all-view clustering methods. However, on some datasets, the view-concatenated all-view methods can not always beat the results obtained by their single-

3.3 Settings of Experiments

For single-view methods, the best clustering Accuracy (Acc) and Normalized Mutual Information (NMI) values on multiple views are shown. For state-of-the-art MVCs, we refer to the settings of parameter values in their papers and give the optimal performance with respect to the ideal parameter setting. For our SWIB method, we set β as +∞ and search the parameter λ from the range {0.1, 0.2, 0.4, 0.6, 0.8}. The detailed parameter selection is given in the following subsection.

3.4 Analysis onMulti-viewClustering Results

The frequently-used clustering metrics of Acc and NMI are used for verifying the effectiveness of the proposed SWIB

view version methods. To some extent, this suggests that we have to resort to the paradigm of multi-view clustering. Actually, from the results, the proposed SWIB reaches much better results when compared with state-of-the-art MVCs methods, e.g., RMKMC, GMC, SMVSC, and OMSC method. This probably benefits from the new design of view weight learning mechanism with combination of view-contained information from individual views and self-supervised information from pseudo-labels of each iteration.

3.5 Parameter Selection

For our SWIB method, we set β as +∞ and search the parameter λ from the range {0.1, 0.2, 0.4, 0.6, 0.8}. Figure 4 shows the clustering Acc and NMI results corresponding to each parameter setting. From this figure, we find that the clustering results maintain almost stable on the parameter search range, and suggests that it is not difficult to select a better parameter in usage and also exhibits possible practicability of SWIB method into real-world scenario. Note that we set λ = 0.2 for 20NGs, WVU and PASCAL datasets, and λ = 0.4 for COIL20 dataset.

3.6 Visualization Validation

For further illustrating the clustering results of the SWIB method, we select two datasets of COIL20 and WVU for clustering visualization with the popular method of t-SNE. The results are revealed in Figure 5, and we obtain that a majority of these views suggest a very compact cluster structure and a margin between different clusters, e.g., the view 2 and 3 of COIL20 dataset. This phenomenon further reflects that our SWIB method is capable of obtaining satisfactory clustering results with the new effective view weight learning mechanism.

3.7 Effectiveness on Large Datasets

To further verify the scalability of the proposed SWIB method, we conduct additional experiments on two largescale multi-view dataset, i.e., multi-modal text-image IAPR dataset and mult-feature ALOI image dataset. IAPR dataset contains 7855 samples of 6 clusters and we extract SIFT image representation and BoW text representation for view description. Additionally, ALOI image dataset has 11025 image samples of 100 clusters, and we extract color similarity, RGB and Haralick feature representations as different views. We show the clustering results of them with compared methods and SWIB method on Figure 6 and Figure 7, where λ = 0.2 for both of the large multi-view datasets. It is seen that the proposed SWIB method consistently works better than the classical single/all-view clustering methods and most state-of-the-art MVC methods (except the second best Acc on ALOI dataset where MLAN obtains the best Acc) on both of the large datasets. These two figures also reveal the scalability of the proposed SWIB and the potential application into practical multi-view environments.

4 Conclusion

In this paper, a novel self-supervised weighted information bottleneck (SWIB) method is designed for addressing the challenging multi-view clustering problem. We combine the weighted information from different views based on information bottleneck theory, and then introduce a new view weight learning mechanism by simultaneously taking into accounting both the quality of view-contained information and the self-supervised information on the data partition of each view. Experimental results on multi-view text, image, video, and multi-modal text-image dataset as well as large-scale multi-view datasets have demonstrated its effectiveness. In the future, we will focus more on the deep version of the weighted MVC method [Huang et al., 2023; Cui et al., 2023], reliable multi-view learning [Xu et al., 2024], incomplete multi-view clustering [Wen et al., 2023; Wen et al., 2021] and extend to more practical applications.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 62206254), the China Postdoctoral Science Foundation (No. 2023M743186), and the National Natural Science Foundation of China (No. 62176239).

Contribution Statement

All authors make contributions for the paper's methodology and experiments. Zhengzheng Lou and Chaoyang Zhang contributed equally to this work and should be considered co-first authors. Shizhe Hu is the corresponding author.