MASTER: A Multi-granularity Invariant Structure Clustering Scheme for Multi-view Clustering

1 Introduction

Multi-view clustering (MVC), as a fundamental task in machine learning, is attracting more and more prominent attention across various domains, such as image recognition and text classification, in the last few years [Wang et al., 2024a; Zhou et al., 2024; Jin et al., 2023; Liu et al., 2024a; Yu et al., 2023]. It can be interpreted as an evolution of vanilla clustering for samples collected with multiple views.

∗Corresponding authors: Qingchen Zhang and Peng Li.

That is, MVC is capable of distilling complementary structure information of samples from multiple views/modalities to learn partition patterns in an unsupervised manner, such that samples within the same group are more similar to each other than samples from different groups [Chen et al., 2023b; Cui et al., 2024; Zou et al., 2024; Wang et al., 2024b].

With an encouraging renaissance of deep learning, some edge-cutting deep multi-view clustering methods have made significant progress in mining intrinsic patterns of data [Ren et al., 2024; Gao et al., 2024]. The deep multi-view clustering methods can be roughly divided into two groups, i.e., noncontrastive learning methods and contrastive learning methods, in accordance with self-learning mechanisms. The former draw on some vanilla clustering strategies, such as kmeans and subspace clustering, as self-learning mechanisms to train neural networks, and they capture clustering-friendly complementary information of data via minimizing structure divergences between similar samples and maximizing structure divergences between dissimilar samples to model multi-view data partition. For example, [Xu et al., 2023] introduces a self-supervised multi-view embedded clustering scheme to constrain structure consistencies between viewconsensus features and view-specific features for MVC. The latter construct inter-view and intra-view contrastive learning mechanisms between data in multiple views to train their own multi-view fusion network, and they learn sample-invariant complementary information of data structure via pulling positive pairs of samples close and pushing negative samples apart to support the partition of multi-view data. For instance, [Yan et al., 2023] introduces a structure-guided contrastive learning clustering scheme to weight similarities between samples in extracting invariant complementary information.

Although previous methods improve clustering performance by a significant margin, there still exist two challenges in mining intrinsic patterns of multi-view data. First, most of them perform self-learning mechanisms in a single space to extract data structure from multiple views for pattern mining, ignoring the fruitful multi-granularity structure information hidden in different-level feature spaces. Second, they learn generalized representations of samples in each view via the reconstruction constraint to preserve consistent and complementary information, which causes degradations of the discriminative ability of view-specific representations. When confronted with multi-view data of complicated distributions, they may not produce desired clustering patterns.

To address the challenges, a Multi-granularity invAriant Structure clusTERing scheme (MASTER) is proposed to define a bottom-up clustering process, which captures fruitful structure information hidden in different-level feature space in a cascaded manner for multi-view data partition. In detail, MASTER consists of a sample-oriented representation learning strategy (SRL), a reliable neighborhood-driven diffusion strategy (RND), and a semantics-invariant cluster strategy (SIC). SRL leverages self-learning reconstructions with information-theoretic overclustering in encodingdecoding paradigms. It maximizes cross-view mutual information to perform sample clustering that interprets each sample as a single cluster in the low-level feature space, capturing invariant structure in the sample granularity for the enhancement of the discriminative ability. RND conducts a cross-view neighborhood distillation mechanism in the highlevel feature space via imitating data diffusion of the clustering process. It introduces the cluster assignment information to dynamically refine neighbor relationships, learning invariant local structure in the neighborhood granularity. SIC defines dual divergences between assignment predictions of multi-view samples with simplexes of the semantics space. It leverages the space geometry to capture invariant global structure in the category granularity, maximizing inter-cluster separation and intra-cluster compactness for intrinsic pattern mining. Finally, extensive experiments are carried out on 8 benchmark datasets in comparison with 11 methods, and the results show MASTER achieves state-of-the-art performance.

Thus, the key contributions of the article can be concluded in the three-fold aspects.

• A multi-granularity invariant structure clustering scheme is proposed to define a bottom-up clustering process, which constructs a consistent hierarchy structure from the perspective of sample, neighborhood, and category to extract fruitful complementary and consistent information for MVC.
• MASTER imitates data diffusion in the clustering process to implement a cross-view neighborhood distillation mechanism, which can dynamically capture reliable affinity relationships in the neighborhood granularity to facilitate the extraction of invariant local structure from complementary information.
• Plenty of experiments are conducted on 8 real-world datasets and the results show MASTER achieves the state-of-the-art performance on multi-view data clustering in comparison with 11 methods.

2 RelatedWork

2.1 Non-contrastive LearningMethods

Non-contrastive learning methods introduce various selflearning mechanisms along with novel multi-view neural architectures, which measure structure divergences between samples, to extract clustering-friendly complementary information for clustering. Those methods can be divided into two-stage and one-stage methods.

The former train multi-view architectures with the aid of innovative self-learning mechanisms to learn clusteringfriendly fusion representations, which perform additional clustering algorithms to divide multi-view samples [Zhou et al., 2024]. For instance, [Li et al., 2023] introduces an adversarial encoding-decoding architecture via defining an inter-view consistent cycle and an intra-view consistent cycle of data, which endows fusion representations of multiview samples with clustering structure. [Dong et al., 2023] introduces a multi-view encoder-decoder architecture for bipartite graphs, which aligns complementary information in each view to produce informative fusion representations of multi-view samples. The latter design effective self-learning mechanisms to train multi-view fusion neural architectures to produce clustering results in an end-to-end manner, which can cooperate structure discovery of data with complementary information fusion between views to promote sample partition [Xu et al., 2024]. For example, [Zhou and Shen, 2020] designs a multi-view adversarial-attention fusion architecture, which leverages the geometry induced by simplexes of semantic space to guide divisions of multi-view data. [Gao et al., 2024] proposes an adaptive multi-view semantics-invariant fusion network via constraining semantics alignments between views with samples, which further designs clustering as a Markov decision process of data partitions in an end-to-end manner.

2.2 Contrastive LearningMethods

Contrastive learning methods define cross-view contrastive learning mechanisms along with novel multi-view fusion architectures, which maximize similarities between positive samples and dissimilarities between negative samples, to learn view-invariant structure from views. Those methods can be also divided into two-stage and one-stage methods.

The former usually construct cross-view contrastive learning to model fusion features with view-invariant information of samples, and perform additional clustering constraints for pattern mining [Lu et al., 2024]. For example, [Xu et al., 2022] proposes a multi-level feature learning architecture via stacking contrastive learning networks on the middle layer of view-specific encoding-decoding networks, where crossview consistencies of high-level features and semantic labels are constrained in a parallel manner for MVC. [Luo et al., 2024] introduces a view-agnostic multi-view fusion contrastive architecture via interpreting the feature-dimensional concatenation between normalized data in each view as datalevel fusion of samples, which conducts the vanilla contrastive learning on samples with the help of the manual noising/deleting data of samples in some views. The latter leverage cross-view contrastive learning to capture the distributions of data partition, which benefits from explorations of view-invariant information of samples in the semantic space [Zhang et al., 2024b]. For instance, [Chen et al., 2023a] proposes a cluster-level multi-view contrastive learning clustering method, which leverages view-invariant information via pulling positive pairs of clustering centroids close and pushing negative pairs of clustering centroids apart. [Yang et al., 2023] proposes a dual contrastive learning paradigm with an attention-weighted multi-view fusion encoder-decoder network, where pseudo-label graphs are used to calibrate global and local cross-view similarities between samples in end-toend clustering, respectively.

3 Method

A multi-granularity invariant structure clustering scheme is proposed to define a bottom-up clustering process. It designs a sample-oriented representation learning strategy, a reliable neighborhood-driven diffusion strategy, and a semanticsinvariant cluster strategy, to construct a consistent hierarchy structure from the perspective of sample, neighborhood, and category. The architecture of MASTER is shown in Figure 1.

3.1 The Sample-oriented Representation Learning Strategy

The sample-oriented representation learning strategy (SRL) leverages the invariant structure of each multi-view sample to enhance the discriminative ability of view-specific representations, which captures more complementary and consistent information from the low-level feature space in the sample granularity. To accomplish this, SRL constructs self-learning reconstructions within encoding-decoding paradigms in each view to model intrinsic view-specific representations from the data manifold. Then, SRL defines an information-theoretic overclustering that interprets each sample as an independent cluster, which endows view-specific representations with the discriminative invariant sample structure.

Specifically, given a multi-view dataset X = {xvi }N,V i=1,v=1 with C classes, which includes N samples with V views, SRL constructs an encoder-decoder pair that perform the self-learning reconstruction on data in each view, to capture view-specific representations zvi , i.e., zvi = Encv (xvi ) and ˜xvi = Decv (zvi ), which extracts generalized complementary and consistent information from data manifold. Thus, SRL constrains the self-learning reconstruction on data in the lowlevel feature space via

where || · || stands for the Euclidean norm and ˜xvi is the i-th reconstruction sample in the v-th view. Eq. (1) can utilize the data manifold preservation to distill generalized complementary structure information for subsequent data partition.

Then, SRL maximizes the cross-view mutual information between view-specific representations to implement the information-theoretic overclustering, which endows viewspecific representations with discriminative structure in the sample granularity. In detail, given view-specific representations zvi and zli in two views, SRL performs the informationtheoretic overclustering via maximizing I(zvi ; zli ), which is further transformed into the sample-level clustering on the basis of the data processing inequality, as follows:

where I(; ) represents the mutual information. sli denotes the probability vector of overclustering, and each element svi [m] = exp(zvi [m])/∑ Mm′=1 exp(zvi [m′ ])(M > C). Then, SRL leverages the uniqueness concept of samples that each multi-view sample is a single cluster to maximize mutual information I(svi ; sli ).

where E stands for the expectation operator. S denotes the set of svi . p(svj |sli )/p(svj ) measures the influence of knowing sli on svj , j ̸= i. SRL constrains the information-theoretic overclustering on the multi-view dataset as

Thus, SRL leverages the structural information of each sample to enhance the discriminative ability of generalized representations via

where α is a trade-off parameter.

3.2 The Reliable Neighborhood-driven Diffusion Strategy

The reliable neighborhood-driven diffusion strategy (RND) focuses on extracting cross-view invariant structure from complementary information in the neighborhood granularity to promote local consistency of affinity relationship in the high-level feature space for multi-view sample partition. To achieve this, RND defines a cross-view neighborhood distillation mechanism that imitates inherent paradigms of data diffusion in the clustering process, i.e., approaching, leaving, and staying, to dynamically learn an invariant structureenhanced neighbor relationship with the help of the cluster assignment information. In detail, RND models the approaching paradigm for the in-neighborhood multi-view samples with the same cluster assignment, to enhance the intraneighborhood consistency of data structure. RND models the leaving paradigm for the out-of-neighborhood multi-view samples with different cluster assignments, to strengthen the inter-neighborhood divergences of data structure. Meanwhile, it models the staying paradigm for the multi-view samples that do not satisfy the above two conditions, to ensure the robustness of the data structure. RND learns reliable neighbor relationships via modeling three diffusion paradigms in the high-level feature space to enhance the exploration of invariant local structure.

Specifically, given the view-specific representations Zv = {zvi }Ni=1 of the v-th view in the low-level feature space, RND conducts a feature mapping function ϑv to generate the highlevel representations Hv = {hvi }Ni=1 , i.e., hvi = ϑv (zvi ). Subsequently, RND introduces the cross-view indicator matrix Yvl ∈ RN×N of cluster assignments to implement the neighborhood distillation mechanism. The mechanism adaptively refines neighbor relationships to generate a diffusion mask matrix Tvl ∈ RN×N between the v-th view and the l-th view:

where ψl (hvi ) denotes theK-nearest neighbor set of hvi in the l-th view. yvl ij = 1 indicates that two samples are assigned to the same cluster (yv i = ylj ), vice versa. In Eq. (6), tvl ij = 1 indicates that in-neighborhood samples with the same cluster assignment should approach each other. tvl ij = 0 indicates that the out-of-neighborhood samples with different cluster assignments should leave each other. tvl ij = -1 indicates that the sample does not satisfy the above two conditions should stay stationary.

Thus, RND utilizes the neighborhood distillation mechanism induced by the data diffusion, to dynamically maximize intra-neighborhood similarities and minimize interneighborhood similarities in the high-level feature space, which is formulated as follows:

where H denotes the cross entropy function.

3.3 The Semantics-invariant Cluster Strategy

The semantics-invariant cluster strategy (SIC) aims to capture the invariant global structure of the semantics space from complementary and consistent information in the category granularity, which ensures inter-cluster separation and intracluster compactness for intrinsic pattern mining. To this end, SIC defines dual divergences, i.e., the simplex constraint and orthogonal constraints, between the assignment predictions of multi-view samples with the simplexes of the semantics space in two dimensions, which leverages the geometry of the semantics space to guide explorations of invariant global structure in the category granularity.

To be specific, given the view-specific representations Zv = {zvi }Ni=1 of multi-view data in the v-th view, SIC defines a semantic mapping function ϕv from view-specific representations of samples to the assignment predictions Pv = {pvi }Ni=1 , i.e., pvi = ϕv (zvi ), which can leverage the discriminative complementary and consistent information of each view to capture invariant global structure of multi-view data.

Algorithm 1 MASTER

Input: The multi-view dataset X = {xvi }N,V i=1,v=1

Parameter: Learnable parameters in encoders Encv , decoders Decv , feature mapping functions ϑv , and semantic mapping functions ϕv of multiple views

Output: Clustering partition P

• 1: while not converged do
• 2: Sample structure learning:
• 3: Extract view-specific representations by encoderdecoder pairs zvi = Encv (xvi ) and ˜xvi = Decv (zvi ).
• 4: Compute probability vectors of overclustering by svi [m] = exp(zvi [m])/∑ Mm′=1 exp(zvi [m′ ]).
• 5: Neighborhood structure learning:
• 6: Generate high-level representations hvi = ϑv (zvi ).
• 7: Generate diffusion mask matrices Tvl by Eq. (6).
• 8: Global structure learning:
• 9: Obtain assignment predictions pvi = ϕv (zvi ).
• 10: Update learnable parameters by minimizing Eq. (5), Eq. (7), and Eq. (10).
• 11: end while

Afterwards, SIC interprets the C simplexes in the Cdimensional semantics space as the cluster centroids, and it minimizes the divergence between samples and the corresponding cluster centroid via the cross entropy, which can push multi-view samples within the same cluster towards the same simplex to capture clustering-friendly invariant structure of multi-view data. SIC computes the loss induced by the centroids with C dimensions as follows:

where cik is the k-th element of the i-th simplex of the semantics space, and pik denotes the k-th element in the average assignment prediction vector of the i-th multi-view sample. In such a manner, SIC pushes samples within the same cluster towards the same simplex and samples of the different clusters apart, maximizing the inter-cluster separation and intra-cluster compactness.

Then, SIC recasts each column of the assignment prediction matrix in each view as the representations of cluster centroids in the sample dimension, and it forces inter-view and intra-view orthogonal constraints between the centroids, which can further maximize cross-view consistencies to capture the invariant global structure across views. SIC computes the loss induced by the centroids with N dimensions with the simplex via contrastive learning as follows:

where I denotes the identity matrix with C × C dimensions.

Eq. (9) constrains inter-view consistencies and intra-view compactness of multi-view sample assignments. Thus, SIC captures the invariant structure of the semantics space in the category granularity via:

The overall loss of MASTER is shown in Eq. (11). Algorithm 1 lists the detailed steps of MASTER.

where β and γ are trade-off parameters.

4 Experiment

4.1 Experiment Settings

Benchmark Datasets: Plenty of experiments are conducted on 8 real-world datasets to validate the performance of MASTER. In detail, MNIST-USPS is a handwritten digit dataset that contains 5000 bi-view samples within 10 classes. BDGP is a Drosophila embryo dataset that contains 2500 bi-view samples within 5 classes. CCV is a video dataset with 6773 tri-view samples belonging to 20 classes. Fashion is an image dataset about products, which contains 10000 tri-view samples within 10 classes. Caltech-2V, Caltech-3V, Caltech-4V, and Caltech-5V are created from the Caltech dataset that consists of RGB images with multiple views. The statistics of the 8 datasets are listed in Table 1.

Comparison Methods: 11 methods with state-of-the-art performance are used to verify the superiority of MASTER, which are composed of non-contrastive learning methods and contrastive learning methods. The former consist of SDMVC [Xu et al., 2023], MVCAN [Xu et al., 2024], DMAC-SI [Gao et al., 2024], CAMVC [Zhang et al., 2024a], DSMVAGC [Wang et al., 2024a], and AEVC [Liu et al., 2024b]. The latter include DealMVC [Yang et al., 2023], GCFAgg [Yan et al., 2023], SCM [Luo et al., 2024], CSOT [Zhang et al., 2024b], and DIVIDE [Lu et al., 2024]. In the experiments, the comparison methods are implemented according to settings in the original papers, and the grid search is utilized on the trade-off parameters suggested by the authors to guarantee the best performance of the comparison methods.

Implementation Details: MASTER consists of V encoderdecoder pairs, a feature mapping network, and a semantic mapping network, which are implemented via fullyconnected layers with the following architectures: D-500500-2000-256, 256-2000-500-500-D, 256-128, and 256-C, respectively. D and C represent the dimensionality of the original data and semantic features, respectively. MASTER optimizes the overall networks via the Adam optimizer with a learning rate of 0.0003 across all datasets. In the optimization, the number of training epochs is set to 350. The batch size is set to 256. The values of α, β, and K are determined via the ablation experiments with the grid search strategy on each dataset. γ is set to 1 on all the datasets.

Evaluation Metrics: Three standard clustering metrics, i.e., accuracy (ACC), normalized mutual information (NMI), and purity (PUR) are utilized to verify the superior performance of MASTER, and the larger value of the metrics indicates the better clustering performance. In the experiments, the results shown in Tables ??, Table 3, and Figure 2, are the average of 5 runs for all the metrics to guarantee a fair comparison.

4.2 Clustering Performance

Table ?? demonstrates the numerical results produced on the 8 datasets. Those results show that MASTER outperforms all the comparison methods by a significant margin in terms of the three evaluation metrics, which can be concluded in three observations below. ① BDGP, MNIST-USPS, and FashionMV: The clustering results are very high and close to the maximum value 1, meaning that there is not much room for improvement on the three datasets. That is, the multi-view samples, which are assigned into incorrect clusters by the secondbest method, are very hard to distinguish in pattern mining. In such a condition, MASTER can still assign more samples into the correct clusters and achieve better performance in terms of all the evaluation metrics on the three datasets. ② CCV and Caltech-2V: The clustering results are relatively low, meaning that it is hard to recognize some multi-view samples. That is, the multi-view samples that are often assigned to incorrect clusters by the second best method, are very hard to distinguish in pattern mining. In such a condition, MASTER can still assign more samples into the correct clusters and achieve better performance in terms of all the evaluation metrics on the three datasets. ③ Caltech-3V, Caltech-4V, and Caltech5V: The clustering results are of significant gains, indicating that MASTER is competent to capture intrinsic patterns from the complementary and consistent information of multi-view data. In detail, MASTER outperforms all the comparison methods more and more significantly as the number of views increases. Especially, MASTER outperforms the second-best method by 2.10%, 1.78%, and 2.17% in terms of ACC, NMI, and PUA on Caltech-5V.

4.3 The Ablation and Hyper-parameter Analyses

Ablation Results: Table 3 demonstrates the ablation results on MNIST-USPS and Caltech-2V, which is utilized to evaluate the effectiveness and rationality of MASTER in terms of ACC and NMI. As shown in Table 3, there are three variants. MASTER w/o SRL, MASTER w/o RND, and MASTER w/o SIC represent removal of SRL, RND, and SIC, respectively. Two observations can be concluded from Table 3. ① The three variants yield inferior results on both datasets. That is, the removal of each component causes a decrease in clustering performance, which validates the effectiveness of each component. ② MASTER outputs the best clustering results on both datasets in terms of ACC and NMI, which further validates the design rationality of MASTER.

Hyper-parameter Results: Figure 2 presents the influence of the trade-off parameters α, β, and γ on Caltech-2V in terms of ACC. In detail, all trade-off parameters are set in {100, 10, 1, 0.1, 0.01, 0.001}. Then, one parameter is fixed and the other two are adjusted in the hyper-parameter analyses. As shown in the results of Figure 2, MASTER is robust to the selection of trade-off parameters. That is, when α, β, and γ are in {10, 1, 0.1}, {10, 1, 0.1, 0.01}, and {1, 0.1, 0.01}, respectively, MASTER can produce stable results close to the optimal result within a large range of settings. In addition, Figure 2(d) illustrates the influence of the selection of in-neighborhood sample number K in RND, and the results show that MASTER is of high robustness and can find the optimal number of in-neighborhood samples.

4.4 The Visualization Results

Figure 3 visualizes the clustering results that are carried out on 5 datasets to further evaluate the clustering superiority of MASTER. In detail, 1000 multi-view samples are randomly selected from each of 5 datasets, which are further input into MASTER to produce assignment predictions. Then, the tSNE algorithm is conducted on the assignment predictions to output the visualization of clustering results.

As shown in Figure 3, multi-view samples belonging to the same cluster are densely clustered within the same region, and there are no overlaps between distinct clusters. This observation substantiates that MASTER is competent to mine patterns that ensure the inter-cluster separation and intra-cluster compactness.

5 Conclusion

In this paper, a multi-granularity invariant structure clustering scheme is proposed to define a bottom-up process, which extracts multi-level information in sample, neighborhood, and category granularities. Specifically, MASTER captures invariant sample structure in the low-level feature space, invariant local structure in the high-level feature space, and invariant global structure in the semantics space, in which the three components cooperate to enhance the discriminative ability of the complementary and consistent information for MVC. Finally, extensive experiments on 8 real-world datasets verify that MASTER achieves state-of-the-art performance in comparison with 11 baselines.

Acknowledgments

This study was supported by Hainan Provincial Natural Science Foundation of China (Grant No. 825CXTD608) and grants (No. 62162023 and No. KYQD(ZR)-21079).