Contrastive and View-Interaction Structure Learning for Multi-view Clustering

1 Introduction

In contemporary times, data originating from diverse domains, sensors, or feature extractors is readily amassed owing to the prevalence of network edge devices. Recent years have witnessed notable success in Multi-view Clustering (MVC)

∗Corresponding author

[Wen et al., 2022], which enhances clustering performance by integrating varied content information from multiple perspectives. MVC [Li et al., 2023; Wen et al., 2023c; Trosten et al., 2023; Wen et al., 2023b] methods can be roughly divided into graph-based [Tan et al., 2023], co-training [Kumar et al., 2011], subspace-based [Tang et al., 2022], multiple kernelsbased [Liu et al., 2020], and deep learning-based [Xu et al., 2023; Wen et al., 2023a] approaches respectively. Among the numerous MVC methods, Deep Learning-based Multiview Clustering (DMVC) stands out for its superior performance attributed to the remarkable representational capacity of deep neural networks. For instance, Multi-VAE [Xu et al., 2021] utilizes a deep generative network to disentangle visual representations into view-common and view-specific features, which are assumed to follow a discrete Gumbel Softmax distribution and a continuous Gaussian distribution, respectively. Completer [Lin et al., 2021] focuses on acquiring complementary multiple representations through a within-view reconstruction task. Simultaneously, it incorporates a contrastive task to leverage the consistency of multiple representations extracted by view-specific deep encoders. Although remarkable progress has been achieved by existing DMVC methods, they merely take advantage of multiview (content) information from the same sample, neglecting to consider the topological relation (structure) information among different samples. As an unsupervised representation learning task, mining the intrinsic relationship of samples is crucial for multi-view clustering. On the other hand, even contrastive learning significantly improves the performance of unsupervised representation learning, which recognizes all of the other samples as negatives, importing many false negative pairs and increasing the intra-class distance.

In this paper, we propose a contrastive and view-interaction structure learning framework for multi-view clustering (SERIES) to address the aforementioned issues, as shown in Figure 1. SERIES aims to explore the structure relations among different samples and decrease the intra-class distance, finally achieving a consistent structure graph for spectral clustering. Specifically, we first utilize both the multi-view data and structure graphs as the input of view-specific deep graph autoencoders to learn multiple latent representations. Then, the soft negative pair aware contrastive learning module introduces a dynamic sample weighting strategy of negative pairs to pull together similar samples while pushing apart the dissimilar samples in the feature space. According to the view-specific representations, we use a cross-view dual relation generation mechanism to generate the structure graphs for each other, which fully uncovers the consistent topological structure across multiple views. Finally, the view-specific structure graphs are integrated into a unified one for subsequent spectral clustering. The major contributions are summarized as follows:

• We propose a contrastive and view-interaction structure learning framework for multi-view clustering, which is able to reveal the intrinsic structure of data points and achieve a more compact cluster structure. Extensive experiments are conducted to validate the efficacy of our approach.
• Different from the existing DMVC methods, we simultaneously adopt view-specific graph autoencoders and design a cross-view dual relation generation mechanism to capture the intrinsic structure of samples and model the consistent structure graph for clustering.
• To reduce the intra-class distance, we propose a soft negative pair aware contrastive loss, which can utilize a dynamic sample weighting strategy to distinguish similar samples from negative pairs.

2 RelatedWork

In recent years, advancements in multi-view clustering have demonstrated improved performance through the exploration of complementary information from various views. These approaches can be broadly categorized into co-training, subspace-based, multiple kernels-based, and graph-based methods, respectively. Co-training multi-view clustering [Kumar et al., 2011; Kumar et al., 2011] employs coregularization on clustering hypotheses to accomplish consistent clustering across all views. Multi-view subspace clustering methods [Huang et al., 2022] usually project multi-view data into a common subspace latent space, uncovering accurate group information for a set of data points. Multiple kernel clustering [Liu et al., 2023] involves computing several kernel matrices for each view, mapping multi-view data into a high-dimensional Hilbert space for clear data point separation. Although these methods exhibit strong performance, they often neglect the structural relations between instances. In contrast, graph-based multi-view clustering methods construct graphs for each view and execute spectral clustering on the consensus graph, allowing for comprehensive capture of topological structure information among instances.

Recently, Graph Neural Network (GNN) [Wu et al., 2020] has drawn lots of attention for its capacity to uncover both structure information and content information within samples. For instance, Graph Convolutional Network (GCN) [Kipf and Welling, 2017] first extends convolutional neural networks to graph structure data, which proposes a layerwise propagation rule to learn a hidden representation containing both local topological and attributes of nodes. Considering assigning different importance to different nodes and improving efficiency, graph attention networks [Velickovic et al., 2018] introduce masked self-attentional layers into the graph convolutional network to enhance aggregation capability. Owing to its adeptness in leveraging structural relationships, the GNN has been applied to multi-view clustering [Wen et al., 2020a; Wen et al., 2023d]. O2MGC [Fan et al., 2020] is the first attempt to introduce GNN into MVC, which learns the node representation from the structure graph and node content information by GCN encoder and reconstructs multiple graphs using various decoders. After that, SGDMC [Huang et al., 2023] employs an attention-allocating approach to compute node similarity, mitigating the negative impact caused by noisy nodes.

3 Method

In this work, we propose a contrastive and view-interaction structure learning framework for multi-view clustering, which is composed of three submodules and the detailed flowchart is illustrated in Figure 1.

3.1 View-specific Deep Graph Autoencoders

The primary limitation of previous deep multi-view clustering works is that the topological structure is not fully utilized. In this work, we address this gap by employing graph neural networks to propagate structure relations during the representation learning stage.

Given the multi-view data {Xv ∈ Rn×dv }mv=1 encompasses n samples and v views, where dv represents the feature dimension of the v-th view, we explicitly construct view-specific graphs {Av ∈ Rn×n }mv=1 using the k-Nearest Neighbors (kNN) algorithm. Then, in order to learn the multiple representations containing both content and structure information, we introduce the graph autoencoder, which consists of view-specific graph encoders and view-specific graph decoders:

View-specific graph encoders. The graph encoders can enhance view-specific representations by aggregating neighbor information through structure graphs. Specifically, for v-th view, we feed both feature matrix Xv and graph Av into a multi-layer GCN to learn the instance-level latent representation Zv , and the computation of the l-th layer of GCN is formulated as follows:

where ˜Av = Av + In ∈ Rn×n is the affinity matrix with added self-connections, and ˜Dvii = ∑j Avij ∈ Rn×n denotes the corresponding degree matrix. ϕ(·) is the activation function, W(v,l) and b(v,l) serve as parameters for the l-th GCN layer. When l = 0, Z(v,0) corresponds to the original data Xv of the v-th view, and the latent representation Z(v,1) is obtained by:

View-specific graph decoders. In order to guide the viewspecific representations to maintain the instance structure relationship in the latent space, we employ view-specific graph decoders to reconstruct graph ˆAv . Specifically, the inner product decoder is adopted to predict the links between instance i and j, which can be formulated as:

The graph reconstruction error measuring the difference between ˆAv and Av of all views can be calculated by:

Meanwhile, to embed the content information of multiview data into instance latent representation Zv , the content decoders are introduced to reconstruct ˆXv . More specifically, for the v-th view, a view-specific decoder network gv (·), parametrized by µv , is employed to decode Zv into ˆXv :

The content reconstruction loss of all views is defined as follows:

3.2 Soft Negative Pair Aware Contrastive Learning Module

As a key module of deep learning-based multi-view clustering methods, contrastive learning is adopted to align multiple representations from different views of each instance and learn the corresponding discriminative features. Specifically, for mn samples {X11 , . . . ,X1i , . . . ,Xmi , . . . ,Xmn }, standard contrastive learning typically treats (Xvi ,Xui ) as positive pair and other mn -m samples to be negative pairs. However, samples that belong to the same cluster should not be treated as negative pairs, which conflicts with the purpose of the clustering task. To address the problem, many works [Trosten et al., 2021; Xia et al., 2023] introduce the pseudo labels to remove within-class samples from negative pairs. Although this adjustment mitigates the issue of false negative pairs, it is difficult to achieve reliable pseudo-labels during the training process, especially in the initial stage of training.

Different from previous works that categorize negative pairs by hard pseudo-labels, we propose a soft negative pair aware contrastive loss, aiming to bring correlated samples closer while distinguishing uncorrelated ones. Specifically, we introduce a weight modulating function ρ(·, ·) to dynamically adjust the weights of the sample pairs during training. Based on the view-specific latent representation Zvi , ρ(·, ·) can be defined as follows:

where hvi is obtained by a view-shared project head hvi = σ(z(v,l) i ), and σ(·) is introduced to filter out the view-specific noise. s(·, ·) is the similarity function, which can be computed as s(hvi , hpj ) = (hvi )T hpj . β ∈ [1, 5] is the penalty factor employed to adjust the degree of penalization for both uncorrelated samples and correlated samples. For instance, when β = 2, the similarity of the correlated samples pair is 0.8, but the corresponding weight is 0.04. In contrast, for uncorrelated samples pair, the similarity is 0.1, while the resulting weight is 0.81, which significantly surpasses 0.04.

According to the dynamic weight modulating function ρ(·, ·), we formulate the soft negative pair aware contrastive loss between view v and u as follows:

where τ denotes the temperature parameter. The soft negative pair aware contrastive loss across all views is defined as:

3.3 Cross-viewDual Relation GenerationModule

Through the graph autoencoders and the soft negative pairaware contrastive learning module, we can obtain discriminative feature Hv , which contains both the structure and content information. To fully fuse complementary information across multiple views, we introduce a cross-view dual relation generation module in this subsection. This module generates view-specific affinity matrices by leveraging information from both the target view and other views, facilitating the exploration of a consistent topological structure graph.

Specifically, we first employ the self-relation generation layer SR(·) to generate the affinity matrix Gv of v-th view, which aims to represent each instance as a combination of others from v-th view. The generation proposes is formulated as follows:

To explore the global structure of samples under the current view, we minimize the following reconstruction loss:

Furthermore, to fully integrate the complementary information across different views, we propose a dual-relation generation layer DRu (·) to generate the affinity matrix G(u,v) of v-th view, which can be formulated as:

where G(u,v) is the affinity matrix generated by u-th view. It utilizes the representations Hu from u-th views to represent Hv , and the reconstruction loss is formulated as follows:

The overall reconstruction loss can be computed as follows:

Accordingly, the affinity matrix of v-th view is obtained by:

In this way, the view-specific affinity matrix Sv has the ability to fuse the complementary structure relations from multiple views, and the consensus affinity matrix S is simply obtained by:

3.4 The Overall Loss Function of SERIES

In summary, we have introduced a contrastive and viewinteraction structure learning framework for multi-view clustering. In the training stage, the view-specific graph autoencoders, soft negative pair aware contrastive learning module, and the dual relation generation module are jointly optimized according to the following objective function:

Finally, we obtain a desirable consensus affinity matrix S and pass it through a spectral clustering algorithm to achieve the final clustering result. The whole learning process of SERIES is summarized in the Algorithm 1.

Algorithm 1 The Algorithm of SERIES

Input: Multi-view data {X(v) }mv=1 ; Training iterations T . Process:

1. Construct the graphs for each view and obtain the viewspecific affinity matrices {A(v) }mv=1 .

Pretrain:

1. Pertrain the deep graph autoencoders of each view by optimizing Lvg_rec , Lvc_rec in Eq. 4 and 6.

Finetuning:

3. for epoch = 1 to T

1. Obtain the Zv,Gv,G(u,v) , Sv of each view by Eqs. (1,10,12,15).
2. Update network parameters by using Adam to minimize the objective in Eq. 17.

9. end for

return: The consensus affinity matrix S.

Perform spectral clustering using S.

4 Experiment

4.1 Experimental Settings

Datasets. The following datasets are carried out for evaluation: (1) HW [Perkins and Theiler, 2003] consists of 2000 samples from 10 types of handwritten digits, all of which are presented by two features. (2) Reuters [Amini et al., 2009] comprises 6 categories, corresponding to 1200 articles, and each article is written in 5 languages. (3) Noisyminist [Wang et al., 2015] contains 70k instances described by

2 views. In this study, we choose a subset of Noisyminist, consisting of 15,000 instances from 10 classes, for the comparative experiment. (4) VOC [Hwang and Grauman, 2010] is a two-view dataset that encompasses image-view and textview modalities, which consists of 5,649 instances distributed across 20 distinct classes. (5) Hdigit [Chen et al., 2022] includes 10,000 instances categorized into 10 classes, whose two views are crafted from both MNIST Handwritten Digits and USPS Handwritten Digits. (6) Mfeat [Wang et al., 2019] comprises 2000 instances from 10 subjects, with six features extracted to form this multi-view dataset. The detailed information is summarized in Table 2.

Metrics. We employ six widely used metrics to evaluate our model, including Accuracy (ACC), Normalized Mutual Information (NMI), Purity (PUR), Adjusted Rand Index (ARI), Precision (P), and F-score. The specific definitions of these metrics are explained in [Cao et al., 2015].

Baselines. Our proposed method is compared with nine state-of-the-art methods, which are summarized as follows:

• LMSC [Zhang et al., 2017] utilizes a self-supervised reconstruction task to acquire latent representations for

multiple views while concurrently investigating the inherent complementarity for MVC.

• MCGC [Zhan et al., 2019] dynamically learns a consensus graph to reveal more robust relationship between data points, which has exactly k connected components aligning with the number of clusters.
• LMVSC [Kang et al., 2020] finds some representative data points as anchors to construct anchor graphs, which can present the global structure relations and are beneficial to alleviate the computational issue.
• CDIMIC [Wen et al., 2020b] introduces the deep autoencoders and self-paced strategy to extract the highlevel features and reduce the negative impact of outliers.
• SiMVC [Trosten et al., 2021] is a simple deep MVC model obtaining a fused representation by weighted average multiples representations, which is proposed to prioritize views in the feature space.
• CoMVC [Trosten et al., 2021] extends SiMVC by incorporating a contrastive alignment mechanism, which aims at increasing the distance between distinct clusters.

• MFLVC [Xu et al., 2022] utilizes contrastive regularization terms to align the instance-level and cluster-level representations, which aims to explore multi-view consistency.
• CMGEC [Wang et al., 2021] is a graph-based deep multi-view clustering method, which considers the structure information among samples and introduces a mutual information maximization module to discover the consistent topological structure.
• DFP-GNN [Xiao et al., 2023] adopts GNN models and a new dual fusion mechanism to learn a unified representation, which combines both content and structure information across different views.

Implementations. All experiments are conducted on a Linux platform utilizing an Intel(R) Core(TM) i9-11900 2.50GHz CPU, 64GB RAM, and GeForce RTX 3090 Ti GPU. The view-specific deep graph autoencoders are pre-trained for 200 epochs, and the entire model is fine-tuned for an additional 100 epochs. The dimensions of the encoders, decoders, and the cross dual relation generation layer are set to {dv, 512, 2048, 256}, {256, 2048, 512, dv} and {256, dv} respectively. The activation function is specified as ReLU. In our study, the trade-off hyperparameters λ1, λ2 are selected from the range {0.1, 0.2, . . . , 0.9, 1.0}. All other baselines are implemented by the recommended network structure and parameters for fair comparison.

4.2 Comparison Results

Table 1 reports the clustering results of our method and other baselines on six datasets, where the best and the sub-optimal performance are denoted in bold and underlined, respectively. OM represent "Out-of-memory error". From the results, we have the following observations:

1. Among all the compared methods, SERIES achieves superior performance in most cases, and improvement of our algorithm on some datasets is significant. For example, on the VOC dataset, our algorithm surpasses the sub-optimal algorithm SiMVC by 36.25%, 24.60%, 16.61%, 15.20%, 26.71%, 35.68% in terms of six metrics, respectively. For Hdigit dataset, our algorithm improves 2.08%, 4.50%, 2.12%, 4.67%, 4.11%, 4.13% across six metrics compared to the second-best DFP-GNN method. These observations validate the excellent effectiveness of our model, which deeply investigates the topology structure between samples benefiting the partition of similar samples into the same cluster.
2. Compared with other state-of-the-art contrastive learning-based multi-view clustering methods, i.e., CoMVC and MFLVC, our method still shows significant advantages. MFLVC treats all of the other samples as negative samples which inevitably brings in some false negative pairs. CoMVC introduces the pseudo-label information to guide the selection of positive and negative pairs, which heavily depends on the quality of pseudo-labels. However, the contrastive loss proposed in our method aims to mitigate the impact of false negative pairs by introducing a dynamic weighting strategy, which reduces the distance between relevant samples and increases the distance between irrelevant samples.
3. Observed from Table 1, the shallow methods are inferior to the deep learning-based approaches in most cases, particularly on the VOC and Hdigit datasets. This discrepancy can be attributed to the deep learning-based multi-view methods effectively capturing the intrinsic features of instances. Notably, the graph-based method DFP-GNN, surpasses some deep learning-based methods, indicating the significance of exploring structural information between samples. Nevertheless, our approach consistently outperforms both other graphbased and deep learning-based multi-view clustering methods across all datasets. This consistency highlights the efficacy of our cross-view dual relation generation module and soft negative pair aware contrastive learning module.

4.3 Ablation Study

To verify the effectiveness of each submodule, we compare our proposed SERIES with its degenerated methods to conduct the ablation study in this subsection. Concretely, in Table 3 we analyze the following cases:

• SERIES-D: The affinity matrix Sv is only composed of Gv , omitting the dual generation process in Eq. 12.
• SERIES-SC: SERIES w.o. the soft negative pair aware contrastive loss in Eq. 8.
• SERIES-C: The soft negative pair aware contrastive loss is replaced by the standard contrastive loss [Chen et al., 2020], which treats all of the other samples as negative pairs.

From the results in Table 3, we have two observations as follows: (1) In comparison to SERIES-SC and SERIES-D, SERIES exhibits the best performance, indicating that both the soft negative pair aware contrastive learning module and cross-view dual relation generation module enhance the performance of the baselines. These modules facilitate the exploration of structural relations between instances and the acquisition of instance intrinsic features. (2) The results of SERIES-C are consistently lower than those of SERIES-SC across all metrics on both Mfeat and VOC datasets. This observation indicates that our proposed soft negative pair aware contrastive loss effectively mitigates the issue of false negative pairs. Unlike the standard contrastive loss which aims to increase the distance between one sample and all other samples, our method focuses on dynamically adjusting the weight according to the similarity of the samples, which is more suitable for the final clustering task.

4.4 Parameter Sensitivity Analysis

In this subsection, we analyze the hyper-parameters λ1 and λ2 in our method on Mfeat and VOC datasets. Figure 2 illustrates the ACC and NMI of our approach as λ1 and λ2 vary within the range of {0.1, · · · , 1.0}. As depicted in Figure 2, our method consistently demonstrates strong performance across a broad range, particularly exhibiting promising results when λ1 ∈ [0.1, 0.5] and λ2 ∈ [0.5, 1.0] on Mfeat and VOC datasets.

4.5 Visualization Analysis

In this subsection, we utilize t-SNE [Van der Maaten and Hinton, 2008] to visualize the final consensus representations in our method and other compare methods. As shown in Figure 3, the consensus representation learned in our method is more compact than others, which demonstrates our proposed SERIES could better explore the cluster structure compared with other baselines.

5 Conclusions

In this work, we propose a contrastive and view-interaction structure learning framework for multi-view clustering, named SERIES, to better reveal the structure relation between different instances. We first utilize the view-specific graph autoencoders to achieve latent representations containing both content and graph information. Then, the soft negative pair aware contrastive learning module introduces a dynamic sample weighting strategy to alleviate the false negative pair problem, which could better learn discriminative features for instances. Finally, to cluster samples with the structure graph of samples, we adopt the cross-view dual relation generation module to explore the consistent affinity matrix across multiple views. The effectiveness of SERIES has been verified on various multi-view datasets as compared with other state-of-the-art methods.

Acknowledgments

This work was supported by the Fundamental Research Funds for the Central Universities (No.2022JBZY019), the Beijing Natural Science Foundation (No. 4242046).