Simple Contrastive Multi-View Clustering with Data-Level Fusion

1 Introduction

Multi-View Clustering (MVC) can leverage rich information among multiple views to explore comprehensive cluster structures in multi-view datasets [Bickel and Scheffer, 2004; Xu et al., 2013; Zhang et al., 2024], so it has been become an important subdomain for the unsupervised clustering analysis. In MVC, learning representative features from multi-view data plays a decisive role in clustering performance, and works with different feature learning methods have been proposed [Ren et al., 2022; Zhang and He, 2023; Chen et al., 2022]. Motivated by the recent success of deep learning, research interest is largely paid to study deep MVC with neural networks [Wen et al., 2022; Fang et al., 2023].

According to the methodology for interacting multi-view information, existing deep MVC methods can be summarized into two categories, i.e., feature-level fusion in Figure 1(a)

∗Corresponding Author (jiexuwork@outlook.com).

and feature-level consistency in Figure 1(b). Feature-level fusion first utilizes un-shared encoder networks to learn deep features of different views, and then establishes a fusion module at the feature-level to achieve information fusion across all views. On the fused feature, early work directly applies traditional single-view clustering to MVC, such as subspace methods and spectral methods [Abavisani and Patel, 2018; Huang et al., 2019], where decoder networks are usually stacked behind fusion modules to regularize the fused feature. Subsequent work incorporates weighting or attention strategies into fusion modules to quantify the importance of views [Zhou and Shen, 2020; Yin et al., 2020]. Feature-level consistency similarly employs different encoders to learn deep features for individual views, but then conducts consistency optimization objective among features to explore their mutual information, such as CCA methods and contrastive learning methods [Andrew et al., 2013; Wang et al., 2015; Tian et al., 2020]. Recently, contrastive learning based deep MVC showcases great success, where multiple views of a sample are used to construct positive pairs and their features' consistency is encouraged by minimizing contrastive loss [Lin et al., 2021; Xu et al., 2023]. For example, [Trosten et al., 2021] first perform contrastive learning among multiple features and then fuse them to generate clustering predictions. [Xu et al., 2022] propose to learn multi-level features and clustering predictions for different views without feature fusion. The contrastive MVC methods could learn instance-discriminative features by leveraging views to selfsupervise each others, and has inspired a lot of work to advance different issues in deep MVC [Lin et al., 2022; Trosten et al., 2023; Liu et al., 2023; Yan et al., 2023; Jin et al., 2023; Chen et al., 2023; Yang et al., 2023].

While remarkable progress has been made by existing deep MVC methods, deep MVC still faces significant challenges. First, the methods including both feature-level fusion and feature-level consistency need to build un-shared encoder networks for different views to learn features. However, the diversity of multi-view data (varying dimensions, sparsity, and data formats) renders the customization of encoders nearly impossible. Second, existing methods often employ encoders of the same structure for different views, resulting in suboptimal solutions and model redundancy. Third, the features obtained through more encoders might increase the risk of losing inherent information of data, thereby hindering the subsequent feature-level fusion or consistency operations from exploring the useful discriminative information across views.

To address the aforementioned issues, we propose a novel framework entitled SCM: Simple Contrastive Multi-view clustering with data-level fusion as shown in Figure 1(c). Firstly, to avoid using multiple encoder networks as in previous methods, we propose shifting the fusion step from the feature-level to the data-level. In order to ensure that the fused data retains the information within multi-view data, we employ normalization and concatenation operations to achieve the data-level fusion without other operations. In this way, the discriminative information of different views can be encapsulated within different dimensions of the fused data, allowing us to search for data partitions in the fused data space, and thus handle multi-view learning problems as conveniently as single-view learning with a shared encoder network. Secondly, to enhance the robustness of deep model towards data noise and unavailability in real-world multi-view scenarios, we propose noise multi-view data augmentation and missing multi-view data augmentation to process the fused data. Furthermore, we employ instance-discriminative contrastive learning on the two types of augmented data, ensuring that the learned features are conducive to explore cross-view discriminative information while filtering the effect of noisy and unavailable data. Thirdly, we leverage the foundational SCM framework to conduct feature clustering and end-to-end clustering with known and unknown class number. Our main contributions are listed as follows:

• We propose a novel deep multi-view clustering framework (SCM) by data-level fusion for processing multiview data, which addresses the challenges of network customization and redundancy in previous methods.
• We develop two multi-view data augmentation techniques that specifically consider data noise and unavailability, marking contrastive learning with a shared encoder can effectively learn useful information from data.
• We implement several variants of our SCM framework equipped with simple network structure. Extensive experiments indicate that our method achieves comparable or superior clustering performance relative to state-of-

the-art methods. The simplicity of SCM is advantageous for its extension to other multi-view learning domains.

Notation definition In this paper, we represent matrices with uppercase bold letters and vectors with lowercase bold letters. Given a multi-view dataset {Xv ∈ RN×dv }Vv=1 , N and V respectively denote the sample size and the view number, and dv is the sample dimensionality of the v-th view data Xv .

2 Method

In this paper, we propose our SCM framework as shown in Figure 1(c), which not only can avoid the model customization for different views (this issue will exist in previous deep MVC methods as Figure 1(a-b)) by our data-level fusion with a shared network, but also can learn robust features by our multi-view data augmentation with contrastive learning.

2.1 Data-Level Fusion and Data Augmentation

To begin with, we introduce the basics in SCM framework, i.e., data-level fusion and multi-view data augmentation.

Data-level fusion Unlike previous methods that perform fusion operations at the feature-level, we advocate for data-level fusion of multi-view data to synergistically utilize the discriminative information of multiple views. The fused data will establish a bridge to mitigate the gap between multi-view learning and single-view learning, and avoid the redundancy and customization issues of multi-view encoder networks.

Specifically, we express the data-level fusion as a function:

where N(·) denotes the min-max normalization that brings the variables of different views into a uniform scale without distorting the ranges of values, [·] is the concatenation operation, and ¯X ∈ RN×D (D = ∑v dv) is defined as the fused data for all views. In this way, the discriminative information of different views can be encapsulated within different dimensions of the fused data. Traditional feature-level fusion after dimensionality reduction with different encoders might result in the loss of information due to data processing inequality.

Hence, in our data-level fusion, we refrain from employing complex mappings to ensure that the fused data retains the discriminative information from all views' raw data.

Multi-view data augmentation Motivated by the beneficial effects of data augmentation [Shorten and Khoshgoftaar, 2019] in computer vision, we propose multi-view data augmentation techniques targeted for the fused data to increase model representation ability, which simulates the scenarios of noisy and unavailable data in practical multi-view learning.

Specifically, considering the presence of noisy data within multi-view datasets, we design the noise multi-view data augmentation by adding noise on some views for each sample:

To be specific, we denote the v-th view data in the fused data as ¯Xv = N(Xv ) ∈ RN×dv . Then, for the i-th data ¯xvi ∈ ¯Xv , its noise-augmented data ¯nvi ∈ ¯Nv is generated by

where δv i is randomly sampled from an uniform distribution, and ϵ ∈ Rdv is random noise sampling from a Gaussian distribution N(0, σ2 ). p is a threshold that controls the proportion of noise-augmented data within multi-view data.

Further, given the case of data unavailability in multi-view datasets, we also design the missing multi-view data augmentation by masking values on some views for each sample:

To be specific, we denote the v-th view data in the fused data as ¯Xv = N(Xv ) ∈ RN×dv . Then, for the i-th data ¯xvi ∈ ¯Xv , its missing-augmented data ¯mvi ∈ ¯Mv is generated by

where A ∈ {0, 1}N×V is a random indicator matrix, which ensures that for the i-th sample, data from different views can be zeroed out to simulate the state of data unavailability while guaranteeing that at least one view remains available. We have ∑i (I{∑v aiv < V })/N = r where I{·} denotes the indicator function, and r is a threshold that controls the proportion of missing-augmented data within multi-view data.

The augmented data ¯Xn and ¯Xm are dynamically generated during training and thus will make SCM can be robust to the data noise and unavailability by the interaction among multiple views. Moreover, ¯Xn , ¯Xm ∈ RN×D are the same in data format and can be trained with a shared deep model.

2.2 Contrastive Clustering in SCMFramework

Given the fused data and its augmented data, we present our SCM equipped with contrastive leaning, as well as its variants with reconstruction regularization and end-to-end clustering. Contrastive learning For the augmented data ¯Xn and ¯Xm, we utilize a shared encoder Eθ (parameterized by θ) to extract their features ¯Zn ∈ RN×Z and ¯Zm ∈ RN×Z , respectively:

We then apply instance-discriminative contrastive learning on ¯Zn and ¯Zm, to explore discriminative information across multiple views within the fused data. Specifically, for a minibatch samples B, {¯zni ∈ ¯Zn , ¯zmi ∈ ¯Zm}i=1,...,|B| are |B| positive pairs. For each ¯zni , its (2|B| - 2) negative pairs is {¯zni , ¯zvj }v=n,m j̸=i and {¯zvj }v=n,m j̸=i is denoted as a set of s-. The InfoNCE [Oord et al., 2018] loss for a single positive pair with multiple negative pairs is given by:

where τ denotes a temperature parameter and the distance between two sample features (e.g., ¯zni ∈ ¯Zn and ¯zmj ∈ ¯Zm) is measured by cosine similarity:

The overall InfoNCE loss for the batch is defined as follows:

In this SCM framework, contrastive learning encourages the encoder to map positive pairs closer together relative to negative pairs in the feature space, thus learning to discriminate different samples of the fused data. Specially, the feature ¯Zn can learn from noiseless views in data ¯Xm, and the feature ¯Zm can learn from available views in data ¯Xn , so we achieve the interaction of multi-view information in a shared model and aim to increase the model robustness. Then, we design model regularization and end-to-end prediction for clustering. Reconstruction regularization Since clustering is an unsupervised learning task, using a decoder network to reconstruct the original data from learned features can naturally create a self-supervised signal, that encourages the capture of discriminative structures hidden within the data. This reconstruction regularization has been successfully applied to many deep MVC methods [Trosten et al., 2023], which typically use unshared decoder networks for reconstructing different views. Within the context of our proposed multi-view data augmentation, we introduce a novel approach by employing a shared decoder network for achieving reconstruction regularization.

Specifically, we leverage a shared decoder network Dϕ (parameterized by ϕ) and establish three different reconstruction losses for the fused data and augmented data as follows:

where ¯Hn , ¯Hm, ¯H are the hidden features between data ¯Xn , ¯Xm, ¯X and features ¯Zn , ¯Zm, ¯Z, respectively. θ′ denotes partial network parameters within θ. For example, we have ¯Hn = Eθ′ ( ¯Xn ) and ¯Zn = Eθ( ¯Xn ) = Eθ\θ′ ( ¯Hn ). It is noteworthy that we impose the reconstruction loss on ¯Hn , ¯Hm, ¯H to avoid the inductive conflicts with the contrastive loss applied on ¯Zn , ¯Zm [Xu et al., 2022]. Then, the overall reconstruction loss is formulated as follows:

This overall reconstruction loss could be viewed as a combination of different reconstruction regularization [Xu et al., 2023]. Concretely, LnRE encourages the hidden feature ¯Hn to reconstruct the noiseless data ¯X from noise-augmented data ¯Xn . LmRE enables the hidden feature ¯Hm to reconstruct the available data ¯X from missing-augmented data ¯Xm. L-RE regularizes the hidden feature ¯H obtained by the noiseless and available data ¯X, for usage in subsequent clustering tasks. End-to-end clustering Existing deep MVC methods for achieving end-to-end clustering primarily employ two strategies: I) In feature-level fusion methods, they often obtain cluster pseudo-labels on the fused features and then train a cluster network through self-training; II) For feature-level consistency methods, they usually set up separate cluster networks for different views and achieve consistent clustering through contrastive learning. It is worth noting that existing methods tend to require pre-setting the number of clusters K to design the dimensionality of the model's cluster network for each dataset, and our experiments in Section 3.2 find that fixed K is harmful for clustering performance. To make our model architecture compatible with different Ks of datasets, we propose end-to-end clustering on the basis of our SCM.

Specifically, we add a H-dimensional cluster network behind the feature ¯Z and obtain clustering labels ¯Q ∈ RN×H :

where Rω is a linear MLP network that organizes the dimensionality of clustering prediction to H. To extract known clustering structure information from ¯Z, we utilize a clustering method that does not depend on class number, such as Density Peaks [Rodriguez and Laio, 2014], which automatically searches the feature space to obtain a set of anchor points A = {aj}|A| j=1 , aj ∈ RZ . Further, through nearest neighbor assignment, we obtain the clustering labels ¯P ∈ {0, 1}N×H (pij ∈ ¯P) for N samples as follows:

Actually, we have ¯P = [{0, 1}N×|A| ; {0}N×(H-|A|) ]. As a result, the first |A| dimensions of the matrix ¯P contain the cluster information in the learned features, and the last (H - |A|) dimensions are all zeros. To achieve end-to-end clustering, we minimize the following mean squared error:

In implementation, the dimensionality number H can be set much larger than the potential class number of dataset, thus decoupling the design of the model's neural network structure from specific dataset. The final clustering prediction can still be obtained as follows, e.g., for the i-th sample:

Input: Multi-view dataset {X1 , X2 , . . . , XV } Setting: SCM(λ1, λ2 = 0), SCMRE(λ1 = 1, λ2 = 0), SCMEC(λ1 = 0, λ2 = 1), SCMEC+RE(λ1, λ2 = 1), rates of data augmentation p, r, std σ, batch size |B|, network parameters θ, ϕ, ω, learning rate η Data-level fusion ¯X= F(X1 ,X2 , . . . ,XV ) while not converging do Sampling mini-batch data ¯XB from ¯X ¯XnB = f nDA( ¯XB; p, σ), ¯XmB = f m DA( ¯XB; r) Compute ¯HnB , ¯HmB , ¯HB, ¯ZnB , ¯ZmB if λ1 == 0 then Compute L = LCO Update θ ← θ - η∇L(θ) else Compute L = LCO + LRE Update θ, ϕ ← θ, ϕ - η∇L(θ, ϕ) if λ2 == 1 then Compute ¯Z on ¯X and infer ¯P on ¯Z while not converging do Sampling mini-batch data ¯XB from ¯X ¯XnB = f nDA( ¯XB; p, σ), ¯XmB = f m DA( ¯XB; r) Compute ¯HnB , ¯HmB , ¯HB, ¯ZnB , ¯ZmB , ¯QB if λ1 == 0 then Compute L = LCO + LEC Update θ, ω ← θ, ω - η∇L(θ, ω) else Compute L = LCO + LRE + LEC Update θ, ϕ, ω ← θ, ϕ, ω - η∇L(θ, ϕ, ω) Output: ¯Z for SCM/SCMRE, ¯Q for SCMEC/EC+RE

Finally, we summarize the loss function in our method as

We leverage λ1, λ2 ∈ {0, 1} to obtain different variants of SCM framework: SCM(λ1, λ2 = 0), SCMRE(λ1 = 1, λ2 = 0), SCMEC(λ1 = 0, λ2 = 1), SCMEC+RE(λ1, λ2 = 1), whose training steps are shown in Algorithm 1.

Complexity analysis We let E denote the total training epochs, V and N are the number of views and the sample size of a dataset, |B| is the batch size in the mini-batch optimization. For each batch, the computational complexity of multi-view data augmentation is 2O(|B|), that of contrastive loss, reconstruction loss, and end-to-end clustering loss are O(|B|2 ), 3O(|B|), and O(|B|), respectively. The computational complexity to obtain clustering labels is O(N). For E training epochs, the computational complexity approximates to O(N) + (EN/|B|)O(|B|2 ) which is linear to sample size N. In terms of the memory consumption of deep model, the complexity in our SCM is 1/V of that in previous deep MVC methods, as the deep model in SCM is a shared network while that of other methods need V individual networks.

3 Experiment

3.1 Experimental Setup

Datasets We conduct experiments on 8 public datasets, including BDGP [Cai et al., 2012], DIGIT [Peng et al., 2019], Fashion [Xiao et al., 2017], NGs [Hussain et al., 2010], VOC [Everingham et al., 2010], WebKB [Sun et al., 2007], DHA [Lin et al., 2012], and COIL-20 [Nene et al., 1996]. Baselines The comparison methods include K-Means [MacQueen, 1967], Density peak clustering [Rodriguez and Laio, 2014], and deep MVC methods CPSPAN [Jin et al., 2023], CVCL [Chen et al., 2023], DSIMVC [Tang and Liu, 2022a], DSMVC [Tang and Liu, 2022b], MFLVC [Xu et al., 2022]. Implementation details To facilitate a fair comparison, we employed the same network architecture in [Xu et al., 2022; Tang and Liu, 2022a] to implement SCM. It is important to note that whereas previous methods necessitated multiple encoder-decoder networks, our SCM framework requires only one. Specifically, the encoder network structure can be depicted as a fully connected (Fc) MLP [Rosenblatt and others, 1962] with the configuration of ¯X- Fc500 - Fc500 - Fc2000 - ¯H- ¯Z- ¯Q, and the decoder network structure is ¯H - Fc2000 - Fc500 - Fc500 - ¯X. The activation function for the cluster network ¯Q is Softmax, while ReLU [Nair and Hinton, 2010] is used for all other activation functions. For all datasets used in our experiments, the dimensions of ¯H, ¯Z, and ¯Q were set to 256, 128, and 64, respectively. The optimizer was Adam [Kingma and Ba, 2014] with a learning rate of 0.0003, and the batch size was set to 256. Both the noise and missing rates of multi-view data augmentation were set to 0.25, and the noise variance was 0.4. Our SCM is implemented by PyTorch and its code is available in https://github.com/SubmissionsIn/SCM.

3.2 Result Analysis

Tables 1 and 2 showcase the clustering results of all comparison methods with known and unknown class number, respectively, where mean values of 5 runs are reported. Evaluation metrics include clustering ACC, NMI, and ARI.

Clustering with known class number When the class number is known, we report clustering results of comparison methods as shown in Table 1. Obviously, our methods, including vanilla SCM and reconstruction regularized SCMRE, achieve superior clustering performance. Specifically, the average results across 8 datasets indicate that our SCM achieved the improvement of 5% in ACC and 2% in NMI compared to the currently best-performing methods, while SCMRE achieved the improvement of 6% in ACC and 5% in NMI. In addition to superior clustering performance, our proposed data-level fusion makes SCM have simple contrastive multiview learning paradigm, which can avoid the issues of model redundancy and network customization in previous methods. Clustering with unknown class number To further explore the impact of the unknown class number, we report end-toend clustering results of different methods in Table 2. If the class number in models is fixed, previous end-to-end deep MVC methods often yield degraded results. This is because end-to-end clustering methods typically depend on the truth class number of datasets to design their model structures. In contrast, our methods (SCMEC and SCMEC+RE) still achieve best performance, for instance, the ACC of SCMEC and SCMEC+RE have 9% and 13% improvements to the best comparison methods, respectively. The reason is that our SCM has the decoupled design between the model structure and the setting of class number, which transfers the problem of sensitive class number from the model structure to the density peak algorithm and is beneficial for its applicability.

3.3 Ablation Study

In this part, we first investigate the key contributions in our proposed noise and missing multi-view data augmentation, and then analyze the different components in loss function.

Multi-view data augmentation As shown in Table 3, firstly, SCM w/o DA is a variant without any data augmentation, which achieves unsatisfactory clustering performance. Furthermore, SCM w/ f nDA and SCM w/ f m DA are variants that incorporate the noise multi-view data augmentation and the missing multi-view data augmentation defined in this paper, respectively, and they both show significant improvements over SCM w/o DA. Finally, SCM combines two types of data augmentation and achieves further substantial improvements, confirming the importance of our specially designed noise and missing multi-view data augmentation.

Loss components In Table 4, we conduct ablation studies on three losses using SCMRE and SCMEC+RE as baselines. Compared to SCMRE, removing the reconstruction regularization loss in variant SCMRE w/o LRE results in a slight performance decline, while removing the contrastive loss in variant SCMRE w/o LCO leads to a severe performance decrease. Furthermore, compared to the end-to-end clustering setting of SCMEC+RE, removing the end-to-end clustering loss in variant SCMEC+RE w/o LEC also significantly degrades model performance. These results indicate that LCO plays the most crucial role in contrastive multi-view clustering, and LEC is vital for end-to-end clustering, with LRE serving a supporting role to regularize the feature learning.

3.4 Model Analysis

In this part, we conduct visualization analysis on the parameters and the training process in our SCM framework.

Data augmentation rates {p, r} In SCM framework, we tune the noise multi-view data augmentation rate p and the missing multi-view data augmentation rate r within the range [0, 0.25, 0.5, 0.75, 1.0], with results depicted in Figure 2. We observe that moderately increasing p and r significantly benefits the model in learning precise clustering structures within multi-view datasets. The underlying mechanism is that our noise and missing multi-view data augmentation compel the model to focus on the interaction and complementarity across views, thereby making the feature learning more robust to inherent noise and unavailable samples in multi-view data. In comparison experiments, p and r were set to 0.25.

Trade-off parameters {λ1, λ2} In our method, trade-off parameters λ1, λ2 ∈ {0, 1} control the different settings in SCM framework. In Figure 3, we adopt the setting of SCMEC+RE and further explore the sensibility of λ1 and λ2 within the range of [10-3 , 10-2 , 10-1 , 100 , 101 , 102 , 103 ]. The optimal values of λ1 and λ2 are different across different datasets and this outcome is within expectations. Trade-off parameters generally have a minimal sensibility on the performance within the range of [10-1 , 101 ]. In all experiments, we did not specifically tune λ1 and λ2, and their values were fixed at 0 or 1 to implement different variants of SCM.

Training loss and performance In Figure 4, we record the loss and clustering accuracy curves during the training process of SCM. It is observed that the loss curve exhibits a smooth and consistent decline, indicating that SCM framework has well convergence. Concurrently, the steadily rising clustering accuracy suggests that the model is progressively learning the correct clustering structure of the dataset.

4 Conclusion

In this paper, we propose a novel contrastive multi-view clustering framework with data-level fusion, namely SCM. Specifically, our proposed data-level fusion effectively integrates multi-view information and avoids the issues of customization and redundancy of networks in previous methods. Moreover, we define two types of multi-view data augmentation approaches based on the data-level fusion, which enhances the robustness of model towards noisy and unavailable views in multi-view data. We apply the SCM framework to feature clustering and end-to-end clustering with known and unknown class number, and extensive experiments validate its effectiveness and superiority. Our SCM simplifies multiview contrastive learning with a shared deep network, and we hope it could bring fresh insights into multi-view learning.

Acknowledgments

This work was supported in part by the National Key Research & Development Program of China under Grant 2022YFA1004100, Medico-Engineering Cooperation Funds from University of Electronic Science and Technology of China under Grant ZYGX2022YGRH009 and Grant ZYGX2022YGRH014, Guangxi Natural Science Foundation (2023GXNSFBA026010), Research Fund of Guangxi Key Lab of Multi-source Information Mining & Security (22-A03-02), Innovation Project of Guangxi Graduate Education (XJCY2022009).