Aligning Contrastive Multiple Clusterings with User Interests

1 Introduction

Clustering is a mainstream unsupervised learning technique that groups samples into several disjoint clusters [Jain et al., 1999]. As data grows more complex, relying on a single clustering may no longer be sufficient. In many situations, different orthogonal solutions may exist to meaningfully group a given dataset [Bailey, 2018]. For example, a face image dataset can be clustered based on identity and pose (as shown in Figure 4), respectively. To address this problem, multiple clustering approaches have been explored to partition a given dataset in various ways [Yu et al., 2024].

The large majority of existing multiple clustering methods explore the data clustering structure by focusing on the data. One approach seeks alternative clusterings in a semisupervised and sequential manner, where each clustering complements already explored ones [Bae and Bailey, 2006; Chang et al., 2017], or simultaneously optimizes multiple sets of cluster centers or meta-clusterings in an unsupervised fashion [Tokuda et al., 2021; Yao et al., 2023]. In contrast, other techniques aim to explore non-redundant clusterings in different feature subspaces, including ISAAC [Ye et al., 2016], MSC [Hu et al., 2017], MISC [Wang et al., 2019], MVMC [Yao et al., 2019], and Nr-kmeans [Mautz et al., 2020], thus discovering diverse clusterings in linear subspaces. Other methods aim to obtain alternative clusterings in nonlinear subspaces generated by deep neural networks [Miklautz et al., 2020; Wei et al., 2020b; Wei et al., 2021; Ren et al., 2023b; Yao et al., 2023]. However, a critical point is that these approaches often assume the clusters are distinguishable as long as the data structure is known, which is not always the case. A one-to-many relationship exists in multiple clusterings, and relying solely on the data context may be insufficient to determine the appropriate clusterings. User expectations play a crucial role in the multiple clustering task, as they define the relevance and practical utility of the identified clusters.

Two main challenges exist in finding multiple clusterings that align with the user's interests. The first is identifying the significant features to represent user interests. This involves the extraction of effective and valuable information, typically embedded in high-dimensional spaces. The second challenge is how to align the user's interests with the clustering results. Although multiple clustering algorithms are capable of generating diverse clustering outcomes, users typically need to invest considerable effort to understand these results. Moreover, it is difficult to extract meaningful insights about the clustering semantics directly from the algorithmic framework.

To address these challenges, we propose the Contrastive Multiple Clusterings (CMClusts) approach, which leverages tailored data augmentation strategies to perform contrastive comparisons, thereby producing distinct clusterings aligned with user expectations. Targeted at the first challenge, we design an interest-guided data augmentation, to augment the original data to account for the user's different interests. As an example, the user's visual/semantic priors are typically readily available and easy to specify, e.g. we can intuitively select the most salient features in images, such as color, identity, and shape. To address the issue of insignificant features, the interest-guided data augmentation generates more diverse training samples by applying transformations on the original data. By capturing the invariant features under such transformations, we can effectively distill meaningful representations from the data. Feature compositions are complex and challenging to distinguish. By taking the original dataset as an anchor and contrasting it with the augmented data that preserves the desired features while altering others, we can guide the model to focus on the invariant features according with user interests. In addition, introducing augmented negative samples that break the invariance of the desired features and maximizing their distance from positive samples can yield more discriminative results. CMClusts formulates this process through the adoption of a clustering-level contrast that uses a triplet loss function to model the relationships between positive and negative samples (created through different data augmentations), aligning desired features with different interests. By minimizing the distance between anchors and positive samples, CMClusts ensures the effective extraction of desired features and the quality of clusterings, while maximizing the distance between anchors and negative samples guarantees the distinctiveness of different clusterings. As a result, CMClusts finds diverse clusterings of quality, and these clusterings align with user-interests.

The main contributions of our work are outlined below: (i) Our CMClusts establishes a principled integration of user-specified clustering interests with data augmentation techniques, employing contrastive learning between augmented and original data to disentangle and extract the desired features from the original feature space, thereby effectively aligning with user interests.

(ii) CMClusts introduces the clustering-level contrast that treats augmented data for the other clustering as negative samples. This contrast not only enhances the discrimination between positive and negative samples but also boost the diversity between different clusterings.

(iii) We conduct experiments using real-world benchmark datasets, and compare CMClusts against representative and competitive multiple clustering methods [Cui et al., 2007; Yang and Zhang, 2017; Miklautz et al., 2020; Ren et al., 2023b; Yao et al., 2023; Yao et al., 2024a]. Extensive results demonstrate the advantages of CMClusts in generating diverse clusterings aligning with user's interests.

2 RelatedWork

2.1 Multiple clusterings

Multiple clustering approaches aim to generate different alternative clusterings for more comprehensive analysis of the data [Bailey, 2018; Yu et al., 2024]. It differs from the multiview clustering [Zhang et al., 2020] and subspace clustering [Elhamifar and Vidal, 2013], which generate only one clustering from diverse views or different cluster subspaces.

Early solutions aim to find alternative clusterings in the original feature space [Yu et al., 2024]. Some of them take prior clusterings as the reference to sequentially generate the other alternative clustering in a semi-supervised manner [Yang and Zhang, 2017; Chang et al., 2017]. Hence the quality of alternative clustering depends on referenced ones. In contrast, other methods simultaneously find multiple clusterings by jointly optimizing quality and diversity by cluster centroids [Jain et al., 2008], mutual information [Dang and Bailey, 2010] and other criteria [Ren et al., 2023a]. Recently, [Yao et al., 2023; Yao and Hu, 2024] proposed two augmentation guided deep multiple clustering solutions (AugDMC and DDMC), which discover multiple clusterings through prototype-based representation learning and variational expectation-maximization framework, respectively. These methods learn feature consistency by disrupting the structural integrity of the original data, which is prone to introducing noise and semantic bias. Besides, [Yao et al., 2024a; Yao et al., 2024b] utilized the alignment capabilities of multi-modal large models (e.g. CLIP [Radford et al., 2021]) to align users' textual descriptions of preferences with corresponding images, thereby generating clustering results that align with user expectations. However, these large model based solutions require additional textual descriptions, as they are unable to effectively extract user interested features solely from the data itself, and need a higher computational cost with demanding multi-modal data.

Another line of methods explore diverse feature subspaces and clusterings therein, where some methods [Cui et al., 2007; Wang et al., 2019; Mautz et al., 2020] adopt orthogonal or independent subspace analysis to discover multiple clusterings, while others explore non-redundant clusterings by maximizing the feature statistical independence [Hu et al., 2017; Niu et al., 2013]. More recent solutions merge deep subspace learning to find multiple clusterings. ENRC [Miklautz et al., 2020] trains a non-redundant clustering network leveraging deep autoencoders. iMClusts [Ren et al., 2023b] diversifies multi-head attentions with redundancy control to generate diverse salient nonlinear subspaces and clusterings therein. [Wei et al., 2020a; Wei et al., 2020b; Wei et al., 2021] found non-redundant clusterings from multi-view data and heterogeneous networks. These deep multiple clustering methods often obtain better results than shallow ones. However, they may produce meaningless clusterings, as they disregard user's interests or prior knowledge of the data, causing the relationships between subspaces and the embodied clusterings are ambiguous.

2.2 Contrastive clustering

As an emerging paradigm of self-supervised learning, contrastive learning aims to project data into a new space to reduce intra-class distances and increase inter-class distances [Chen et al., 2020]. For this merit, it has been widely applied to clustering [Zhu et al., 2022; Yin et al., 2023]. For example, MiCE [Tsai et al., 2020] introduces the instancelevel contrastive learning to obtain discriminative representations to optimize the clustering, while [Zhong et al., 2021] further proposed the cluster-level contrastive framework to get more compact cluster. Subsequently, [Xu et al., 2022] proposed the contrastive multi-view clustering to solve the conflict between consistent public information and inconsistent private information of multi-view data.

These contrastive clustering methods can only generate a single clustering. In contrast, our CMClusts uses different data augmentations to align with user's interests, generating corresponding clustering results in the augmented data. By contrasting the clusterings across different augmented data, CMClusts facilitates clustering-level contrast to mine multiple clusterings with both quality and diversity.

3 Methodology

In this section, we elaborate on the proposed contrastive multiple clustering framework, including the user interest-guided data augmentation process, clustering-level contrastive feature extraction, and clustering loss function. The conceptual framework is presented in Fig. 1, and a notation table is included in the supplementary file.

3.1 User Interest-guided Data Augmentation

CMClusts aims to augment the original data to align with user's interests. Here, the user interests refer to the type of clusterings that users are expected or interested in. We consider image data as an example to illustrate the data augmentation process; other types of data can also be augmented using proper tools.

Given a dataset X ∈ RN×D , an image xi can be augmented through a function Aug(·) based on the user's interest set I = {I1, · · · , IK}:

where xk,i ∈ Rd corresponds to xi augmented with interest type Ik. In essence, CMClusts frames the desired features as consistency information between the augmented and the original data. While the choice of data augmentation techniques is tailored to specific applications. For example, if I = {color, shape}, CMClusts can take random rotation and color adjustment as augmentation techniques, which can preserve the invariant color features or shape context to align with user interests. Namely, rotation does not alter the color features, while color adjustment maintains the shape context. In this way, we can obtain K augmented representations {Xk}Kk=1 .

Remarks. Compared to the hard-to-obtain reference labels (reference clusterings), these alike user interests provide valuable clues for generating different and meaningful clusterings. To pursue knowledge-driven non-redundant clusterings, most methods resort to the explicit division rules, such as the link constraints and known clustering labels [Bae and Bailey, 2006; Yang and Zhang, 2017; Ren et al., 2023b]. These methods heavily depend on the quality of the reference information. Unfortunately, the generated clusterings maybe not of any interest to the users. In practice, the user's visual/semantic priors are generally available and more interesting. For example, we can intuitively select the most salient features in images, such as color, identity and shape. Similarly, by glancing conference papers, we can choose the interested clustering criteria (i.e., topics and authorship).

3.2 Clustering-level Contrastive Feature Extraction

After generating augmented data toward user interests, distinct deep neural networks {fθ1 (·), · · · , fθK (·)} are employed to extract feature representations Z ∈ RN×d from the augmented samples as follows:

Different deep networks facilitate the tailored design of feature extraction, ensuring adaptability to specific interests. Utilizing the same neural network for extracting diverse features may diminish the model's discriminative capability across different interests. Notably, CMClusts is not tied with any specific network architecture.

Unlike canonical contrastive clustering methods that minimizes the distance between different augmentations of the same instances [Li et al., 2021] (instance-level contrast) or minimizes the overall distance between instances within the same cluster [Zhong et al., 2021] (cluster-level contrast), CMClusts defines a clustering-level contrast that aims to maximize the diversity between clusterings of respective augmentations. This contrast can reduce the redundancy between clusterings. To compute the clustering-level contrast, we approximate it via a triplet loss as follows:

where zi is the i-th anchor sample in X, zj,i (j ̸= k) is the feature vector of the i-th sample in the j-th augmented data view. Typically, a single augmented view suffices. α is a margin that defines a minimum distance by which the negative sample should be farther from the anchor than the positive sample. The distance between zi and zk,i, which can also be computed with other metrics, is computed as:

For the first term of the triplet loss Ltri, as described in Section 3.1, the augmentation operations are determined based on the user's interests, ensuring the invariance of the desired features while altering other features. Therefore, by contrasting the original data X with the augmented data Xk and minimizing the distance between their feature embeddings, CMClusts learns the invariant features between the two, which correspond to the user's desired features.

For the second term of the triplet loss Ltri, increasing the distance between the anchor zi and zj,i in another augmented view increase the diversity between two clusterings. This is driven by the goal to explore different clusterings.

3.3 Clustering Loss Function

The clustering-level contrast mainly focuses on the diversity between different clusterings. CMClusts further designs a clustering loss to pursue the compact and separate relationship within and between clusters of a specific clustering, thereby improving the quality of respective clusterings. Specifically, the clustering loss contains the intra-cluster loss and inter-cluster loss. The intra-cluster loss pursues that samples within clusters are more compact, while the intercluster loss ensures that samples between clusters are more dispersed. The intra-cluster loss is computed as:

where xk,i and xk,j are samples of the l-th cluster in the k-th clustering. In addition, we ensure the separability between clusters by increasing the distance between the cluster centers, and define the inter-cluster loss as follows:

where cl1 and cl2 are centers of cluster ˜ C l1 k and ˜ C l2 k , respectively.

The total clustering loss incorporates both intra-cluster and inter-cluster losses, with a trade-off parameter β balancing these two losses as follows:

Minimizing the intra-cluster variance and maximizing the inter-cluster separation can promote the formation of wellstructured and semantically meaningful cluster representations, as our experiment will show.

3.4 Overall Objective

To learn the specific features of interest to the user across different augmented data views and generate high quality, diversified clusterings, we can jointly optimize CMClusts by combining the aforementioned losses as follows:

However, the computation of clustering loss is dependent on the initial clustering labels. Therefore, we adopt a tailored alternative optimization strategy. Initially, CMClusts is trained using the triplet loss Ltri to extract the desired clustering features. Then, k-means is employed to generate the initial clustering labels. After this, the model parameters are frozen, and the clustering center positions are adjusted by the clustering loss Lclu to obtain the final clustering labels.

Algorithm 1 lists the procedure of CMClusts. Specifically, Line 1 initializes the augmented data views {X}Kk=1 , Lines 2-6 iteratively optimize the parameters of θk and extract different sets of desired features {Hk}Kk=1 to align with user interests.It is worth noting that the contrastive loss is computed across clusterings, but only the feature extractor of the current clustering is updated; others remain fixed. Lines 9-11 optimize the clustering loss to generate K clusterings.

The time complexity of data augmentation is O(wh), where w and h are the width and height of the image, respectively. The time complexity of an m-layer feature extraction network is O(∑mi=1 di-1di), where di is the number of neurons in the i-th layer, d0 = D and dm = d. The time complexity of the triplet loss Ltri and clustering loss Lclu are both O(Nd). Therefore, the overall time complexity of CMClusts is O(∑mi=1 di-1di + 2Nd + wh). Compared with Multi-Sub [Yao et al., 2024a] using the CLIP pre-trained model, CMClusts does not require an additional text encoder and thus has a lower complexity and runtime.

4 Experiments

4.1 Experimental setup

Baselines. We compare CMClusts against representative multiple clustering methods, including OSC [Cui et al., 2007], MNMF [Yang and Zhang, 2017], ENRC [Miklautz et al., 2020], iMClusts [Ren et al., 2023b], AugDMC [Yao et al., 2023], and Multi-Sub [Yao et al., 2024a]. These compared methods were discussed in Section 2, and their configurations are deferred into the supplementary file.

Datasets. Seven benchmark datasets (ALOI [Geusebroek et al., 2005], Fruit [Yao et al., 2023], CMUFace [Ren et al., 2023b], COIL [Nayar, 1996], Cards [Yao et al., 2023], We-

Algorithm 1 CMClusts: Contrastive Multiple Clustering

Input: Dataset X; the number of clusterings K; the user's interest set I = {Ik}Kk=1 ; the training rounds {τk}Kk=1 for K clusterings;

Output: Clustering partition: {Ck}Kk=1 .

• 1: Initialize augmented data {Xk}Kk=1 based on I.
• 2: for k = 1 →K do
• 3: for τ = 1 → τk do
• 4: Update the latent representations Zk by Eq. (2).
• 5: end for
• 6: Compute triplet loss Ltri by Eq. (3).
• 7: Update the parameters of feature extraction network with Adam optimizer.
• 8: Extract the desired feature Hk: Hk = fθk (Xk).
• 9: Generate the k-th initial clustering ˜ Ck = kmeans(Hk).
• 10: Compute the intra-cluster loss Lintra via Eq. (5).
• 11: Compute the inter-cluster loss Linter via Eq. (6).
• 12: Update the cluster centers via Eq. (7).
• 13: end for
• 14: Generate K clusterings {Ck}Kk=1 .

bKB and Mice [Ren et al., 2023b]) are used to evaluate the performance of CMClusts and other baselines. The first five are image datasets, the sixth is a textual dataset, and the last is a mouse single-oocyte transcriptome dataset with two different clusterings (2 or 4 clusters). These datasets have been widely used to validate multiple clustering methods [Bailey, 2018; Yu et al., 2024]. Their statistical information are given in Table 1. More details are provided in the supplementary file. In experiments, we implement data augmentation by enriching features that are relevant to the desired clustering based on user's visual priors or interests. For example, we augment data with respect to color by varying shapes (i.e. rotation, flipping, random cropping) and ensure color invariance. For the expected shape clustering, we can augment data with respect to shape by varying color (i.e. hue, saturation, contrast changes) and keep the shape unchanged.

Evaluation metrics. To quantitatively evaluate the performance of each method, we measure the quality and diversity using the Normalized Mutual Information (NMI) and Jaccard Index (JI) with reference to distinct ground-truth labels. A higher score indicates the generated clustering more consistent with the groundtruth, which is with higher quality and more diverse from the other clustering.

Implementation. For image data, CMClusts adopts the pre-trained ResNet50 [He et al., 2016] as the backbone network for feature extraction, and defines a multilayer perceptron (MLP) with rectified linear units (ReLU) as the activate functions to project the learned features into a nonlinear representation. For text data, CMClusts integrates traditional text feature extraction techniques, such as the Bag-of-Words model and TF-IDF, to encode textual features, and then employs a multi-layer perception network (MLP) as the backbone for further feature extraction and representation learning. For single-cell Mice data, CMClusts obtains its feature matrix and maps it to a lower-dimensional space using an MLP. And the α and β are hyperparameters. For a detailed analysis, please refer to the supplementary file. Additionally, all the methods are implemented in PyTorch 2.4 and tested on a server with NVIDIA L40 GPUs. The source code of CMClusts is available at https://www.sduidea.cn/codes.php?name=CMClusts.

4.2 Results and analysis

Table 2 provides the average clustering results of 10 independent runs of each method on benchmark datasets, with the significantly best results are highlighted in bold (pairwise t-test with 95% confidence). We can find that CMClusts achieves the best results in almost all cases, proving its effectiveness. In addition, we have the following important observations: (i) Deep vs. Shallow methods: In most cases, deep multiple clustering methods, ENRC, iMClusts, AugDMC, Multi-Sub and CMClusts, perform better than the shallow OSC and MNMF. This clear gap confirms the advantage of deep learning based methods, owing to the expressive representation capability. The performance of CMClusts surpasses that of other methods due to its capability to effectively incorporate user interests, encoding these interests through diverse data augmentations. By leveraging contrastive learning among augmented features, the original features, and other irrelevant augmented features, CMClusts enhances the salience of desired features while promoting diversity across different augmented features. This capability is critical for generating diverse and high-quality clusterings. ENRC and iMClusts obtain competitive results in simple task like ALOI, but struggle with more complex tasks, which may be due to their inability to generate multiple orthogonal or non-redundant subspaces from large data with complex structures. AugDMC outperforms ENRC and iMClusts in complex scenarios such as Fruit and CMUFace, because the discriminative data view obtained by data augmentation is beneficial for generating multiple clusterings. However, AugDMC's mediocre performance on other datasets indicates the limitation of its simple adoption of prototype representation learning. Multi-Sub demonstrates some improvements over the aforementioned methods, reflecting the strong representational power of large multimodal models in capturing clustering structures. Nevertheless, it cannot cluster text data and transcriptomic datasets, and clustering is also challenging for data that is hard to generate textual cues (e.g., identity-based clustering in CMUFace). CMClusts gains competitive performance on text dataset WebKB and transcriptomic dataset Mice, highlighting the versatility of its data augmentation and clustering-level contrast.

(ii) With interest vs. Without interest: CMClusts outperforms other competitive baselines in most cases, which suggests the effectiveness of users' interest on boosting multiple clusterings. OSC and MNMF obtain the alternative clustering with reference to already explored clusterings, they have a poor overall quality and diversity. iMClusts rectifies each feature subspace by incorporating weakly-supervised prior knowledge, but these knowledge maybe not always accurate and available in complex data. Although ENRC projects data into different subspaces and provides alternative clusterings without requiring prior knowledge, the results are not good as CMClusts, due to the lack of redundancy control. Multi-Sub aligns image and textual features through a large multi-modal model, but it requires additional textual prompts, which may lead to suboptimal performance on datasets with a wide variety of categories and complex descriptions. In contrast, CMClusts leverages interest-guided data augmentations and clustering-level contrast to extract features aligned with user interests, without requiring additional textual prompts.

4.3 Visualization of Multiple Clusterings

To intuitively check the quality of clustering generated by CMClusts, we employ t-SNE [Van der Maaten and Hinton, 2008] to visualize the clustering space learnt by CMClusts and five baselines (excluding MNMF due to its poor performance) on the Fruit dataset as a two-dimensional scatter plot. The color clustering results are shown in Figure 2, and the species clustering results are provided in the supplementary, where different colors represent distinct cluster labels. We can observe that the cluster distributions produced by the baselines are more compact with small inter-cluster distances, making it difficult to obtain well-separated and distinguishable structures. In contrast, the clustering results of CMClusts exhibit clear inter-cluster boundaries and tightly distributed intra-cluster points, confirming that CMClusts can effectively generate higher-quality clusterings.

In addition, to vividly show the multiple clusterings generated by CMClusts and check the alignment between userinterests and the clustering outcomes, we present multiple clusterings guided by user-interests on the ALOI and CMUFace datasets in Fig. 3∼Fig. 4. Each clustering outcome corresponds to a specific type of clustering expected by the user. In Figure 3, the first clustering reflects the outcome under the user's color preference, where the color differences between clusters are distinct, while the shapes are mixed. The second clustering coincides with the user's shape preference. Here, the colors are more chaotic, but the shapes are clearly distinct. Similarly, as shown in Figure 4, CMClusts effectively identifies three clusterings based on identity, pose, and glass or not, corresponding to different user-defined criteria.

4.4 Ablation Study

To investigate the contributing factors of CMClusts, we introduce four variants: w/oAug, w/oContrast, w/oClusteringLoss, and wJointOpt, which respectively remove data augmentation in Eq. (1), contrastive learning in Eq. (3), clustering loss in Eq. (7), and the separate optimization strategy for the feature extraction networks in Eq. (8). Figure 5 presents the average NMI values of CMClusts and its variants. Similar trends are observed in JI, whose results are provided in the supplementary file. We find that CMClusts outperforms its variants, demonstrating that data augmentation, contrastive learning, clustering loss, and the separate optimization strategy are indispensable for generating high-quality clusterings with diversity.

The NMI of w/oContrast are generally the lowest, confirming the critical role of clustering-level contrast in capturing discriminative latent features from augmented data views. Solely the augmented data and feature extraction networks fail to produce satisfactory clustering results. Additionally, the performance of wJointOpt is noticeably inferior to CMClusts, highlighting that, compared to jointly optimizing the triplet loss and clustering loss, optimizing the network with the triplet loss to obtain effective initial clustering labels and then fine-tuning with the clustering loss gives better clustering performance.

We also observe that removing data augmentation leads to a significant drop in NMI. This is because data augmentation generates diverse enhanced views from complex data, reflecting user's respective interests. Without data augmentation, only using original data considerably reduces the diversity of clustering results. Furthermore, the clustering loss also plays an important role in fine-tuning the clustering outcome with moderate improvement. In summary, these factors enable CMClusts to achieve high-quality multiple clusterings.

In addition, we investigate the hyperparameter sensitivity of CMClusts and provide the results in the supplementary file. The results indicate that the performance of CMClusts is not greatly impacted by hyperparameters, and the fluctuation range of NMI is less than 0.1.

5 Conclusion and FutureWork

This paper introduces CMClusts, a novel contrastive multiple clustering solution aligning with user interests. CMClusts fulfils user interests by extracting desired features from augmented data, and explores multiple clusterings with diversity by clustering-level contrast. CMClusts outperforms competitive baselines in both quality and diversity on benchmark datasets. The explored distinctive clusterings indeed accord with different user interests. Next, we will integrate foundational models with multiple clusterings to enhance the applicability of multiple clustering across broader domains and validate its generalizability in real-world tasks.

Acknowledgments

This work is supported by National Key Research and Development Program of China (No. 2023YFF0725500), NSFC (62272276 and 62432006), Shandong Provincial Natural Science Foundation (No. ZR2024JQ001), Taishan Scholars Program (No. tsqn202306007 and tsqn202408317).

Contribution Statement

Shan Zhang and Liangrui Ren contributed equally to this work and are designated as co-first authors (indicated by †). Guoxian Yu, identified as the corresponding author (denoted by ∗), provided overall conceptual guidance, project supervision, and coordination. Jun Wang, Yanyu Xu, and Carlotta Domeniconi made substantial contributions to the study design, experimental implementation, and manuscript refinement.