Mixture of Clustering Experts with Dual Consistency for Multi-View Clustering

1 Introduction

Multi-view clustering [Chao et al., 2021; Fang et al., 2023b; Zhou et al., 2024] aims to uncover latent semantic structures by integrating complementary information from multiple heterogeneous data sources. Compared to single-view approaches, it can more fully exploit both the consistency and complementarity across different views and achieve more accurate clustering [Wang et al., 2023], with wide applications in computer vision, multimedia analysis, and information retrieval.

∗Corresponding author

In recent years, multi-view clustering has achieved significant progress owing to the powerful nonlinear representation learning capability of deep learning. Autoencoder-based methods [Yang et al., 2021; Xu et al., 2021; Yan et al., 2023; Bai et al., 2024] learn latent representations frommulti-view data and integrate cross-view information through feature fusion mechanisms. Graph-based approaches [Wang et al., 2021; Huang et al., 2023; Ling et al., 2023] further explicitly model the relational structure among samples by fusing multi-view graph structures to capture cross-view similarities, thereby enhancing clustering discriminability. In addition, contrastive learning-based methods [Ke et al., 2021; Pan and Kang, 2021; Feng et al., 2024] design cross-view contrastive objectives to enforce representation consistency of semantically identical samples across different views, facilitating the learning of discriminative features. For example, [Zhang et al., 2019] designed a nested autoencoder framework to perform view-specific representation learning and multi-view information encoding within a unified framework. [Feng et al., 2024] introduced view-specific contrastive encoders and generated pseudo labels on fused representations, utilizing these pseudo labels to guide the positivenegative pair selection in contrastive encoders. By effectively exploiting both the consistency and complementarity among views, these deep multi-view clustering methods demonstrate remarkable clustering performance.

However, most existing multi-view clustering methods characterize each semantic cluster using a single prototype and assume a compact cluster assignment. Some representative approaches [Xu et al., 2022b; Chen et al., 2023; Yang et al., 2021; Yan et al., 2025] introduce an explicit clustering layer on top of learned representations and perform end-to-end optimization. Although such designs are structurally simple and efficient, they essentially assume that each semantic cluster can be adequately represented by a single centroid. However, data distributions often exhibit diversity within clusters. As illustrated in Figure 1, samples belonging to the same semantic category may contain multiple latent substructures. Modeling such complex distributions with a single prototype is therefore inadequate to capture the underlying intra-cluster substructure, which inevitably limits clustering performance. To alleviate this limitation, several graph-based methods [Zhang et al., 2024; Chen et al., 2025] attempt to enhance intra-cluster modeling by constructing sample-to-anchor graphs. Nevertheless, their performance heavily depends on the quality of graph construction, which is often sensitive to noise. Moreover, graph structures are usually built independently for each view, making it difficult to ensure semantic consistency across views. Therefore, how to effectively model complex intra-cluster structures while simultaneously preserving semantic consistency across multiple views remains a challenge.

To address the challenge, we propose a Mixture of Clustering Experts with Dual Consistency for Multi-View Clustering termed MoCE-DC. The framework is shown in Figure 2. Specifically, MoCE-DC introduces a set of learnable clustering experts to collaboratively model complex cluster structures. Each expert captures complementary semantic characteristics within a shared prototype space, enabling the model to decompose intricate clusters into multiple cooperative substructures. To facilitate adaptive expert utilization, a gating mechanism is employed to dynamically weight expert contributions according to view-specific representations, allowing each sample to be clustered through the most relevant expert combinations. To further ensure consistency across views, MoCE-DC incorporates a dual-level consistency alignment mechanism. At the sample level, cross-view prediction consistency is enforced by aligning the cluster assignment of each sample with the consensus of its assignments in the other views. At the global level, a consensus target distribution is constructed by aggregating multi-view information, which serves as a unified semantic reference to guide all views toward a consensus clustering structure and effectively mitigate semantic drift. In addition, to alleviate the commonly observed expert collapse issue in mixture-of-experts models, we introduce a gating balance regularization that promotes balanced expert activation and encourages cooperative expert learning. Collectively, these components enable MoCE-DC to simultaneously refine intra-cluster substructure modeling and enforce cross-view semantic consistency.

The main contributions of this work can be summarized as follows:

1. We propose a novel multi-view clustering framework, termed MoCE-DC, which explicitly models intra-cluster diversity by introducing multiple cooperative clustering experts. MoCE-DC enables different experts to capture complementary substructures within the same semantic cluster, significantly enhancing the expressive capacity for complex clusters.
2. We propose a dual-consistency alignment strategy to en-

hance cross-view semantic consistency. By jointly enforcing sample-level prediction consistency and global semantic agreement across views, MoCE-DC effectively reduces cross-view semantic discrepancy and stabilizes the clustering process.

1. Extensive experimental results on several benchmark datasets demonstrate that the proposed method consistently outperforms existing state-of-the-art multi-view clustering approaches.

2 Method

In this section, we introduce our proposed method MoCEDC. First, we introduce the key notations employed throughout this paper. Consider a multi-view dataset denoted as {Xv ∈ RN×dv }Vv=1 , where N denotes the number of samples, V represents the number of views, dv denotes the sample dimension of the v-th view, and xvi denotes the v-th view of the i-th sample. The goal of multi-view clustering is to partition the N samples into C clusters by effectively fusing information from all V views.

To achieve this, we first project the multi-view data into a low-dimensional latent space using a set of view-specific deep autoencoders. Formally, for the v-th view, we construct an encoder network Dv e and a decoder network Dv d , parameterized by θv and ϕv , respectively. Given an input sample xvi , its latent representation is derived as zvi = Dv e (xvi ; θv ). These autoencoders are trained by minimizing a reconstruction loss, which ensures that the essential information of each view is preserved in the latent space:

2.1 Mixture of Clustering Experts

To explicitly model the inherent diversity and latent substructures within clusters, we introduce a Mixture of Clustering Experts module (MoCE). This module associates each cluster with a set of specialized experts, with each expert dedicated to capturing a latent substructure present within the cluster. Concurrently, we design a learnable gating network that dynamically routes samples to relevant experts, enabling the model to adaptively compose complex clusters from these elementary substructures.

Formally, given the view-specific latent representations {Zv }Vv=1 , the MoCE module generates view-specific soft cluster assignments Qv ∈ RN×C by adaptively aggregating the predictions of M experts:

where Evm ∈ RN×C denotes the soft cluster assignment produced by the m-th expert for view v, and Gvm ∈ RN×1 represents the corresponding sample-wise gating weights. The Hadamard product ⊙ applies the gating weights to expert predictions in a sample-wise manner. Under this formulation, samples belonging to the same cluster can be jointly explained by several experts with varying activation strengths, enabling MoCE to capture complex cluster geometries.

Shared-Prototype-Based Clustering Experts. To explicitly model diverse clustering behaviors while preserving cross-view consistency, MoCE introduces a set of shared clustering prototypes {µk}Ck=1 , which are shared across all experts and views. These prototypes serve as common semantic anchors, enabling different experts to produce distinct clustering patterns while remaining aligned in a unified prototype space.

Given a sample i from view v, the m-th expert first transforms the representation via an expert-specific function fm(·) and then computes its soft assignment to each prototype. The Student's t-distribution is adopted to measure the similarity between the transformed feature and each prototype. The resulting soft assignment is given by

Here, fm(·) captures expert-specific clustering preferences, while the shared prototypes enforce semantic alignment across experts and views. By aggregating Evimk over all samples, we obtain the expert clustering output Evm.

GatingNetwork for Expert Selection. To adaptively combine multiple clustering experts, MoCE employs a shared gating network πψ(·) to generate expert-wise routing weights. For each sample i and view v, the gating weight assigned to expert m is defined as:

This gating mechanism enables the model to dynamically select the most relevant experts for each input. To further improve routing sparsity and computational efficiency, a top-K routing strategy is adopted, where only the K experts with the highest routing probabilities are activated.

2.2 Dual-Level Multi-ViewConsistency Learning

Although MoCE enhances intra-cluster modeling by allowing multiple experts to capture diverse substructures, this increased flexibility may amplify semantic discrepancies across views in unsupervised multi-view settings. To address this issue, we propose a dual-level consistency learning framework, which enforces alignment from both the sample level and the global semantic level. The sample-level mechanism ensures balanced and consistent cluster assignments across views, while the global-level supervision provides a unified semantic anchor to prevent accumulated view bias.

At the sample level, we aim to align clustering assignments across views while avoiding degeneration of the shared clustering prototypes. To achieve this, we introduce a balanced assignment strategy combined with an alternating prediction mechanism. Specifically, for each view v, we compute a balanced assignment ˜Qv ∈ RN×C by solving the following entropy-regularized optimization problem:

where ⟨·, ·⟩ denotes the matrix inner product, H(Q) is the entropy regularization term, and ε is a hyperparameter. We have provided a more detailed optimization process in supplementary materials. This formulation enforces balanced cluster utilization, ensuring that each cluster receives a comparable number of samples. The resulting distribution is smoother and more stable, facilitating reliable cross-view alignment.

Based on the balanced assignments {˜Qv }Vv=1 , we adopt an alternating prediction strategy in which each sample's soft cluster assignment in one view is aligned with the average of its assignments in the other views. For each view v, a consensus target is constructed by averaging the balanced predictions from all other views:

The alternating prediction loss then minimizes the crossentropy between the soft cluster assignment Qv and this target ¯Q-v :

Through this alternating alignment mechanism, clustering predictions from different views are progressively aligned, while avoiding direct pairwise matching that may propagate noisy errors.

While sample-level alignment improves local consistency, clustering decisions based solely on individual views may suffer from view bias, as the view-specific representation zv captures only information from the v-th view. To provide a more discriminative and view-invariant semantic reference, we further introduce a global target distribution from the global multi-view representation.

For each sample xi, we form a global representation by concatenating its view-specific representations:

where D = ∑Vv=1 dv. Let H = [h1, h2, . . . , hN ]⊤ ∈ RN×D denote the global representation. We then performK-means clustering on H to obtain global cluster centroids {cj}Cj=1 . The soft cluster assignment of hi to the j-th global cluster is computed via the Student's t-distribution:

yielding a soft assignment matrix S ∈ RN×C . To emphasize high-confidence assignments, we sharpen S into a target distribution P as follows:

The global target distribution P provides a unified supervisory signal that encodes the consensus clustering structure across all views.

We enforce each view's cluster assignment Qv to align with P via a global consensus alignment loss, defined as the sum of Kullback-Leibler (KL) divergences:

This loss encourages the view-specific clustering assignment Qv to align with the consensus semantics encoded in P, thereby reducing view-specific bias.

The overall clustering objective is formulated by jointly minimizing the sample-level alternating prediction loss and the global consensus alignment loss:

2.3 Gating Load Balance

During training, mixture-of-experts (MoE) models often suffer from expert load imbalance: a large fraction of input instances are consistently routed to only a few experts, while the majority remain underutilized. In multi-view clustering, this issue is exacerbated by the absence of ground-truth labels, as the gating network tends to converge to degenerate solutions that over-rely on a sparse subset of experts, thereby undermining model capacity and assignment diversity.

To mitigate this, we introduce an expert load balancing regularization term. Consider a model with M experts and a dataset of N samples across V views, let T = N ×V denote the total number of sample-view pairs. For each expert m, we define its routing frequency as:

where gv i = [gv i1 , . . . , gv iM ]⊤ denotes the gating distribution for sample i in view v, and TK(gv i ) returns the indices of the top-K experts with the highest routing probabilities. fm measures how often each expert is actually activated under sparse top-K routing. In parallel, we compute the mean routing probability assigned to expert m:

While fm captures hard activation statistics, pm reflects the soft routing tendency of the gating network. Based on these two complementary statistics, we formulate the gating load balancing loss as:

This objective penalizes skewed routing patterns by assigning stronger regularization to experts that are both frequently activated and assigned high routing probability. Overused experts receive stronger corrective signals, whereas underutilized experts are penalized less aggressively.

Algorithm 1 Training Procedure for MoCE-DC

Input: Multi-view data {Xv }Vv=1 , number of clusters C, number of experts M, hyperparameters α, β.

Output: Clustering results {yi}Ni=1 .

• 1: Pretrain autoencoders for each view by minimizing the reconstruction loss Eq. (1).
• 2: while not converged do
• 3: Update target distribution P using current features by Eq. (10).
• 4: for fixed target distribution P do
• 5: for each view v = 1 to V do
• 6: Extract latent representation Zv ←Dv e (Xvi ).
• 7: Compute expert outputs Evm via Eq. (3).
• 8: Compute routing weight Gvm via Eq. (4).
• 9: Compute view-specific soft clustering assignment Qv via Eq. (2).
• 10: end for
• 11: Compute total loss: L←LR +αLC +βLB.
• 12: Update all parameters via backpropagation.
• 13: end for
• 14: end while
• 15: return {yi}Ni=1 .

2.4 Joint Optimization Objective

The overall training objective of MoCE-DC integrates three complementary components:

where LR denotes the multi-view reconstruction loss, LC is the multi-view clustering loss, and LB is the expert load balancing loss. The hyperparameters α, β > 0 control the relative importance of each term. When the network training is completed, the final clustering result can be obtained by

The training procedure is summarized in Algorithm 1.

Complexity Analysis. Let N, V , C, M, |B|, and dh denote the number of samples, views, clusters, experts, batch size, and latent dimension, respectively. The per-batch computational cost consists of (i) view-specific encoding at O(|B| ∑v=1 dvdh), and (ii) MoCE routing and prototype assignment at O(|B|VMCdh). Since the gating, aggregation, and consistency losses are asymptotically dominated by these terms, the per-batch complexity simplifies to O(|B|(∑v=1 dvdh + VMCdh)). Consequently, the perepoch training cost scales linearly with N. The periodic Kmeans update for the target distribution incurs an additional O(TkNCV dh) per update cycle, which remains negligible due to its infrequent execution.

3 Experiments

In this section, we compare our method with the state-of-theart clustering methods on benchmark datasets. The effectiveness of the method is demonstrated through extensive experimentation.

3.1 Datasets

To validate the effectiveness of MoCE-DC, we conduct experiments on seven benchmark datasets as shown in Table 1. BDGP [Cai et al., 2012], Caltech-5V [Fei-Fei et al., 2004], OutdoorScene [Fang et al., 2023a], CiteSeer [Fang et al., 2023a], Handwritten [Du et al., 2025], NUSWIDEOBJ [Zhao et al., 2025], and CIFAR10 [Yan et al., 2025]. We further construct Caltech-4V, Caltech-3V, and Caltech-2V datasets based on Caltech-5V to evaluate the robustness of MoCE-DC.

3.2 Experimental Setup

We compare MoCE-DC with two single-view clustering algorithms and ten state-of-the-art multi-view clustering algorithms. The two single-view clustering algorithms are Kmeans [McQueen, 1967] and IDEC [Guo et al., 2017]. The multi-view clustering methods can be broadly categorized into two groups. The first group comprises graph-based multi-view clustering methods, including CAMVC [Zhang et al., 2024], DMAC [Wang et al., 2025], GMVC [He et al., 2025], ALPC [Chen et al., 2025]. The second group includes end-to-end multi-view clustering approaches with explicit clustering layers, including SDMVC [Xu et al., 2022a], MFLVC [Xu et al., 2022b], CVCL [Chen et al., 2023], DealMVC [Yang et al., 2023], ADMC [Zhao et al., 2024], SSLNMVC [Yan et al., 2025].

For MoCE-DC, the autoencoder consists of an encoder with layer sizes 2000-500-500-dh and a decoder with layer sizes dh-500-500-2000. The low dimension dh is set to 10. The batch size is set to 512, and training is performed for up to 10,000 epochs or until early stopping is triggered. For other compared methods, we use the public codes and default settings from the original papers to conduct experiments. All experiments are conducted on an NVIDIA GeForce RTX-3090 GPU. The code is available at https://github.com/Systemclock/MoCE-DC.

We apply three wildly used metrics to evaluate the final clustering performance, including clustering accuracy (ACC), Normalized Mutual Information (NMI), Adjusted Rand Index (ARI). For all methods, we ran ten independent runs and reported the mean values.

3.3 Experimental Results

As shown in Table 2 and Table 3, MoCE-DC achieves the best or second-best clustering performance on all seven multi-view datasets, demonstrating strong stability and generalization ability. First, compared with single-view clustering methods, MoCE-DC consistently yields superior results, as it effectively exploits the complementary information across multiple views. Second, unlike methods that obtain cluster assignments through a single MLP-based clustering head, MoCE-DC explicitly models intra-cluster substructures by introducing multiple collaborative experts and adaptively derives the final cluster assignments, which enhances its capability to capture intra-cluster diversity. Finally, although graph-based methods can also reveal latent substructures within clusters to some extent, their performance heavily depends on the quality of graph construction. By contrast, MoCE-DC achieves cross-view alignment through a hierarchical consistency constraint, leading to more robust and improved clustering performance.

3.4 Ablation Study

Table 4 reports the ablation study results, evaluating the contribution of each loss component. First, when only the reconstruction loss LR is used, the clustering performance is extremely poor, indicating that representation learning alone is insufficient to induce meaningful clustering structures. Second, by directly supervising the formation of clustering assignments, the clustering loss LC enables the model to learn more discriminative representations. Third, the gating balance regularization LB can also improve clustering performance. Overall, the best performance is consistently achieved when all loss terms are jointly optimized, confirming the necessity of jointly modeling clustering supervision and consistency alignment in the proposed MoCE-DC framework.

3.5 Parameter Analysis

For the proposed MoCE-DC framework, we investigate the sensitivity of key parameters. Specifically, α and β are selected via grid search from the set {10-3 , 10-2 , 10-1 , 100 , 101 , 102 }. The number of experts is chosen from {C, 10, 15, 20, 25}. As shown in Figure 3, MoCE-DC exhibits relatively stable clustering performance across a wide range of α and β values, indicating that the proposed framework is not overly sensitive to the precise choice of these hyperparameters. As shown in Figure 4, the clustering performance reaches its peak when the number of experts M is set to approximately 10 or 15. IncreasingM beyond this range results in degraded performance, because an excessive number of experts introduces redundancy and weakens effective expert cooperation. Therefore, we set M within the range of 2C to 3C to strike a balance between model capacity and clustering stability.

3.6 Convergence Analysis

As shown in Figure 5, we visualize the clustering metrics and overall loss throughout training to validate the convergence. The total loss drops quickly and then plateaus, and the clustering accuracy correspondingly converges to a stable level. When the target distribution is updated, a temporary increase in the loss can be observed. However, the model subsequently converges to a better clustering solution, indicating that the global target distribution effectively guides the views toward learning more accurate clustering assignments.

3.7 Visualization Validation

As shown in Figure 6, we perform t-SNE visualization of the learned global representations, where each sample is colored according to its final clustering result. The visualization shows clear inter-cluster boundaries and compact intracluster distributions on both BDGP and Caltech-5V. This demonstrates that MoCE-DC effectively enhances the discriminability of the learned representation and produces reliable clustering assignments.

4 Conclusion

This paper proposes a novel multi-view clustering framework, termed MoCE-DC, which integrates a mixture of clustering experts with a dual-level consistency constraint. MoCE-DC decomposes complex cluster structures by leveraging multiple experts to capture diverse sub-cluster patterns within each view. To ensure cross-view consistency, MoCEDC adopts a dual-level alignment strategy, including samplewise cross-view prediction and global consensus regularization. Extensive experimental results demonstrate the effectiveness of the proposed method.

Acknowledgments

This work was supported by the National Key Research and Development Program of China (No. 2022ZD0160402), the National Natural Science Foundation of China (No. 62201471), and Aeronautical Science Fund (No. ASFC20240001053006).

Contribution Statement

All authors made contributions to the paper's methodology and experiments. Daidai Zhu and Yang Zhao contributed equally to this work and should be considered co-first authors. Yang Zhao is the corresponding author.