Bridging Inter-View and Client Heterogeneity: Federated Multi-View Clustering Under Non-IID Data

1 Introduction

With the rapid advancement of sensor technologies and the internet [Driess et al., 2023; Fei et al., 2022; Wang et al., 2025a; Zhang et al., 2025], a multitude of distributed clients are now capable of collecting unlabeled data from multiple perspectives or modalities [Cui et al., 2023]. While fully leveraging these data, preserving data privacy among clients has become a critical requirement, giving rise to Federated Multi-View Clustering (FedMVC [Chen et al., 2023])-an emerging research direction that enables multiple clients to collaboratively learn a unified clustering model without sharing raw data. Such models can be widely applied in real-world scenarios such as recommendation sysc) non-iid client data

∗Corresponding author.

(

(d) non-iid client data

Figure 1: This figure shows the data distribution of the MNISTUSPS dataset under different conditions.

tems and medical diagnosis [Che et al., 2022], thus attracting increasing attention [Chen et al., 2025a; Chen et al., 2025b; Qin et al., 2025; Feng et al., 2025a; Ren et al., 2026; Ren et al., 2025].

Existing FedMVC methods rely on idealized assumptions such as client homogeneity and independently and identically distributed (IID) data-all clients are either single-view or multi-view, with complete category coverage and balanced sample distributions. HFMVC [Jiang et al., 2024] assumes a "single-view, multi-client" setup: each client has data from one view, possibly shared across clients, with roughly balanced sample sizes but differing category compositions. In contrast, FMCSC [Chen et al., 2024] uses a "multi-view, multi-client" configuration: each client holds multi-view features of non-overlapping samples, with balanced sizes and full category coverage (Figure 1(b)). While these methods partially address non-IID data, they operate under mild heterogeneity and are unsuitable for more complex real-world settings. In challenging scenarios, clients may have access to either a single view or multiple views and are subject to multiple non-IID factors, such as incomplete category coverage, significant disparities in sample size, and severe class imbalance (Figure 1(c)-(d)). These compounded heterogeneities hinder effective alignment and integration of distributional differences, causing performance drops.

Unlike the aforementioned studies [Jiang et al., 2024; Chen et al., 2024], real-world FedMVC scenarios involve heterogeneous view structures and non-IID category distributions. Single-view and multi-view clients coexist, with inconsistent view availability across clients. Local data often has incomplete category coverage, severe class imbalance, or long-tail distributions. In healthcare federated learning, for instance [Hudaib et al., 2025; Johnson et al., 2016; Esteva et al., 2019; Sheller et al., 2018], hospitals typically focus on distinct clinical tasks such as cancer, Alzheimer's, or COVID-19 and their datasets exhibit significant class imbalance, limited category coverage, and large sample size variations, all impairing model convergence and generalization.

The presence of heterogeneous mixed-view data under non-IID constraints limits the applicability of many conventional FedMVC methods. These challenges can be broken down into two core issues: (a) Inter-view heterogeneity: Views collected by different clients vary significantly in terms of information type, quality, and expressive capability. For example, images typically contain rich visual structural information, while text emphasizes semantic expression, making it particularly challenging to effectively extract and fuse complementary information from heterogeneous views. (b) Client heterogeneity (non-IID issues): Different clients exhibit high inconsistency in category coverage and sample size (e.g., specialized hospitals in medical scenarios possess markedly different case types and volumes [Hudaib et al., 2025]), posing serious challenges to cross-client collaborative modeling and joint optimization.

To address these challenges, this paper proposes a novel method termed Bridging Inter-View and Client Heterogeneity: Federated Multi-View Clustering under NonIID Data (H2 -FedMVC), which aims to uniformly leverage single-view and multi-view data from heterogeneous clients to uncover clustering structures under mixed-view settings. As illustrated in Figure 2, we first observe that in the absence of unified supervision, directly aggregating local models can easily lead to model misalignment, while reliable cluster centers are difficult to establish at early training stages. To this end, we design a cross-client consensus pretraining mechanism to align local models and determine reliable cluster centers for each sample, thereby effectively alleviating model bias. Second, to handle view discreipancies, we introduce a leader-guided learning mechanism. At the individual level, the cluster center associated with each sample acts as a "leader" to guide contrastive updates; at the social level, clients learn from structurally similar and betterperforming peer models to enable knowledge transfer. Third, to address the client non-IID issue, we propose an expert adaptive aggregation strategy, which dynamically adjusts aggregation weights based on client importance and model performance, enhancing accuracy while improving the stability and lightweight nature of the global model. These learning and aggregation strategies work collaboratively to significantly improve clustering performance in complex heterogeneous environments.

The main contributions of this paper are as follows:

1. A FedMVC framework is proposed to address view heterogeneity and non-IID issues in mixed-view scenarios.
2. A leader-guided learning mechanism and an expertadaptive aggregation strategy are designed to address view and client heterogeneity, marking the first application of Mixture of Experts (MoE) to FedMVC.
3. The superior performance of H2 -FedMVC is verified in various federated scenarios. Experimental results demonstrate its excellent performance under ideal conditions as well as its robust and leading clustering performance even under complex non-ideal conditions.

2 RelatedWork

1. Homogeneous FedMVC. This setting assumes all clients have the same view configuration, such as all being single-view or all being multi-view. This alignment enables consistent model structures and representation spaces, supporting parameter aggregation or representation alignment. Methods like HFMVC [Jiang et al., 2024] further introduce a heterogeneity-aware mechanism to adaptively assess data distribution differences and extract cross-client consistency and complementarity via contrastive learning under vertical FedMVC [Chen et al., 2024].

(2) Heterogeneous FedMVC. This setting allows singleview clients and multi-view clients to coexist, thus being closer to real application scenarios [Tian and Bennis, 2025]. FMCSC [Chen et al., 2024] aims to integrate local and global learning via mutual information in federated settings, address cross-client and cross-view discrepancies, and robustly mine clustering structures from heterogeneous multi-view data.

Although existing methods have achieved certain results, the real FedMVC scenarios still face the challenges of heterogeneous mixed views, including the non-independent and identically distributed category distribution of clients, the structural heterogeneity of the coexistence of single-view and multi-view clients, and the uncertainty of the number and quality of views during the training process. To address the above complex scenarios, the H2 -FedMVC proposed in this paper can be regarded as a more general variant of heterogeneous FedMVC, which can handle the above complex scenarios by bridging the differences between clients and the gaps between views.

3 Methodology

3.1 ProblemDefinition

We consider a federated learning scenario where the data are non-IID. Suppose the dataset is {(x1i , x2i , . . . , xVi , yi) | xvi ∈ RDv , yi ∈ {1, . . . ,K}}Ni=1 , where V is the number of views, K is the number of classes, and N is the number of samples. Here, xvi denotes the v-th view of the i-th sample, RDv is the feature space with dimensionality Dv for that view, and yi is the class label of the i-th sample.

For client α, its local dataset is Dα = {(x1αi , x2αi , . . . , xVαi , yαi ) | xvαi ∈ RDv , yαi ∈ Cα}Nα i=1 , where Cα ⊆ {1, . . . ,K}

and |Cα| = c, indicating that client α contains only c classes out of theK total classes. Let nα,k = |{xαi | yαi = k}|, k ∈ Cα denote the number of samples in client α belonging to class k, and let Nα = ∑k∈Cα nα,k be the total number of samples on client α. For any k1, k2 ∈ Cα, we have |nα,k1 - nα,k2 | ≫ 0, and for another client β, |Nα -Nβ| ≫ 0.

Assume there are in total M + P clients, including M multi-view clients and P single-view clients. We train {f 1 (·;w1 ), . . . , f M (·;wM ), . . . , f M+P (·;wM+P )} and mine complementary clustering structures. Here, f α(·;wα) denotes the expert model produced by client α.

3.2 Cross-Client Consensus Pre-training

Multi-view datasets typically contain redundancy and random noise. Most existing methods employ self-supervised autoencoder models, such as the auto-encoder (AE [Hinton and Zemel, 1993]) and variational auto-encoder (VAE [Kingma and Welling, 2013]), to learn high-level representations from raw data. In H2 -FedMVC, we adopt an encoder-decoder pair for each view, denoted as Evϕv (·) and Dv θv (·), with parameters ϕv and θv , respectively. For a multi-view client α, we assume its dataset size to be |Mα|. We define zvi = Evϕv(xvi ) as the latent representation of the v-th view of the i-th sample, and the auto-encoder output as ˆxvi = Dv θv(zvi ) ∈ RDv . The reconstruction loss is computed between the input xvi and the reconstructed output ˆxvi . Furthermore, the local model can be pre-trained by minimizing the following objective:

Similarly, for a single-view client p, we assume its dataset size to be |Sβ|; thus, it suffices to construct a single encoderdecoder pair. The pre-training objective Lβr can adopt the same form as defined in Eq. (1).

Meanwhile, since in the early stage of training, each expert model has difficulty in finding the corresponding cluster center for each cluster. Inspired by previous works of MVC [Wu et al., 2024; Xu et al., 2023; Xu et al., 2022; Chen et al., 2024], we introduce the collaborative contrastive learning to optimize the clustering structure of each client. It takes the low-level features obtained by the autoencoder {(z1i , z2i , . . . , zVi )}|Mα| i=1 and obtains high-level features {(h1i , h2i , . . . , hVi )}|Mα| i=1 by stacking V nonlinear mappings {H1 (Z1 ;Ψ1 ), . . . ,HV (ZV ;ΨV )}. In addition, a nonlinear mapping H(Z;Ψ) : R∑Vv=1 dv → Rd is used to construct the final representation,

where {hi}|Mα| i=1 = H ∈ R|Mα|×d , and Z ∈ R|Mα|×∑Vv=1 dv . We aim to preserve the representative capacity of low-level features to prevent model collapse while learning the common semantics H among views in the highlevel feature space.

In the high-level feature space, the common semantics H are learned from all views and should be very similar to the common semantics learned from individual views. Based on this, we define {hi, hvj }v=1,...,V j=i as V positive feature pairs, and the remaining {hi, hvj }v=1,...,V j̸=i are V (|Mα| - 1) negative feature pairs. Then, we use cosine similarity to measure the similarity of feature pairs:

where ⟨·, ·⟩ is dot product operator. We introduce the temperature parameter τα to moderate the effect of similarity. Subsequently, the feature contrastive loss is formulated as:

Moreover, multi-view clients aim to assist single-view clients in bridging client gaps and discarding view-private information detrimental to clustering. Concretely,

where f vα (·;wvα) represents the local model that can handle the v-th type of view, which is initialized by the global model f v g (·;wv ). fα(·;wα) represents the local model of multiview client α.

For single-view client β, where each sample has only one view, we design model contrastive learning to achieve consistency. Specifically, client β contains data of the v-th view type {xvi }|Sβ| i=1 , with its local model as fβ(·;wv ). To capture common semantics, we use the same approach as for multi-view clients: two non-linear mappings, Hv (Zv ;Ψv ) and H(Z;Ψ), generate high-level features Hv and common semantics H. The global model fg(·;wv ) after aggregation further enhances its ability to learn generalized common semantics. To align the local model with the global model, we define the model contrastive loss:

where {hgi }|Sβ| i=1 represents the output of the local data after being processed by the global model, and τβ denotes the temperature parameter.

During training, the respective total losses for multi-view client m and single-view client β are:

where Lαr and Lβr are used as reconstruction losses to learn representations for each view individually. Meanwhile, Lαc and Lβc are employed to discover common semantics across views, facilitating the exploration of complementary clustering structures across clients.

3.3 Leadership-Guided LearningMechanism

The heterogeneities between views can be divided into intra-client view heterogeneities and inter-client view heterogeneities. For this purpose, inspired by the particle swarm optimization (PSO [Kennedy and Eberhart, 1995]), we have designed the Client-level Leadership-Guided Learning Mechanism (Individual learning) and the Model-level LeadershipGuided Learning Mechanism (Social learning).

Client-level Leadership-Guided Learning Mechanism

An ideal clustering structure requires the maximum intracluster distance to be smaller than the minimum inter-cluster distance [Bengio et al., 2013]. In such cases, samples within the same cluster naturally converge toward their centroid. For samples of the same category [Kulis, 2013], the angular separation between their refined features h should be minimized, whereas for samples from different categories, it should be maximized.

From a collective view, a cluster can be seen as a population with its centroid acting as a "leader" that captures shared characteristics and serves as a prototype. However, most contrastive learning methods use individual samples as comparison units [Chen et al., 2024; Jiang et al., 2024]. Although these methods are effective in capturing discriminative features at the sample level, they often overlook the consistency at the group level and are unable to model group features.

For multi-view client α, samples of each category naturally converge toward their corresponding cluster center. To exploit this structure and mitigate cross-view distribution discrepancies, we introduce a Leader-Guided Learning Mechanism for instance-level feature learning. This mechanism uses cluster centroids as leaders, encouraging samples to approach their own category centroid while being repelled from those of other categories. The corresponding optimization objective is defined as:

where ka, kb ∈ Cα, ka and kb denote the class labels of the ka-th and kb-th samples, respectively, and |nα,ka | represents the number of samples belonging to class ka on client α. We define the objective function as Lαd = -J α d so as to minimize the loss -J α d .

Meanwhile, as the client-side models are continuously updated during the training process, in each round of aggregation or election, each group dynamically produces a new "leader", namely the updated clustering center. The update rule [MacQueen, 1967] is given as:

For single-view clients, due to their lack of complete view information, they mainly learn through the Model-level Leadership-Guided Learning Mechanism.

Model-level Leadership-Guided Learning Mechanism

In practical application scenarios, a single client usually cannot cover all categories of samples, but different clients often have intersections and overlaps in some categories. Due to sharing similar category data, the models trained by these clients often have high similarity in feature representation capabilities and network parameter spaces. Inspired by this phenomenon, we propose a model-level leadership-driven learning mechanism to achieve "social learning" among models. This mechanism promotes the effective dissemination and coevolution of knowledge among clients by guiding client models to converge towards a "leader model" with stronger representativeness or better performance.

To preserve privacy, for client α, only its expert models f 1 (·;w1 ), . . . , f M+P (·;wM+P ), together with their accuracies and the corresponding cluster centers Hα = {(hα,k1 , . . . , hα,k|Cα| ) | k1, . . . , k|Cα| ∈ Cα}, are transmitted to the server. To address the inter-client view differences, the server computes the similarity between client α and client γ as:

Assume that the client most similar to client α is client γ, that is γ = argmaxζ̸=α Simα,ζ. The model is then updated according to the following rule:

where Case A corresponds to Accα < Accγ, Case B corresponds to Accα ≥ Accγ, and ℓ denotes the learning rate.

Finally, each client applies the K-means [MacQueen, 1967] on the common semantics H obtained from the corresponding global model to calculate the cluster centroids and obtain their local clustering results. For example, for multiview client m, letting {cj}Kj=1 denote the K cluster centroids, we have:

The clustering result for the i-th sample is yi = argminj ∥hi -cj∥2 . By concatenating clustering results from all clients, we obtain the overall clustering results.

3.4 Expert Adaptive Aggregation Strategy

Inspired by these articles [Shazeer et al., 2017; Fedus et al., 2022; Khuong et al., 2025; Feng et al., 2025b], we integrate MoE into FedMVC. After the local training of expert models is completed on each client, the trained expert models are uploaded to the server [McMahan et al., 2017]. The server then aggregates all the received expert models and uniformly distributes them back to each client. To prevent user privacy leakage, the training of the MoE network is carried out entirely locally on the client side. At this stage, only the model parameters of the MoE network are transferred between clients, without sharing any raw data or user sensitive information.

For the α-th client, the local dataset is defined as Dα = {(x1αi , . . . , xVαi ; yαi ) ∣∣ xvαi ∈ Rp , yαi ∈ Cα}Nα i=1 . Each input sample xαi is processed by all expert models{f 1 (·; w1 ), . . . , f M+P (·; wM+P )}, producing a set of refined features{H1αi ,H2αi , . . . ,H M+P αi }. These refined features are averaged to obtain ¯Hαi, and the routing network applies layer normalization to generate an embedding token [Riquelme et al., 2021], given by x = LN( ¯Hαi). Subsequently, a linear transformation computes logits over N experts as G = Wx + b, where G ∈ RN . The top-K experts are selected based on these logits, denoted as{i1, . . . , iK} = TopK(G), ensuring sparse activation of the most relevant experts.

The logits corresponding to the selected experts are passed through a Softmax function to compute their normalized weights,

reflecting the relative importance of each expert.

A weighted combination of the selected expert outputs is then computed, yielding the mixed representation

Finally, the classification head, implemented as a learnable linear layer, generates the predicted label

To finalize the training iteration, the gating network and classifier are jointly optimized using the cross-entropy loss:

where K denotes the number of classes, logitsk represents the unnormalized prediction score for class k, and yk is the one-hot encoded ground-truth label. The Softmax function converts logits into a normalized probability distribution over all classes. Cross-entropy loss penalizes low confidence in the true class and enables joint backpropagation-based optimization of the classifier and gating network in the MoE framework.

4 Experiments

4.1 Experimental Settings

Datasets. We evaluated our method on four multi-view datasets. MNIST-USPS [Peng et al., 2019] includes 5,000 samples from two handwritten digit image sources, forming two views. BDGP [Cai et al., 2012] contains 2,500 samples across five fruit fly categories, each with text and visual views. Multi-Fashion [Xiao et al., 2017] has 10,000 samples in 10 categories, where different styles of the same object are treated as three views. NUSWIDE [Chua et al., 2009] comprises 5,000 images with five views. For specific client settings, please refer to the supplementary materials.

Comparison Methods. We tested two heterogeneous settings to demonstrate our method's adaptability: Dirichlet(∞) and Dirichlet(0.5). Smaller Dirichlet(·) parameters indicate higher heterogeneity and more imbalanced class and label distributions. Under Dirichlet(∞), we compared seven recent methods: HCP-IMSC [Li et al., 2022], DSIMVC [Tang and Liu, 2022], ProImp [Li et al., 2023], FedDMVC [Chen et al., 2023], FCUIF [Ren et al., 2024], FMCSC [Chen et al., 2024] and Energy-DIMC [Wang et al., 2025b]. FedDMVC, FCUIF, and FMCSC are federated multi-view clustering methods; the others are centralized incomplete multiview clustering approaches. For fair comparison, we concatenated data across all clients as input for centralized methods. Multi-view clients provided complete data; single-view clients were treated as having missing views.

Implementation Details. For each encoder-decoder pair, the encoder architecture is Input - Fc500 - Fc500 - Fc2000 - Fc15, and the decoder is symmetric to the encoder. We set the temperature parameters to τα = 0.6 and τβ = 0.7, and use a batch size of 256. The output dimension d is fixed to 15 for all models. Unless otherwise specified, all experiments involving H2 -FedMVC are conducted under the condition M/S = 1 : 1.

4.2 Results and Analysis

Clustering Results. Table 1 presents quantitative results under various heterogeneous mixed-view scenarios, with experiments conducted under the Dirichlet(∞) setting. Each experiment was repeated five times independently, and averages with standard deviations are reported. By varying the ratio of multi-view to single-view clients, we created different scenarios. Results show that H2 -FedMVC outperforms recent methods across all settings, achieving strong clustering performance while preserving data privacy. As the proportion of single-view clients increases, all methods degrade to some extent, as expected. Notably, even when single-view clients double the number of multi-view clients, H2 -FedMVC maintains excellent performance, demonstrating robustness.

Table 2 presents comparison results under the Dirichlet(∞) and Dirichlet(0.5) settings for heterogeneous mixed-view scenarios where no client contains all categories. Each experiment was repeated five times independently, and averages with standard deviations are reported. Different scenarios were created by adjusting the ratio of multi-view to single-view clients. Results show that H2 -FedMVC remains stable in this challenging setting: under Dirichlet(∞), it is almost superior to all the comparison methods. Under Dirichlet(0.5), except in the case where the dataset is MNIST-USPS

and M/S = 1 : 2, it ranks second with no significant drop in accuracy.

Combining the results of the above two sets of experiments, it can be seen that the proposed H2 -FedMVC method exhibits excellent and stable performance in both balanced and unbalanced heterogeneous mixed-view scenarios, fully verifying its effectiveness and robustness.

Ablation Study. Components A and B correspond to the consensus pre-training process and the cluster center initialization strategy, respectively. Component C denotes the Model-level Leadership-Driven Learning Mechanism, which is formally defined in Eq. (12), while Component D represents the Expert Adaptive Aggregation Strategy.

As shown in Table 3, components A and B play a crucial role in clustering performance: the high-level features learned by component A directly affect the quality of subsequent refined features, thereby determining the separability of clustering; without this component, the model struggles to achieve effective clustering results. Component B is used to initialize clustering centers, and its removal leads to unstable clustering in the early training stage and a performance drop. After removing component C, both single-view and multiview clients adopt a unified expert learning strategy. Experimental results show that the introduction of component C can significantly improve performance, verifying the effectiveness of the model-level comparison and guidance mechanism. Component D achieves a balance between performance and lightweight by aggregating the top-K expert models, ensuring category coverage while controlling the model size.

Other Analysis Results. Due to space limitations, temperature optimization and client-number analyses are placed in supplementary materials, showing our algorithm's robustness and stability under incomplete multi-view and non-IID data.

5 Conclusion

In this paper, we propose H2 -FedMVC, a novel method for real-world scenarios with both IID and non-IID heterogeneous mixed views, to mine latent clustering structures across multiple clients. We first design a cross-client consensus pretraining strategy to align local models while collaboratively optimizing clustering structures. Then, we introduce a leaderguided learning mechanism to reduce view heterogeneity and an expert adaptive aggregation strategy to address client heterogeneity. Both theoretical analysis and extensive experimental results demonstrate that H2 -FedMVC consistently outperforms state-of-the-art methods across various IID settings, while also maintaining robust and competitive performance under challenging non-IID scenarios.

Acknowledgements

This work was supported in part by National Natural Science Foundation of China (No. 62476052) and the Open Fund of the Key Laboratory of Cyberspace Big Data Intelligent Security, Ministry of Education (No. CBDIS202501).