Robust Federated Hyperspectral Image Clustering

1 Introduction

By capturing reflectance across hundreds of narrow bands, a Hyperspectral image (HSI) forms a 3-D tensor that fuses spatial and spectral features [Guan et al., 2024b]. Its nearcontinuous spectra facilitate precise material discrimination superior to multispectral sensors, effectively identifying subtle land-cover differences. Consequently, HSI is indispensable for tasks such as mineral exploration, urban planning, and resource assessment [Guo et al., 2026; Zhou et al., 2024; Zhao et al., 2025].

To effectively identify and segment land features from massive complex HSI without manual annotation, HSI clustering has emerged as a fundamental unsupervised technique [Zhang et al., 2025]. While shallow methods offer mathematical interpretability, they often lack the capacity to model complex scenes [Cui et al., 2024]. Consequently, deep learning approaches have gained prominence, evolving into three paradigms: multi-stage methods that decouple representation from clustering [Ding et al., 2025], iterative methods that use alternating optimization [Tu et al., 2024], and simultaneous methods that perform end-to-end training [Luo et al., 2024]. These paradigms excel in capturing nonlinear spectral-spatial dependencies, establishing themselves as the dominant methodology for unsupervised HSI analysis.

While existing HSI clustering methods predominantly rely on the idealized assumption of centralized data availability, practical deployment is severely constrained by data silos stemming from strict privacy regulations and communication bottlenecks [Ben Youssef et al., 2024; B¨uy¨uktas et al., 2024; Xie et al., 2024]. To overcome these limitations, Federated Learning (FL) has emerged as a transformative solution, facilitating collaborative training across distributed clients without the need for raw data exchange. Nevertheless, existing Federated Remote Sensing (FRS) frameworks remain predominantly confined to supervised paradigms [Gao et al., 2024; Xie et al., 2024], which present two major hurdles for practical deployment. First, the prohibitive cost of pixel-level annotation for HSI necessitates a shift toward unsupervised clustering to reduce reliance on labeled data [Guan et al., 2026; Wang et al., 2025a]. Second, current methods often fail to address the pronounced Non-IID nature of HSI data, leading to model divergence or negative transfer during aggregation. Consequently, there is a critical need for a robust, unsupervised federated HSI clustering framework that can effectively manage data heterogeneity.

To address the aforementioned limitations, we first propose a Robust Federated Hyperspectral Image Clustering (RFHC) framework. As illustrated in Fig. 1, this framework enables collaborative training without sharing private HSI data. Specifically, we propose a dual-encoder architecture augmented with a Federated Model Weight Augmentation(FMWA) strategy. This design generates two distinct views using a locally dedicated encoder and a globally shared encoder to construct positive sample pairs, thereby avoiding the spectral distortion often introduced by traditional data augmentation. It incorporates KAN [Yu et al., 2024] to enhance nonlinear feature extraction, capture complex spectral couplings, and bolster robustness against Non-IID heterogeneity while preserving privacy. To achieve stable training on heterogeneous data, we propose a hybrid objective that integrates contrastive learning with a prototype relationship optimization mechanism. This method integrates contrastive learning with a prototype relationship optimization mechanism. Specifically, the contrastive component enhances the separability and uniformity of representations. Simultaneously, global prototypes guide local training to mitigate the heterogeneity of Non-IID data. Consequently, this synergy stabilizes the training process and facilitates the learning of discriminative semantic representations. To improve HSI scalability, we strategically offload the heavy feature learning module to clients while retaining the lightweight clustering module on the server. In summary, our work makes the following main contributions:

1. New Research Task: We pioneer the task of unsuper-
2. vised federated clustering for HSI. This method enables multi-source collaboration without exchanging raw data.
3. Novel FHSIC Framework: We propose FHSIC, a scalable federated clustering framework. It combines customized client-side representation learning with a lightweight server-side clustering module. To address Non-IID data heterogeneity, we introduce prototype relationship optimization and contrastive learning. These mechanisms generate cluster-friendly representations and effectively prevent model collapse.
4. Better Clustering Results: We validated our method across three public HSI datasets. The results show superior performance, setting a new SOTA for unsupervised federated clustering.

2 RelatedWork

2.1 HSI Clustering

HSI clustering partitions pixels into coherent groups using spectral-spatial information without supervision. Existing methods generally fall into shallow and deep categories. Shallow models, utilizing handcrafted features or graph structures [Cui et al., 2024; Zhang et al., 2025; Wang et al., 2025b], often suffer from high computational complexity that limits scalability [Guan et al., 2025a]. Deep clustering methods utilize neural networks through multi-stage, iterative, or simultaneous paradigms [Zhou et al., 2024]. Multi-stage deep clustering refers to a method in which the representation module and the clustering module are optimized separately and then sequentially connected [Guo et al., 2025; Zhao et al., 2025], but risks diminished feature discriminability due to disjoint optimization [Ding et al., 2025]. Iterative deep clustering optimizes the model by alternately executing the clustering module and representation module steps within the training loop [Guan et al., 2024b], yet often incurs high overhead and error propagation risks [Qi et al., 2024; Guan et al., 2025c]. Simultaneous deep clustering optimizes the representation learning module and clustering module end-to-end in parallel [Cai et al., 2021; Cai et al., 2023] but are prone to model collapse without robust regularization [Luo et al., 2024; Guan et al., 2025d]. Despite these advances, the robustness to heterogeneity in federated learning for HSI clustering remains largely unexplored, as most existing methods rely on centralized training.

2.2 Federated Learning for Remote Sensing

FL has emerged as a pivotal distributed paradigm in RS, enabling collaborative model training across data holders without exchanging raw local data to mitigate privacy risks [Moreno- ´Alvarez et al., 2024]. Although FL facilitates the integration of multi-source data via iterative local-global updates [Ben Youssef et al., 2024], its deployment is hampered by severe data heterogeneity and communication bottlenecks [B¨uy¨uktas¸ et al., 2025]. Existing FRS research predominantly targets supervised tasks, employing various strategies to address these challenges. For instance, to tackle data scarcity and heterogeneity in classification, recent works have integrated diffusion models [Li et al., 2024a] or combined CNNs with GCNs [Gao et al., 2024]. In object extraction and segmentation, researchers have introduced prototype matching [Zhang et al., 2023] and geographic heterogeneityaware frameworks to align class distributions. Additionally, optimization techniques ranging from masked autoencoders to blockchain have been proposed to enhance efficiency and security [B¨uy¨uktas et al., 2024]. Despite these advancements, current frameworks exhibit critical limitations for broad-scale HSI analysis: they remain confined to supervised paradigms dependent on unavailable pixel-level annotations, and while addressing general statistical shifts, they fail to resolve the severe Non-IID distributions inherent to decentralized HSI [Moreno- ´Alvarez et al., 2024; Ben Youssef et al., 2024]. Consequently, the potential of FL for unsupervised HSI clustering remains largely unexplored, necessitating robust frameworks to bridge this gap.

3 Methods

3.1 ProblemDefinition

We formulate the unsupervised federated HSI clustering task within a Non-IID cross-silo paradigm. Consider a federated system comprising a central server and M clients {S1, . . . , SM}. Each client Si maintains a private local HSI dataset Di ∈ RW i×Hi×D containing a local class set Ui of size ki, which exhibits significant heterogeneity in both spatial dimensions and spectral distributions compared with other clients' data [Qi et al., 2024]. The primary objective is to partition the global logical dataset D = ⋃Mi=1Di into K = ∣∣∣⋃Mi=1 Ui ∣∣∣ semantic clusters without transmitting raw data. We assume a synchronous environment where all clients participate in each communication round [McMahan et al., 2017]. To address this, we introduce the RFHC framework as shown in Figure 2. Let F i (·) and F(·) denote the local network at client Si and the global network, parameterized by {f L θi , gL ϕi , P L ψi } and {f G θ , gG ϕ ,P G ψ }, respectively. Consequently, the problem is modeled as a distributed optimization process characterized by two coupled objectives:

Client-Side Objective. Each client Si aims to extract discriminative high-level features by optimizing parameters (θ∗ i , ϕ∗i , ψ∗ i ) for its personalized local network F i (·). This is achieved by minimizing a composite loss function that balances local feature quality with global consistency:

Server-Side Objective. The server's objective is to aggregate client models to construct a global model F (·) and to concatenate local embeddings to derive global prototypes C and client-specific pseudo-labels {P1 , . . . ,PM }. Ultimately, the server retains the parameters (θ∗ , ϕ∗ , ψ∗ ) of F (·) that yield optimal clustering performance on the global representation H over T training rounds.

3.2 Local Training

The local training phase optimizes client-specific models to extract discriminative features from private Non-IID data while ensuring global semantic alignment. We propose a specialized architecture that synergizes a dual-encoder specifically for HSI with a prototype-based objective to mitigate model divergence caused by statistical heterogeneity.

Dual Autoencoders Module While many studies employ data augmentation to create positive sample pairs, the rich information inherent in HSI means such transformations risk altering spectral features and corrupting semantic content [Guan et al., 2024b]. Concurrently, the standard autoencoder's reconstruction objective is misaligned to learn discriminative features, as it encourages the latent space to retain excessive input information, creating vulnerabilities for data leakage and impeding the learning of a unified feature representation in a decentralized setting [Balestriero and LeCun, 2024]. To mitigate the curse of dimensionality and spectral distortion from standard augmentation, we introduce Federated Model Weight Augmentation (FMWA). This strategy constructs consistent multi-views via network interaction rather than data transformation. Formally, given pixel cubes Xi = {Xji ∈ Rw×w×d }N i j=1 derived from principal component analysis and sliding windows, FMWA employs a trainable local encoder f L θi (Xi; θi) : Xi ∈ RN i×d ↦-→ ZiL ∈ RN i× ˆ d adapted to private distributions and a frozen global encoder f G θ (Xi; θ) : Xi ∈ RN i×d ↦-→ ZiG ∈ RN i׈ d aggregated from collective knowledge. To refine and unify latent representations, we utilize KAN-based projectors P L ψi and P G ψ to map latent embeddings into high-level representations HiL and HiG [Yu et al., 2024; Guan et al., 2025b]:

where ˜ d < ˆ d with ˜ d denoting the dimensionality of the highlevel representation and B denoting the batch size. This design enables the local network F i (·) to extract features invariant to local perturbations yet aligned with the global semantic structure [Li et al., 2024b; Balestriero and LeCun, 2024].

Prototype Relationship Optimization Module Although the dual encoders module provides a robust feature learning foundation, it lacks a mechanism to enforce a consistent global clustering structure. Since the distributed HSI is Non-IID and suffers from the problem of the same object having different spectra, it may erroneously separate pixels of the same type originating from different clients [Kumar and Lee, 2025; Tan et al., 2022]. This forces the model to learn spurious, client-specific features, thereby exacerbating model divergence [Mu et al., 2023; Guan et al., 2025c]. To rectify pixel-level category misalignment arising from NonIID distributions, this module enforces intra-cluster compactness and inter-cluster separability via prototype-level constraints. In round t, client Si receives global prototypes C = {c1, . . . , cK} ∈ RKט d and pseudo-labels Pi ∈ RN i where pij ∈ {1, . . . ,K} [Kumar and Lee, 2025; Tan et al., 2022]. We construct an assignment matrix Wi ∈ RK×N i where element wikj normalizes the contribution of pixel j to cluster k inversely to cluster size [Mu et al., 2023; Guan et al., 2025c]. Local prototypes Ci are derived by aggregating local features HiL viaWi , allowing gradient backpropagation to the encoder. The k-th local prototype Cik is computed using feature vector hiL,j for the j-th pixel as:

To align local geometry with global structure, we minimize the prototype relationship optimization Loss LiC . This objective operates on positive pairs formed by corresponding prototypes Cik and Ck to encourage consistency and negative pairs involving Cik and Cj for j ̸= k to enforce separation. By employing a sampling operation T (·) on noncorresponding global prototypes {Cj}j̸=k [Tu et al., 2024], the formulation comprises a repulsion term minimizing similarity with negative samples and an attraction term maximizing alignment with positive targets:

Minimizing Eq. (5) guides local features HiL toward global semantic alignment, mitigates the Non-IID problem, and promotes a coherent and globally consistent clustering structure as well as robust feature extraction capability.

3.3 Objective Function

The optimization of the local network F i (·) aims to learn personalized features that are structurally aligned with global semantics. The dual-encoder architecture projects mini-batch features HiL,b and HiG,b into a K-dimensional label space via adaptive local head gL ϕi and frozen global head gG ϕ , yielding assignment matrices YiL,b and YiG,b ∈ RB×K , respectively. The total objective for client Si is a weighted synergy of structural and instance-level constraints:

where α1,2 are balancing hyperparameters. This composite objective establishes a hierarchical optimization strategy: LiC acts as a high-level structural guide, utilizing global prototypes as semantic anchors to alleviate the Non-IID problem. Complementing this, the contrastive duo (LiW ,LiB ) performs fine-grained refinement to ensure representational quality and prevent collapse. This hierarchy ensures that personalized features remain firmly aligned with the global consensus, effectively mitigating model divergence [Cai et al., 2023].

Within-Cluster Contrastive Loss (LW) To enforce view consistency at the instance level, we employ a symmetric InfoNCE objective [Cai et al., 2023; Cai et al., 2021]. Let yiL,j and yiG,j denote the j-th row vectors of assignment matrices YiL,b and YiG,b , representing the probability distributions of the j-th sample. Treating (yiL,j , yiG,j ) as a positive pair, the loss component anchored on the local view is:

where sim(u, v) = u⊤v/(∥u∥∥v∥) is the cosine similarity and τ is a temperature hyperparameter. The total loss LiW enforces symmetry by averaging the objective across both views for all samples in the batch:

Between-ClusterContrastiveLoss (LB) To enhance intercluster separability, we adapt the Barlow Twins objective to minimize redundancy in the label space [Zbontar et al., 2021; Cai et al., 2021]. Distinct from the instance-wise operation in LW, we focus here on the column vectors yiL,k and yiG,j of the assignment matrices, which represent the distribution of the k-th cluster across the batch. We compute the crosscorrelation matrix R ∈ RK×K , where element Rkj is the cosine similarity between the centered local cluster vector yiL,k and global cluster vector yiG,j :

The final objective penalizes deviations from the identity matrix:

where λ is a weighting coefficient controlling the importance of the off-diagonal terms. This formulation comprises an invariance term pushing on-diagonal elements to 1 for alignment and a redundancy reduction term pushing off-diagonal elements to 0 for decorrelation, preventing trivial solutions where all samples collapse to a single cluster.

3.4 Global Training

The global training phase, executed on the central server, orchestrates collaborative learning to refine both the global model and the shared clustering structure. By synthesizing a unified model from decentralized Non-IID data, this phase coordinates two core modules: model aggregation and federated HSI clustering, ensuring global semantic consistency across isolated clients.

Model AggregationModule The primary objective of this module is to consolidate client-side knowledge into a cohesive global model F (·), thereby preventing catastrophic forgetting and establishing a shared consensus. At the end of each round, clients upload their optimal parameters (θ∗ i , ϕ∗i , ψ∗ i ). The server aggregates these using FedAvg [McMahan et al., 2017], scaling contributions by the relative dataset size N i /N to update the global parameters:

The updated components (f G θ , gG ϕ ,P G ψ ) are distributed back as the frozen global view, anchoring local training to the global optimum to mitigate client drift.

Federated HSI Clustering Module This module refines the global clustering structure by generating updated global prototypes and pseudo-labels, acting as a global arbiter to enforce a unified semantic partitioning of the feature space. To analyze the global distribution without physically centralizing data, the server constructs a global feature matrix H by concatenating uploaded local representations HiL ∈ RN iט d . This matrix represents the global logical dataset D:

This aggregated matrix provides a holistic view of the entire dataset's feature distribution. Subsequently, we employ the K-means algorithm K(·) on H to derive the global semantic structure [Zhang et al., 2023; Tan et al., 2022]. The process yields the final cluster assignment vector P ∈ RN and the global weight matrix W ∈ RK×N :

Derived from these assignments, the updated global prototypes C ∈ RKט d are constructed to encapsulate the collective semantic consensus. Specifically, the k-th prototype Ck is synthesized by aggregating the global feature vectors hj for the j-th global pixel drawn from the global feature matrix H. The aggregation is performed using weights wkj from the assignment matrix W, which quantify the membership confidence of pixel j to cluster k:

This weighted aggregation effectively filters local noise, producing robust prototypes C and pseudo-labels {P1 , . . . ,PM } that are partitioned from P and guide the subsequent local training toward a globally aligned feature space. Finally, the clustering quality is validated via the nearest-prototype rule:

4 Experiments

4.1 Experimental Setup

Datasets We evaluate the proposed method on three public hyperspectral datasets: Indian Pines (145 × 145 pixels, 200 bands, 16 classes), Salinas (512 × 217 pixels, 204 bands, 16 classes), and Pavia University (610 × 340 pixels, 103 bands, 9 classes).

Compared Methods To evaluate the performance of our proposed RFHC, we benchmark it against 7 SOTA federated clustering baselines. These include one-shot approaches such as k-FED [Dennis et al., 2021] and NN-FC [Wang et al., 2024], as well as matrix factorization-based algorithms FedMAvg and FedMGS [Wang and Chang, 2022]. Furthermore, we compare RFHC with the cluster-contrastive framework CCFC [Yan et al., 2024] and the lossless fuzzy c-means variants LLF-CM and LLF-FCM [B´arcena et al., 2024].

Implementation Details To ensure fairness, we report averages over ten iterative runs conducted on a 24GB RTX 3090 GPU with 64GB RAM. The training schedule consists of 100 epochs across ten execution runs, utilizing the Adam optimizer with a learning rate of 1 × 10-6 and a weight decay of 5 × 10-4 . The hyperparameters λ, τ , and T are set to 0.005, 0.5, and 100, respectively. Regarding the architecture, we employ a hybrid 3D-2D CNN backbone that consists of stacked convolutional blocks and an MLP projector integrated with a KAN layer to capture complex nonlinear relationships. All baselines adopt their corresponding open-source implementations with optimal parameters. We evaluate performance using overall accuracy (ACC), Kappa, normalized mutual information (NMI), and adjusted Rand index (ARI) [Guan et al., 2024a; Yang et al., 2026; Luo et al., 2024; Cai et al., 2021; Qi et al., 2024], noting that ACC and NMI range within [0, 1] whereas Kappa and ARI range within [-1, 1].

4.2 Experimental Results

We evaluate the clustering performance of RFHC against seven competitive federated baselines across three HSI datasets with 5 clients. The quantitative results, summarized in Table 1, yield the following key observations:

1. Overall Superiority: RFHC consistently achieves dominant performance across the majority of metrics. Particularly on the Pavia University(PU) dataset, RFHC attains an ACC of 52.93% and a Kappa of 40.85%, surpassing the second-best method by 9.48% and the LLF-FCM baseline by 6.64%, respectively. These margins strongly validate the effectiveness of our dual-encoder architecture and prototype optimization in federated settings.
2. Comparison with Shallow and Lossless Methods: While traditional (k-FED) and lossless frameworks (LLFFCM) exhibit competitive structural metrics, RFHC signifi-

cantly surpasses them in classification accuracy. Specifically, RFHC outperforms LLF-FCM by margins of 8.92%, 10.81%, and 10.61% in ACC across Indian Pines(IP), Salinas(SA), and PU, respectively. This indicates that KAN-enhanced nonlinear extraction captures complex spectral semantics better than statistical aggregation.

1. Comparison with Deep Federated Approaches: Deep baselines FedMAvg, FedMGS, and CCFC often struggle with Non-IID HSI scenarios. In contrast, RFHC exhibits substantial improvements, notably boosting ACC by 17.59% on IP and 28.71% on SA compared to the contrastive-based CCFC. This confirms that our FMWA strategy and prototypeguided training effectively mitigate model divergence and representation collapse in Non-IID scenarios.

In addition to quantitative metrics, we assess clustering quality visually. As illustrated in Fig. 3, baseline methods exhibit noticeable intra-class imbalance noise and struggle to differentiate spectrally similar agricultural classes. Moreover, they fail to address the "same object, different spectra" phenomenon, which causes identical categories to be misclassified across different clients. In contrast, RFHC effectively alleviates this heterogeneity to produce smoother and spatially coherent maps with distinct boundaries. This visual superiority is largely attributed to the prototype relationship optimization, which enforces semantic consistency and reduces misclassification in transition regions.

4.3 Ablation Study

Table 2 validates the contribution of each RFHC component.

Ablation on the Main Loss The full objective yields optimal accuracy, demonstrating that these components are mutually reinforcing. Notably, removing the prototype relationship optimization (LC) causes a sharp decline, underscoring its critical role in aligning local-global representations. Similarly, excluding contrastive losses (LB,LW) significantly degrades performance; for instance, removing LW reduces ACC on IP to 35.71%. This confirms that while prototypes handle heterogeneity, contrastive constraints are essential for instance-level discriminability and preventing collapse.

Ablation on the Network Architecture First, the significant gap between random initialization ("Init.") and optimized models ("Train") verifies the framework's learning efficacy. Second, integrating the KAN module (P) enhances representation quality. This demonstrates that KAN effectively captures nonlinear spectral-spatial couplings, thereby bolstering feature robustness against data heterogeneity.

4.4 Parameter Analysis

Impact of Hyper-parameters α1 and α2 We examine the regularization weights α1 and α2 within {10-2 , . . . , 102 }, as visualized in Fig. 4(a). The results reveal that performance degrades significantly when both parameters are small, indicating that the prototype objective alone is insufficient to prevent model collapse. Conversely, increasing these weights enhances accuracy, with the SA dataset achieving optimality at (α1, α2) = (100, 100).

Impact of Hyper-parameter w As shown in Fig. 4(b), the SA dataset exhibits a convex trend regarding window size w. Small patches yield suboptimal results due to insufficient spatial context for noise suppression. Performance peaks at w = 19 but declines thereafter. Consequently, w = 19 is selected to balance spatial smoothing and boundary precision.

Impact of Hyper-parameterM We evaluate scalability by varying the number of clients, where M takes values from the set {5, 10, 20}, as illustrated in Fig. 5 and Table 1. A consistent performance decline is observed across all datasets as M increases due to intensified data fragmentation; for instance, accuracy on IP decreases from 44.40% when M = 5 to 26.23% when M = 20.

5 Conclusion

In conclusion, this study presents RFHC, a robust framework for unsupervised federated hyperspectral image clustering that addresses data silos and annotation scarcity. To ensure feature integrity, the model utilizes a dual-encoder architecture with a federated model weight augmentation strategy, generating consistent views without introducing spectral distortion. Furthermore, we incorporate a prototype relationship optimization mechanism combined with KAN layers to align local training with global consensus, effectively mitigating the adverse effects of severe Non-IID distributions.