Pixel-wise Divide and Conquer for Federated Vessel Segmentation

1 Introduction

Vessel segmentation plays a crucial role in the image registration of blood vessels and in the diagnosis and management of vascular and ophthalmic diseases, such as coronary artery disease, diabetic retinopathy, and glaucoma [Liu et al., 2024]. Advances in deep learning, particularly through U-Net and its variants [Ronneberger et al., 2015; Liu et al., 2022], have significantly improved segmentation accuracy and robustness. However, progress in vessel segmentation is hindered by the lack of large-scale, high-quality, pixel-level annotated datasets, which are expensive and time-consuming to construct. In addition, data collected from different institutions often vary in imaging modalities, acquisition protocols, and labeling standards, resulting in heterogeneous vessel image distribution. Notably, privacy regulations further restrict multi-center data sharing, emphasizing the need for distributed approaches that can preserve privacy while leveraging diverse and heterogeneous datasets. Recently, Federated Learning (FL) has provided an ideal solution for vessel segmentation by addressing data privacy concerns and effectively handling data heterogeneity [McMahan et al., 2016; Hyeon-Woo et al., 2022; Elgabli et al., 2022; Yang et al., 2023a; Huang et al., 2024b; Wang et al., 2025]. By enabling collaborative training without requiring the exchange of raw data, FL offers a reliable framework to advance multiparty vessel segmentation collaboration, enhancing model generalization and scalability across diverse clinical environments.

Federated vessel segmentation faces greater performance challenges than general federated medical segmentation due to vessel morphology inconsistency and vessel-background imbalance. The former arises from significant variations in the distribution of thin and thick vessel pixels across clients, compounded by the complexity of intricate regions, such as vessel intersections. These inconsistencies bias the global model toward dominant features (e.g., thick vessels or simpler regions) while neglecting subtle yet critical details, such as thin vessels and ambiguous boundaries. Existing methods commonly optimize local clients using cross-entropy loss, which treats all units (e.g., images or pixels) uniformly without accounting for fine-grained challenges. Some studies have explored sample reweighting strategies to address data imbalance, enhance the representation of specific samples, and promote fairness across clients [Zhao and Joshi, 2022]. However, these strategies primarily focus on image-level reweighting, while the full potential of pixel-level reweighting remains largely unexplored. This gap is particularly critical for vessel segmentation, where fine-grained structures, such as thin and tortuous vessels, require more precise and adaptive guidance to ensure accurate, reliable, and faithful representation. Consequently, a critical question emerges: "Q1: How can global information be leveraged to provide differentiated guidance for vessel pixels?"

Regarding the vessel-background imbalance, current federated learning frameworks typically perform one-to-one model or parameter alignment, which involves either fullparameter alignment [Li et al., 2020b] or direct point-to-point feature alignment [Mendieta et al., 2022]. However, this approach significantly limits the applicability of federated learning in vessel segmentation, where the foreground-background disparity is particularly pronounced. Overemphasis on oneto-one alignment often leads to prioritizing background pixels, which weakens the semantic representation of sparse categories like vessels. Additionally, the aligned features are unstable and lack robustness when handling heterogeneous client data. Consequently, another key question arises: "Q2: How to use robust feature alignment strategies to balance foreground and background global knowledge?"

Based on the above issues, we propose our method. Concerning Question 1, we propose Federated Morphology Uncertainty Guidance (FMUG), inspired by the sample reweighting strategy in Federated Learning (FL). The first and most important step is to reveal the "hard kernel", which consists of challenging pixels that significantly impact the segmentation performance. Therefore, our method combines global and local uncertainty to guide the model in focusing on these key pixels. To be precise, we generate a global uncertainty map using the predictions of the global model and the ground truth alongside a local uncertainty map derived from the optimization process of the client's local model. These uncertainty maps facilitate a progressive learning process by dynamically identifying areas requiring focused attention at each stage. Leveraging these maps, we employ an uncertainty-based weighted cross-entropy loss to encourage the model to improve its accuracy while enhancing the uncertainty associated with each prediction. As for Question 2, we propose Federated Vessel Decoupling Alignment (FVDA), a foreground-background separation alignment approach. Existing methods primarily focus on aligning local and global directions for each pixel, treating this alignment as a holistic optimization term. In contrast, our work aims to decouple the foreground and background using the provided ground-truth labels. Specifically, we reconstruct foreground and background masks based on pixel labels and align each part independently. We argue that this approach is both compatible and flexible, ensuring semantic consistency while maintaining overall coherence with the global style.

In this paper, we aim to tackle the unique challenges in federated vessel segmentation, including vessel morphology inconsistency, foreground-background imbalance. To address these critical issues, we design innovative approaches that efficiently enhance segmentation accuracy, preserve semantic consistency, and improve generalization across diverse datasets. Our contributions are summarized as follows:

• We propose a novel pixel-level federated vessel segmentation method based on a divide-and-conquer approach, which leverages global uncertainty to guide local models in identifying critical pixels and effectively tackles challenges related to vessel morphology inconsistencies.
• We introduce a foreground-background separation alignment approach for vessel segmentation, effectively decoupling and aligning vessel and background features to ensure semantic consistency, thereby improving the generalization of the global model across diverse and heterogeneous client datasets.
• We conduct extensive experiments on two modalities of vessel datasets: one consisting of color fundus images (DRIVE [Staal et al., 2004], STARE [Hoover et al., 2000], CHASEDB1 [Owen et al., 2009]) and the other composed of optical coherence tomography angiography (OCTA) images (ROSE-1 [Ma et al., 2021] and

OCT-500 [Li et al., 2024]). Our method demonstrates superior performance compared to related approaches.

2 RelatedWork

2.1 Federated Learning

With the introduction of FedAvg by McMahan et al. [McMahan et al., 2017], federated learning (FL) began to emerge as a promising and scalable approach for distributed learning. This groundbreaking work clearly showcased the potential of FL to enable collaborative model training without requiring raw data exchange, addressing privacy and security concerns that traditional centralized methods could not easily overcome. However, it experiences degraded performance when handling non-i.i.d. data (data heterogeneity). Some approaches focus on leveraging shared models as global signals [Li et al., 2020b; Li et al., 2021; Huang et al., 2022; Huang et al., 2023b; Huang et al., 2023a], while others emphasize utilizing statistical distributions [Zhang et al., 2022; Zhou and Konukoglu, 2023; Tang et al., 2024]. Additionally, certain methods are built upon semantic class prototypes [Tan et al., 2022; Mu et al., 2023; Huang et al., 2023c; Huang et al., 2024a], and some rely on gradient aggregation [Karimireddy et al., 2020; Gao et al., 2022] to enhance collaboration in federated learning. These methods have achieved remarkable success, significantly advancing the development and adoption of federated learning. However, these methods are not specifically designed for pixellevel dense prediction tasks. Their granularity in federated learning is insufficient, especially for medical segmentation tasks, where they struggle to perform well in segmenting objects with blurry boundaries or thin and elongated structures.

2.2 Medical SegmentationMeet Federation

Fueled by the rapid development of deep learning, medical image segmentation has achieved remarkable progress [Ronneberger et al., 2015; Isensee et al., 2021; Ma et al., 2024], greatly enhancing clinical diagnosis and treatment by enabling precise and efficient delineation of anatomical structures and pathological lesions, such as brain tumors [Yang et al., 2023b; Zhou et al., 2024], optic discs [Yii et al., 2023; Chen et al., 2024], and lung nodules [Wang et al., 2023a]. However, training fully supervised models often demands costly and labor-intensive pixel-level annotations, particularly in the medical domain. Additionally, the sensitive nature of patient data and strict privacy regulations create significant legal and practical barriers to centralized data collection and sharing. In this context, federated learning (FL) has emerged as a natural, scalable, and promising solution for medical image segmentation [Wang et al., 2022; Wang et al., 2023b]. Jiang et al. [Jiang et al., 2022] effectively mitigates local drift using amplitude normalization and addresses global drift through weight perturbation, thereby resolving style domain discrepancies in medical images.

In this paper, we focus on federated vessel segmentation, where the targets are characterized by variable thickness and tortuous structures. Additionally, the significant foregroundbackground disparity, with sparse vessel pixels compared to the dominant background, poses unique challenges for accurate segmentation in federated settings.

2.3 Federated Vessel Segmentation

Vessel segmentation has experienced rapid advancements in recent years. Accurately segmenting vessels with complex shapes and variable thicknesses remains a challenging and important problem. As a landmark network in medical image segmentation, U-Net [Ronneberger et al., 2015] has demonstrated excellent performance in vessel segmentation, owing to its symmetric encoder-decoder and skip connection architecture. Inspired by U-Net, numerous variant networks for vessel segmentation have been developed [Li et al., 2020a; Liu et al., 2022]. Nevertheless, they all rely on detailed annotations, limiting applicability. Moreover, due to strict patient confidentiality and privacy policies, centralized data collection and sharing are highly challenging and often infeasible.

Federated learning (FL) offers a promising new paradigm for vessel segmentation by addressing the challenges of centralized data collection and the reliance on finely detailed annotations. Unfortunately, existing federated medical segmentation fails to fully address the unique challenges of vascular segmentation due to the high variability in vascular shapes and the uneven distribution of vessel thickness.

In this paper, we introduce the first federated learningbased framework designed explicitly for vessel segmentation. This framework enables clients to effectively focus on key pixels, enhance local model performance, and simultaneously learn more generalized, robust, and contextually aware semantic representations across heterogeneous datasets.

3 Methodology

3.1 Preliminaries

Following the setup of FedAvg [McMahan et al., 2017], typical Federated Learning (FL) usually involves a centralized server and multiple clients. Each client k holds its private dataset Dk, which can be represented at the sample level as:

where nk represents the number of samples in the k-th client's local data. At the beginning of each communication round t, the global model parameter θt is broadcasted by the server to all clients. Each client then performs local optimization on its dataset by minimizing a local objective (e.g., cross-entropy loss), resulting in updated local parameters θtk , which are sent back to the server. The server aggregates these local updates using a weighted averaging mechanism to produce the global model for the next round:

where M denotes the total number of clients, and the weight αk ensures that each client's contribution is proportional to its dataset size. This framework enables collaborative learning without sharing raw data, effectively preserving privacy while utilizing distributed datasets across multiple clients.

For segmentation-specific tasks, the client's dataset can also be expressed at the collection level as:

where P(h,w) denotes the predicted probability for pixel (h,w), and Y (h,w) is its ground truth label. The function argmax(P(h, w)) returns the class with the highest predicted probability. For correctly classified pixels, uncertainty increases when non-predicted classes have higher probabilities, indicating weaker separation. For misclassified pixels, higher probability for the incorrect class leads to increased uncertainty, reflecting overconfidence. This behavior underscores the need to analyze the full probability distribution to assess confidence and detect weaknesses.

3.2 ProposedMethod

Motivation. Federated Learning (FL) has emerged as a privacy-preserving collaborative learning framework, enabling multiple institutions to train global models without sharing sensitive data. In medical image segmentation tasks, existing FL methods primarily focus on addressing data heterogeneity caused by differences in imaging modalities (e.g., MRI, CT) or acquisition protocols, emphasizing mitigating style shifts in global and local feature distributions.

However, segmenting thin and tortuous structures (e.g., blood vessels) presents unique challenges in federated learning. Blood vessel segmentation suffers from vessel morphology inconsistency, where the distribution of thin and thick vessels varies significantly across clients. As a result, the global model often overlooks subtle yet important details, such as thin vessels and unclear boundaries, while focusing on dominant features like thick vessels or simpler regions. Moreover, the vessel-background imbalance, where vessels occupy only a small fraction of pixels, further weakens the representation of sparse categories and causes unstable feature alignment across heterogeneous clients.

Considering these challenges, this work focuses on:

• Leveraging the global uncertainty map to guide local clients in effectively prioritizing key regions (e.g., thin vessels and blurred edges) during training.
• Foreground-background separation semantic alignment to help local clients learn more generalizable features, ensuring consistency across diverse clients while preserving fine-grained segmentation performance.

Overview of Framework. The framework of our method is illustrated in Fig. 1. Specifically, We leverage the uncertainty from the prediction results to assist local models in learning the hard kernels while utilizing decoupled features to mitigate the bias toward the dominant class. Next, we will introduce Federated Morphology Uncertainty Guidance and Federated Vessel Decoupling Alignment in detail.

I) FederatedMorphology Uncertainty Guidance. Considering vessel morphology inconsistency, which represents pixel-level data heterogeneity (e.g., varying vessel crossing degrees), this issue severely hampers the learning of finegrained features. As a result, the global model becomes biased toward dominant features (e.g., thick vessels or simpler regions), leaving it uncertain or even incapable of segmenting thin, elongated structures or ambiguous boundaries.

To address this, we leverage global uncertainty to guide local models in further refining their learning on hard-todistinguish pixels, improving segmentation accuracy. Additionally, we treat the local client optimization process as a progressive learning approach, specifically focusing on uncertain pixels. By incorporating local uncertainty, we dynamically adjust the global uncertainty, deriving stable pixel-level learning weights to enhance model performance.

For a given input X, the model's predictions are represented as:

where Z denotes the logits of the model, and σ represents the softmax function, which transforms the logits into a pixelwise probability distribution P.

Based on these predictions, uncertainty maps are derived to quantify the reliability of the predictions at the pixel level:

where P(h,w) represents the predicted probability for pixel (h,w), and Y (h,w) denotes the ground truth label for that pixel. The function argmax(P(h,w)) returns the index of the class with the highest predicted probability at pixel (h, w). For correctly classified pixels, uncertainty increases when the probabilities assigned to non-predicted classes are higher, indicating a weaker separation between the predicted class and others. Conversely, for misclassified pixels, uncertainty increases when the probability of the incorrect class is higher, reflecting the model's overconfidence in its prediction. This nuanced behavior highlights the importance of understanding the probability distribution across all classes to effectively assess confidence and identify weaknesses or inconsistencies.

Thus, based on the uncertainty maps U g ∈ RH×W and U k ∈ RH×W , the pixel-wise weight map for the entire image is computed as a linear combination of the global and local uncertainty maps:

where the superscripts g and k denote the global uncertainty (aggregated across clients) and the local uncertainty (specific to the current client), respectively. To ensure proper normalization, the weights are scaled as follows:

where the denominator ensures that the sum of all weights across the image equals 1. These normalized weights are then used to compute the weighted cross-entropy loss over the entire image:

where 1Y (h,w)=c selects the predicted probability Pc(h,w) corresponding to the ground-truth class c ∈ {0, . . . ,C-1}.

Here, C denotes the total number of semantic classes (and prediction channels). α(h, w) is a pixel-wise adaptive weight derived from both global and local uncertainty. In our binary vessel segmentation task, C = 2.

This approach adaptively penalizes ambiguity in correct predictions and overconfidence in incorrect ones. Leveraging uncertainty maps, the model focuses on difficult regions, enhancing robustness and performance.

II) Federated Vessel Decoupling Alignment. Vessel-background imbalance poses another significant challenge in federated vessel segmentation, weakening the semantic representation of sparse classes like vessels. Common one-to-one feature learning or parameter-sharing strategies in general federated learning often exacerbate this by overemphasizing background pixels, leading to unstable feature learning and reduced model robustness.

To address this, we extract foreground and background features from intermediate model features F ∈ RC×H×W , where C, H, and W are the number of channels, height, and width. The ground truth Y is resized to H ×W to obtain the mask S, and we optimize foreground and background features separately. The optimization is as follows:

where F[:, h,w] represents the feature vector at pixel (h,w), and S(h,w) is the mask value at pixel (h,w). The features Ffg and Fbg are averaged across all pixels, ultimately yielding a C × 1 feature vector for each respective category.

To ensure consistency between the local and global feature representations, we calculate the Mean Squared Error (MSE) loss for both foreground and background features in the model. The loss is then computed as follows:

Through the combined use of the reweighted cross-entropy loss and the decoupled alignment loss, the final training objective is defined as:

where β controls the balance between the two loss terms. The reweighted cross-entropy loss LFMUG improves segmentation accuracy and confidence, while the decoupled alignment loss LFVDA ensures consistent features between local and global models. The overall algorithm is illustrated in Algorithm 1. Jointly optimizing both losses boosts accuracy and preserves foreground-background consistency.

3.3 Discussion

Conceptual Difference. Most FL methods use cross-entropy loss and sample reweighting but overlook pixel-level weighting and global context, limiting their ability to handle complex pixels. We first introduce global uncertainty in FL for vessel segmentation, guiding local models toward hard pixels and boosting accuracy. Moreover, conventional FL aligns

Algorithm 1: Model training in FVAC

Input: Communication rounds T , local epochs E, number of participants K, the kth participant private data Dk(x, y), private model θk Output: The final global model θT for t = 1, 2, ..., T do Participant Side; for k = 1, 2, ..., K in parallel do θtk ← LocalUpdating(θt , G) Server Side; θt+1 ← 1K ∑Kk=1 θtk LocalUpdating(θt , G): θtk ← θt ; // Distribute global parameter θfixed ← θt ; // Fix global parameter for e = 1, 2, ..., E do for (Xi, Yi) ∈ Dk do Zi = f(Xi, θtk), Z g i = f(Xi, θfixed) Pi = σ(Zi), P g i = σ(Z g i ) /* Uncertainty Estimation */ Ui ← (Pi, Yi), U g i ← (P g i , Yi) in Eq. (6) /* Weight Assignment */ α ← (U g i ,Ui) in Eqs. (7) and (8) /* Feature Decoupling */ Ffgi , Fbgi ← (Fi, Si) in Eq. (10) Fgfgi , Fgbgi ← (Fgi , Si) in Eq. (10) LkFMUG ← (α, Pi, Yi) in Eq. (9) LkFV DA ← (Ffgi , Fbgi , Fgfgi , Fgbgi ) in Eq. (11) LkFV AC ← (LkFMUG, LkFV DA)in Eq. (12) θek ← θek - η∇Lk return θek

models one-to-one, favoring dominant classes like the background and causing unstable feature learning. We address this by leveraging global uncertainty and decoupling foregroundbackground features for separate alignment, boosting segmentation precision and overall performance.

Related Uncertainty Learning. Uncertainty learning in semi-supervised settings helps models focus on informative regions and avoid noisy inputs, improving robustness and accuracy. Inspired by this, we incorporate uncertainty maps into FL to address vessel morphology inconsistencies, which often lead the global model to miss key pixels. Enhancing local learning on such regions mitigates global forgetting and boosts segmentation performance.

Limitation. FVAC achieves strong vessel segmentation via a pixel-wise divide-and-conquer strategy. However, it incurs extra training cost due to the additional forward pass for global uncertainty and overlooks potential attacks on the global model, which may mislead local clients. Future work will focus on enhancing robustness and security against adversarial threats while preserving efficiency and accuracy.

4 Experiments

4.1 Experimental Setup

Datasets and Evaluation Metric. We evaluate FVAC on both color fundus and OCTA retinal datasets. The fundus data include DRIVE [Staal et al., 2004], STARE [Hoover et al., 2000], and CHASEDB1 [Owen et al., 2009], while the OCTA data consist of ROSE-1 [Ma et al., 2021] and OCT500 [Li et al., 2024].

• DRIVE [Staal et al., 2004] contains 40 high-resolution color retinal images (565×584), evenly split into 20 for training and 20 for testing.
• STARE [Hoover et al., 2000] contains 20 manually annotated color retinal images (700×605), with 16 used for training and 4 for testing.
• CHASEDB1 [Owen et al., 2009] contains 28 color retinal images with a resolution of 999×960 pixels, split into 20 images for training and 8 for testing.
• ROSE-1 [Ma et al., 2021], a subset of ROSE, includes 117 OCTA images from 39 subjects, with 90 for training and 27 for testing. Each subject has enface angiograms

of the superficial (SVC), deep (DVC), and combined (SVC+DVC) vascular plexus. We use ROSE-SVC and ROSE-SD (SVC+DVC) as two domain-specific subsets of our OCTA dataset.

• OCT-500 [Li et al., 2024] provides OCTA images from 500 subjects across two fields of view, split into OCTA6MM (300 images) and OCTA-3MM (200 images). We use 280/20 for training/testing in OCTA-6MM and 180/20 in OCTA-3MM. Both subsets are treated as separate domains in our OCTA dataset.

We evaluate all vessel datasets using the Dice coefficient as the sole performance metric, as it is widely adopted in medical image segmentation to reflect the overlap between predicted and ground truth regions.

Comparison Methods. We compare our method, FVAC, with several state-of-the-art approaches, including FedProx [Li et al., 2020b], FedDyn [Durmus et al., 2021], FedOPT [Reddi et al., 2021], MOON [Li et al., 2021], FedProto [Tan et al., 2022], and FedProc [Mu et al., 2023].Although experimental settings may vary slightly across methods, we ensure fairness by preserving key features of each approach for comparison.

Implementation Details. For the experiment, to ensure simplicity and general applicability, we choose U-Net, a widely used and effective architecture, as the backbone for all Federated Learning methods. To ensure the reproducibility and consistency of our results, we fix the random seed across all experiments. In the federated learning process, models are trained using the AdamW optimizer [Loshchilov and Hutter, 2019] with a batch size of 4. The communication round is set to 100, and the local training epoch is 5 for all datasets.

We set up six participants per experiment with different combined datasets and randomly assign domains. For color fundus data, the split is DRIVE:3, STARE:1, CHASEDB1:1; for OCTA, ROSE-SVC:2, ROSE-SD:1, OCTA-3MM:1, OCTA-6MM:2. Each participant receives 1% of the original data from their assigned domains. We also apply data augmentations on each client, including color jitter, gamma correction, flipping, rotation, and random cropping.

We use a learning rate of 1e-4 for both collaborative and local updates on the color retinal datasets across all methods. In our method, all participants share the same hyperparameters (i.e., β = 2). For the OCTA datasets, the learning rate is reduced to 1e-5 due to higher segmentation difficulty from varying modalities and denser vessel structures.

4.2 Results

Performance Comparison. For clarity and brevity, we present representative visualizations from the final model after federated learning, as shown in Fig. 2. Our method preserves more complete vascular structures, especially in regions with complex, twisted, or entangled connections. Though such areas constitute only a small part of the overall network, they are essential for downstream tasks like blood flow modeling or vascular assessment. Based on earlier qualitative results, the final Dice scores across popular methods are reported in Tab. 1, showing that our approach consistently outperforms others. On the OCTA dataset, it achieves a Dice improvement of nearly 3.5% over the baseline. Both qualitative and quantitative results confirm our method's strength in capturing complex and detailed vascular structures.

Hyper-Parameter Setting. We evaluate FVAC under different β values as defined in Eq. (12), searching over [0.5, 1, 2, 3, 4]. As shown in Fig. 3, a small β overemphasizes the weighted cross-entropy loss, causing the local model to overfit difficult pixels and misalign with global features. A large β, in contrast, leads to plain feature alignment and fails to leverage challenging pixels. We choose β = 2 as the optimal setting and apply it consistently in all experiments.

Ablation Study. Compared to FedAvg, FMUG better captures challenging vessel structures, such as edges, capillaries, and crossing or adjacent vessels, as shown in Fig. 4. These hard-to-classify pixels, caused by morphology inconsistency across clients, often degrade performance. To address this, FMUG leverages uncertainty to guide the global model in helping local models focus on such regions, as shown in Tab. 2. Meanwhile, FVDA improves segmentation by separately aligning foreground and background features. When combined, FVAC maintains sensitivity to morphology variations while learning more robust and generalized representations, boosting segmentation performance.

5 Conclusion

In this paper, we are the first to explore federated vessel segmentation. To address two key challenges in federated vessel segmentation, namely vessel morphology inconsistency and vessel-background imbalance, we propose a pixel-wise divide-and-conquer approach called Federated Vessel-Aware Calibration (FVAC). By leveraging global uncertainty, our method directs the local models' attention to the hard kernel, which consists of pixels with high uncertainty caused by variable vessel morphology. Additionally, we use foregroundbackground separation alignment to mitigate the semantic bias induced by pixel-level class imbalance, ultimately leading to more robust semantic feature learning. Numerous experiments have clearly demonstrated the effectiveness and generalizability of the FVAC method, particularly in handling complex and detailed vascular structures. In the future, we will further explore innovative solutions to enhance the performance, robustness, and security of federated vessel segmentation, with a particular focus on reliably defending against potential adversarial threats to the global model.

Contribution Statement

Tian Chen, Wenke Huang, and Zhihao Wang contributed equally to this work and are marked with ∗ . Yongchao Xu is the corresponding author and is marked with † .

Acknowledgments

This work was supported in part by NSFC 62222112, 62225113, 62325111, 62176186, 62361166629, 62176188, and 623B2080, the Innovative Research Group Project of Hubei Province under Grants (2024AFA017) and the National Key Research and Development Program of China (2024YFC3308400).

We thank Zekun Shi for his valuable collaboration. This work would not have been possible without him.