A Novel SAM Coupling Mechanism for SAR Image Segmentation

1 Introduction

The Segment Anything Model (SAM) [Kirillov et al., 2023; Ravi et al., 2024; Carion et al., 2025] achieves zero-shot object segmentation based on the SA-1B dataset; however, due to its training data predominantly comprising natural images, its performance is constrained in non-natural domain images, especially in Synthetic Aperture Radar (SAR) images characterized by weak boundary contrast and low signal-to-noise ratios, as well as in fine-grained object segmentation tasks. Interactive prompts tend to confuse foreground objects with background noise. Furthermore, SAM employs the Vision Transformer (ViT) architecture [Liu et al., 2021], which relies on global feature matching for prompt-based inference, thus lacking effective constraints on pixel-level boundary delineation.

∗Corresponding author.

To enhance the generalization capability of the SAM in non-natural domains, existing research primarily focuses on improvements in structural adaptation and prompt mechanisms. On the one hand, approaches such as intermediate representations, lightweight adaptation [Chen et al., 2024a; Gong et al., 2024; Yang et al., 2023], internal structure enhancement [Chen et al., 2023; Pu et al., 2025; Zhang et al., 2025a], multi-scale and local feature modeling [Chen et al., 2024c], and hybrid architecture designs [Xiong et al., 2026] are employed to mitigate issues such as low contrast, complex textures, and fine-grained edge ambiguity. On the other hand, strategies involving semantic or visual language prompts [Kweon and Yoon, 2024; Ren et al., 2024], prompt self-correction [Radman and Laaksonen, 2025], and multiprompt fusion are adopted to strengthen target representation and improve segmentation performance in complex scenarios [Hu et al., 2024]. However, current methods often treat backbone modeling and prompt generation as disjoint processes, lacking effective collaboration, which restricts crossscene adaptability.

We categorize the synergistic relationship between the backbone network and prompt generation into two types: strong coupling and weak coupling. Strong coupling training frequently results in gradient conflicts and unstable training processes due to inconsistent optimization objectives among networks [Chen et al., 2018]. Consequently, most approaches adopt a weakly coupled multi-stage strategy, introducing adaptive or hierarchical instructions during the learning of backbone characteristics to achieve joint optimization [Wang et al., 2025; Li et al., 2024]. A representative method, AlignSAM [Huang et al., 2024], optimizes prompt configurations in the final binary mask layer through reinforcement learning, as illustrated by the black arrow in Figure 1. However, this kind of coupling mechanism operates in the low-dimensional output layer, imposing limited constraints on the prompt generation process and struggling to handle complex structures and weak boundary scenarios.

In this paper, we train a reinforcement learning network based on Proximal Policy Optimization (PPO) [Schulman et al., 2017] as a point prompt optimizer and design a hypothesis testing-based reward function inspired by statistical principles to provide an interpretable credit assignment for prompt generation. The optimizer determines subsequent prompt points based on the current state and the reward signal from the backbone, thus achieving a high-dimensional closed-loop coupling between the backbone network and the prompt mechanisms, as illustrated by the yellow arrow in Figure1. The framework relies solely on the standard SAM interface, ensuring compatibility with various versions. Experimental results validate its effectiveness, particularly in crossdomain applications such as SAR image segmentation. Our main contributions are summarized as follows:

• A novel higher-dimensional coupling mechanism for SAM has been proposed, which enables synergistic and iterative optimization between prompt generation and the primary architecture through reward signal feedback prior to binary masking.
• A point impact decomposition module was designed that constructs point-level contributions from highdimensional features before masking and employs contribution regularization to achieve a fine-grained representation of the mask boundaries.
• A PPO-based reinforcement learning prompt optimizer is introduced, coupled with the establishment of a reward function grounded in point-wise contribution hypothesis testing, thereby enabling stable point-level prompt strategy learning.
• The methodology has been validated on the FSS-seg and HRSID datasets, demonstrating performance that significantly outperforms existing baseline approaches.

2 RelatedWork

2.1 Non-natural Domain Image Adaptation of SAM

Early applications of SAM in non-natural domain imagery have primarily focused on medical images, which are characterized by low boundary contrast and poor signal-to-noise ratios. To address its limitations in feature representation and spatial modeling, research efforts have been made to refine SAM through intermediate representations [Yang et al., 2023], prompt modeling, and lightweight adaptation architectures [Gong et al., 2024; Chen et al., 2024a]. Subsequently, SAM has been extended to remote sensing and radar domains, where performance in SAR imaging, multi-resolution scenarios, and noisy or blurry environments [Pu et al., 2025; Chen et al., 2024c; Zhang et al., 2025a] is enhanced through parameter fine-tuning, category-level adaptation, cross-resolution feature alignment, and image priors.

2.2 PromptingMethods in Interactive Segmentation

Early interactive segmentation methodologies predominantly used energy minimization and graph-cut frameworks [Rother et al., 2004], leveraging user-supplied points or boundary constraints to achieve global optimization [Boykov and Jolly, 2001]. With the advent of deep learning, prompts such as clicks, extreme points, or coarse bounding boxes have been encoded into end-to-end neural networks, enabling more efficient segmentation [Maninis et al., 2018; Chen et al., 2022]. Reinforcement learning approaches model the segmentation process as sequential decision-making, autonomously generating or refining prompts to minimize manual intervention [Liao et al., 2020; Wang et al., 2024b; Chen et al., 2025b; Huang et al., 2025]. Following the introduction of SAM, research has explored prompt enhancement [Dai et al., 2023], sequential optimization, as well as the integration of reinforcement and adversarial learning [Wang et al., 2025; Liu et al., 2025b], to achieve automated prompt generation and calibration [Chen et al., 2025c; Li et al., 2024], thus enhancing the robustness and performance of segmentation.

2.3 Weak Coupling Between the Trunk and Prompts in SAM

Current collaborative research based on SAM predominantly employs weakly-coupled strategies involving frozen backbone modules [Chen et al., 2025c; Li et al., 2024; Zhang et al., 2025b]. GenSAM [Hu et al., 2024] maps text prompts to visual prompts, but struggles with cross-modal alignment and provides limited generalization enhancement for SAM through textual information. DAPSAM [Wei et al., 2024] addresses the key connection between generalpurpose models and non-natural domains via domain adaptation, but faces challenges in balancing extreme domain adaptation with lightweight efficacy. The representative AlignSAM [Huang et al., 2024] Binary mask feedback reinforcement learning optimization. However, its reliance on lowdimensional mask-level prompts restricts the localization accuracy for fine-grained structures and low-contrast boundaries.

3 Methodology

The framework of the proposed method is shown in Figure 2. Based on the SAM segmentation backbone , we design modules and strategies such as PID and Hypothetical Testing Reward, and provide a unified feature and decoding interface for the SAM series of models.

In SAM, the input image is first processed by the image encoder to extract multi-scale features, which are flattened into a sequence Fflat. The point prompts are embedded in the feature vectors by the prompt encoder and concatenated with the learnable Mask tokens Tmask, the IoU token Tiou and Fflat to form the Transformer input sequence. This sequence is then processed by a bidirectional Transformer to fuse image features with prompt information.

To improve SAM's transfer efficiency in cross-domain tasks, we incorporate Low-Rank Adaptation (LoRA) [Zhong et al., 2024] into the multi-head self-attention layers of the SAM image encoder. For each projection matrix W, a trainable low-rank side path is added to the Query and Value projections. During fine-tuning, only the side-path parameters A and B are updated, while the original parameters W0 are kept fixed.

3.1 Point Impact Decomposition

Within the SAM mask decoder, we incorporate a point impact decomposition mechanism that disassembles the intricate effects of point prompts on mask prediction into relatively independent feature representations, thereby rendering the point impact observable and steerable. This framework comprises three components: point-wise feature modulation, point-wise mask generation, and linear mask reconstruction, which collectively enable explicit modeling and aggregation of the per-point contributions from each prompt.

Conditional Feature Modulation at the Point Level. To embed the information of the prompt points into the backbone feature maps and obtain the corresponding point modulation vectors, each point embedding pi is processed by a lightweight MLP:

Here, ϕ(·) represents a fully connected two-layer network, where W1 and W2 are learnable weight matrices, and b1 and b2 are bias terms. The modulation vector gi encodes the relative importance of the point i for each channel of features. Subsequently, a channel-wise multiplication operation ⊙ is applied to transform the global feature Fbase into the corresponding point-specific feature map fi. This enables each point to derive a semantically distinct feature representation from the shared feature Fbase, guided by its own modulation vector:

Point-level Mask Generation. Each point-specific feature map fi is decoded into a spatial contribution mask through a shared lightweight convolutional head:

The convolutional head h(·) consists of a 3 × 3 convolution, layer normalization, ReLU activation, and a 1 × 1 convolution. Generates the logarithmic map of the original contribution Mi for each point. The shared convolutional head design ensures that all points follow the same decoding rules, making the contribution masks of different points comparable.

Linear Mask Reconstruction. The ultimate segmentation mask is generated through a linear aggregation of pixel-level contributions to reconstruct the final prediction:

(4)

The weights wi and biases b are learnable parameters, wfg and wbg are globally learnable scalars, PL denotes point-type labels, and the final probability mask is obtained through a Sigmoid function per-pixel σ(·). This framework transforms the global segmentation task into a linear combination of points-level contributions, where each Mi explicitly repre-

sents the spatial impact of the corresponding point, allowing visualization and interpretable analysis of the segmentation process and providing reward variables for subsequent pointprompter optimization.

PID Regularization. To ensure that the point-level contribution mask Mi output of the PID decoder is interpretable, distinguishable, and locally concentrated, this paper introduces two important regularization expressions:

The local consistency in pointwise Llocal serves as a basis for the interpretability of PID. It constrains the contribution mask of each point prompt to exhibit behavioral patterns within its local neighborhood that are consistent with the semantic meaning of the point type. For foreground points, higher activation values are encouraged at the point location pi compared to other positions within a radius neighborhood r; on the contrary, suppression is applied for background points. Mathematically, this is expressed as the summation of difference penalties computed separately for different point types based on activation variances within their respective neighborhoods:

Here, B denotes the batch size, N represents the number of prompt points per sample, m is the batch index, i is the point index within the sample, and Nr refers to the set of neighborhood points centered at the point pi with radius r, where |Nr| indicates the number of neighboring points and is used to average the loss within the neighborhood. M (m) i (pi) represents the contribution mask value of the i-th point in the m-th sample at its own position, while M (m) i (q) denotes the contribution mask value of the same point at the position of a neighborhood point q. signPLi is a sign function determined based on the point label. The term max(0, ·) denotes the Hinge function, which imposes a penalty only when the local relational constraint is violated.

The loss of overlap Loverlap is designed to improve discriminability between the contribution masks of different points, thereby preventing multiple points from generating redundant and ambiguous contributions to the same region. It penalizes the sum of the element-wise spatial products of the contribution masks for any pair of points:

The normalization factor B · (N2 ) is introduced to eliminate the impact of batch size B and the number of points per sample N on the magnitude of the loss. For each sample m, the summation is performed on all pairs (i, j) satisfying i < j to avoid redundant calculations. Here, M (m) i and M (m) j represent the contribution masks of the points i-th and j-th in the sample m-th, respectively. The L1 norm of the product matrix yields a quantitative measure of the degree of overlap. Minimizing Loverlap encourages non-overlapping contribution masks across distinct points, thereby enhancing the discriminability and complementarity of pointwise contributions.

3.2 RL-based Point Prompt Optimization

Building upon the reinforcement learning-based point prompt optimizer, a new Hypothesis Test Reward Function based on the reward variable provided by PID has been designed., enabling precise credit assignment at the point level for each action generated by the policy network. The iterative procedure is elaborated in Figure 3.

State Representation and Action Space. At each iterative step t of the reinforcement learning process, the state st of the policy network is constructed by concatenating three component elements of the characteristic [Fimg;Mt-1;Et], which correspond to the deep features of the image, the prediction from the previous step and the embedding of the current prompt, respectively. In this way, the policy network integrates visual semantics, segmentation progression, and interaction history together. Its action space consists of selecting the next coordinate point on the image plane as the prompt.

Reward Function. The global reward metric quantifies the final segmentation quality by evaluating the predicted mask obtained in the current interaction round using the Dice coefficient. Specifically, upon completion of the t-th interaction round, the global reward is defined as:

Here, M (t) final denotes the predicted mask obtained under the current prompt set P (t) , Y represents the ground truth mask, and ε is a numerical stability constant.

To resolve credit assignment in interactive point-based prompt learning, we introduce a hypothesis testing-based reward that quantifies the marginal contribution of each prompt point, allowing backbone-guided prompt optimization. At the t-th decoding step, the hypothetical testing reward for the j-th action point is defined as:

Where Nt denotes the number of points triggered in the decoding process in the interaction t-th, Mk represents the point-level mask generated by the PID decoder for the k-th point prompt, wk is a type-dependent weight learnable and b3 is the bias term. The first term corresponds to the global segmentation quality under the actual prediction where point j is active, while the second term corresponds to the hypothetical scenario where point j is inactive, constructed by removing its corresponding contribution from the decoded logits. The difference between these two terms quantifies the performance gain attributable to point j in the final outcome of the segmentation.

Policy Network. We employ a proximal policy optimization algorithm based on the Actor-Critic architecture[Konda and Tsitsiklis, 1999] to train the policy network. The Critic network, denoted as Vϕ, functions as a state-value estimator to compute advantage functions to reduce the variance in policy gradients. For advantage estimation, we adopt the generalized advantage estimation method:

Here, rt represents the immediate reward obtained after executing an action in the state st. γ denotes the discount factor, while Vϕ(·) signifies the state-value function parameterized by the Critic network. δt refers to the temporal-difference error and At corresponds to the generalized advantage estimate (GAE), where λ serves as the decay coefficient of GAE to balance bias and variance.

3.3 Loss Function

The segmentation backbone loss is composed of binary crossentropy loss, Dice loss, and point-guided loss:

The binary cross-entropy loss quantifies the pixel-wise agreement between the predicted probabilities and the ground truth labels. Given the predicted logit map M, the loss function is defined as follows:

The Dice loss function optimizes the overlap between the predicted foreground region and the ground truth foreground region, demonstrating strong robustness against class imbalance issues. Given the predicted mask as ˆ Y , the loss is defined as:

Point-guided loss encourages the alignment of predicted mask boundaries with the user-provided point prompt locations to enhance the model's comprehension of interactive intent:

For each point prompt pi, the distance from its location to the boundary of the predicted mask ∂ ˆ Ym is calculated, where ∂ denotes the boundary and ˆ Ym represents the predicted mask for the m-th sample. δ is a non-negative threshold hyperparameter defining the acceptable alignment tolerance along the boundary. The discrepancy between the computed distance and the threshold δ yields a non-positive value when the distance is less than or equal to δ, and a positive value when the distance exceeds δ. Minimizing this loss term encourages the predicted mask boundary to align as closely as possible with the point prompt location, thereby enhancing the accuracy of the model in boundary localization.

4 Experiments

4.1 Datasets and Implementation Details

Datasets. Based on FuSARShip1.0 [Hou et al., 2020], a pixel-level semantic segmentation dataset, termed FSS-Seg, is constructed. Through annotation of the original data, the resulting data set contains 11,857 ship images that span seven semantic categories. Following the standard evaluation protocol, the dataset is divided into 9,470 images for training and 2,387 images for testing. Additionally, we also evaluate our method on the public HRSID dataset, which contains 5,604 SAR ship images and 16,951 ship targets. Detailed dataset statistics and specifications are reported in Appendix.

Implementation Details. All models were implemented using NVIDIA GeForce RTX 3090 24GB GPUs and PyTorch 2.7.1. The models were trained with AdamW optimizer, starting with an initial learning rate of 1e-3, a batch size of 8, and ran for 100 epochs. The input images were uniformly resized to 224×224 pixels, and the random seed was fixed to 1234.

Upon completion of the segmentation backbone training, the learning rate for the policy network was set at 2e-4, with a discount factor γ = 0.98 and a GAE parameter λ = 0.97. Each interactive episode was limited to a maximum of 8 steps, starting with an initial token count of 1.

4.2 Results and Analysis

The experiments cover three generations of SAM base models and their representative enhanced variants, as well as SAR-oriented specialized adaptation methods. Quantitative evaluations were conducted using Dice coefficient, Intersection over Union (IoU), Precision, and 95th percentile Hausdorff Distance (HD95). The experimental results are summarized in Table 1. The following analysis focuses on the performance on FSS-Seg.

SAM1. We compared the proposed mechanism with the original SAM1[Kirillov et al., 2023], SAM-Adapter[Chen et al., 2023], CAM-SAM[Kweon and Yoon, 2024], Grounded SAM[Ren et al., 2024], and PPO[Liu et al., 2025b]. The presented method demonstrates substantial improvements over the original SAM and its enhanced variants in key accuracy metrics. Compared to the reinforcement learning-based PPO

approach, our method achieves relative gains of 6.68% in Dice and 11.19% in IoU.

SAM2. We conducted a comparative analysis of the proposed mechanism against the original SAM2[Ravi et al., 2024], SAM2-Adapter[Chen et al., 2024b], DetSAM2[Wang et al., 2024b], SAM-OCTA2[Chen et al., 2025b] and CA-SAM2[Huang et al., 2025]. Compared to SAM2, the proposed approach achieved improvements of 63.75% and 23.41% in Dice and IoU, respectively. Furthermore, despite the coexistence of multiple strong adaptation Figure 5：Distribution of rewards r for three strategies (Random, Heuristic, and PPO) measured at K = 8 points across three experiments. Dots and crosses denote the corresponding measurements. Positive and negative rewards are indicated by r > 0 and r < 0, respectively.

Among them：(a)Bulk Carrier sample (b)Container sample (c)GeneralCargo sample.Horizontal axis represents point index.The vertical axis represents the hypothesis testing reward score. and heuristic prompt methods, our method still attained the best performance in three evaluation metrics.

https://anonymous.4open.science/r/PI D-93D2 SAM3. Our approach still improves overall accuracy, although with a smaller improvement compared to SAM1 and SAM2. However, the results validate the cross-generational transferability of this mechanism and confirm its capacity to sustain stable performance gains on more advanced models.

SAR-specific methods. We selected CWSAM[Pu et al., 2025], pointSAM[Liu et al., 2025a], and PolSAM[Wang et al., 2024a] for a comparative evaluation. The CWSAM approach achieved a comparatively high accuracy of 90.03%, but exhibited notable boundary discrepancies, with HD95 reaching 23.6307 and IoU maintained at 70.23%. In contrast, the proposed method achieves a more favorable balance between overlap quality and boundary precision.

Visualization. For a comprehensive comparative analysis, Figure4 presents qualitative results on the FSS-Seg dataset, comparing our model with three other approaches including Grounded SAM[Ren et al., 2024] and CASAM[Kweon and Yoon, 2024]. It can be observed that, under challenging scenarios, such as strong scattering points, streak artifacts, tailing interference, or elongated target structures, certain methods are prone to generating false background detections or exhibiting fragmented object contours. In contrast, the proposed method demonstrates superior capability in preserving the structural continuity and contour integrity of vessel regions while effectively suppressing clutter interference.

4.3 Module Comparison

In three distinct experiments, we conducted distributional measurements of rewards r for three types of policy, namely random policy[Dai et al., 2023], heuristic policy[Dai et al., 2023], and proximal policy optimization[Schulman et al., 2017], at designated points K = 8. As indicated in Table2, the PPO policy exhibits the optimal global performance, substantially outperforming the other two methods. Furthermore, Figure5 illustrates the distribution of rewards between the eight measurement points. Compared with alternative approaches, the PPO policy maintains more consistent positive rewards in the majority of points, with fewer occurrences of negative rewards, and the reward distribution is more concentrated.

4.4 Ablation Study

Table 3 presents the significant performance improvements achieved by each component and their synergy effects. The PPO strategy [Schulman et al., 2017] improves the Dice coefficient, achieving a balance between filling in omissions and suppressing false detections. The PID module improves the Dice coefficient and the IoU by optimizing boundary delineation, but its effect in reducing false detections remains limited. When they operate independently without synergy, inconsistent optimization directions may lead to boundary jittering. However, the collaborative integration proposed in this paper resolves such conflicts, achieving the best performance and the lowest HD95 value in all three metrics, thus simultaneously optimizing regional characteristics and boundary features.

5 Conclusions

This study proposes a high-dimensional coupled optimization mechanism for SAM to address the joint optimization of prompt generation and segmentation. Through pointwise impact decomposition, the backbone decoder quantifies perpixel contributions and constrains contribution maps via regularization. A hypothesis testing-based reward mechanism is further introduced for the reinforcement learning policy network to guide autonomous prompt optimization with finegrained weight allocation, enabling weakly coupled optimization between the backbone and prompt modules. Experimental results on SAR semantic segmentation datasets demonstrate significant improvements in segmentation performance and cross-domain adaptability, providing an interpretable and learnable framework for automated prompt optimization.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62002146, and in part by the Youth Project of the Liaoning Provincial Department of Education under Grant LJ212410165083.