Object-Level Backdoor Attacks in RGB-T Semantic Segmentation with Cross-Modality Trigger Optimization

1 Introduction

The security of deep neural networks has become a critical issue [Liu et al., 2023b; Jiao et al., 2023; Liu et al., 2023c; Liu et al., 2024c; Wang et al., 2025]. Backdoor attacks, as a prominent security threat, surreptitiously compromise model to preserve normal functionality on benign inputs while executing attacker-specified malicious behaviors when Figure 1: Exhibition of poisoned samples from different methods and the corresponding attack success rate at a 5% poisoning rate. We apply the trigger as ˆ X = X ⊕ T ∗ α. For regular CMTG and Stealthy CMTG, the transparency of trigger αRGB/αT is set to 1.0/1.0 and 0.1/0.2, respectively. Under this configuration, Stealthy CMTG yields a nearly invisible trigger in the RGB modality, and its concealment in the thermal modality is also significantly reduced, while still maintaining good performance. Detailed experimental results are presented in Table 1.

∗Corresponding author

encountering predefined trigger patterns. While extensively studied in vision tasks like classification and object detection [Tran et al., 2018; Gu et al., 2019; Yao et al., 2019; Wang et al., 2019; Liu et al., 2020; Liang et al., 2024; Liang et al., 2025], recent research shows growing interest in application to single-modality semantic segmentation [Li et al., 2021; Mao et al., 2023; Lan et al., 2024]. Notably, despite employing diverse trigger designs, these methods adhere to the Class-level Backdoor Attack (CBA) paradigm, indiscriminately affecting all victim-class objects without finegrained target differentiation, which fails to satisfy the operational requirements of attackers in real-world attack scenarios. The current widespread use of multi-modality sensors [Wang et al., 2022; Wang et al., 2024; Di et al., 2023; Liu et al., 2024a; Liu et al., 2025] makes this a critical challenge in multi-modality segmentation. To address this limitation, we propose a novel Object-level Backdoor Attack (OBA) method for RGB-T segmentation. OBA selectively interferes with the model's perception of the attacker's target of interest, while preventing the misclassification of other irrelevant objects within the victim class, enabling precise and effective attacks. The differences of these two attacks are illustrated in Figure 2.

Figure 3: Static trigger vs. Optimized trigger. For the static trigger, we directly apply the Hello Kitty pattern from Blended [Chen et al., 2017] method in RGB and grayscale formats (shown as (a)) to RGB-T image pairs. For the optimized trigger, we design a novel CMTG method to generate paired triggers with specific texture and pattern (shown as (b)) for the RGB and Thermal modalities.

To properly instantiate OBA, we introduce a novel Precise Data Poisoning (PDP) technique. Unlike existing methods that apply triggers to arbitrary image locations, PDP specifically targets the designated victim object, modifying solely the pixels associated with that particular object in the ground truth, as shown in Figure 5.

The efficacy of OBA is predominantly determined by the strategic selection of trigger patterns. Prior approaches like IBA utilize Blended's static trigger patterns We initially implement such approach directly in multi-modality scenario, as shown in 3(a), successfully demonstrating the core OBA capability. However, our empirical analysis revealed several issues: 1⃝ Data poisoning introduces backdoors by training models to associate triggers with target semantics. However, static patterns (e.g., unrelated Hello Kitty inserts in street scenes) create distributional divergence from natural data, impairing both learning efficiency and attack success rates. 2⃝

When migrating a single-modality trigger to a multi-modality task, simply converting an RGB image into grayscale may not align with the perception patterns of multi-modal segmentation models. 3⃝ Considering physical-world attack deployments, especially for thermal modality trigger, overly complex patterns (especially with pixel value changes) may be impractical to implement using simple thermal devices or cooling materials. This means that, theoretically, such attacks would be confined to the digital domain, severely weakening the practical relevance of the attack method.

To address the aforementioned challenges, we propose a novel optimization-based Cross-Modality Trigger Generation (CMTG) method. Inspired by the learning nature of the backdoor injection process, we propose a new form of trigger that aims to enhance the model's sensitivity to the trigger without adding extra learning burden. We leverage the concept of Universal Adversarial Perturbation [Moosavi-Dezfooli et al., 2017] and employ iterative optimization to construct texture patches infused with strong target class semantic information derived from the pretrained model, using all victim objects in the training set for training. Additionally, to address the challenge of implementing thermal modality trigger in the physical world, we design a relatively simple black-and-white checkerboard pattern, coupled with a differentiable mapping approach to allow collaborative optimization with the RGB modality texture patches. Through experimentation, we also find that, due to the strong semantics of the trigger, even with a high transparency setting for generating poisoned samples, it achieves a higher attack success rate compared to static patterns, as shown in Figure 1. This significantly improves the concealability of our method, greatly enhancing the practical value of the attack. Extensive experimental results validate our OBA paradigm and CMTG scheme. The main contributions are summarized as follows:

• A novel object-level attack paradigm that enables the attacker to accurately target a specific victim object of interest.
• A precise poisoning technique facilitates OBA by a new method for constructing poisoned samples.
• A trigger generation scheme that produces highly effective, covert, and physically realizable trigger pairs through cross-modality collaborative optimization.

• (a) RGB Input

(b) Class-level

(c) Object-level

Figure 5: Different label modification of CBA and OBA. The trigger is applied to the left "Car" (originally colored with purple). Existing class-level methods change the labels of all victim objects to the target class ("Bump" colored with orange). Our OBA only modifies the label of the designated object.

2 RelatedWork

2.1 RGB-T Semantic Segmentation

Current multi-modality semantic segmentation methods are designed around a dual-stream Siamese architecture that integrates thermal and RGB images. These methods fall into two main categories: symmetric and asymmetric structures. Symmetric models such as MFNet [Ha et al., 2017a], GMNet [Zhou et al., 2021], MDRNet [Zhang et al., 2021], EGFNet [Fan et al., 2022], and LASNet [Li et al., 2023] treat thermal and RGB features equally across all scales, while asymmetric models like RTFNet [Sun et al., 2019], FEANet [Deng et al., 2021], and FuseSeg [Sun et al., 2021] consider the thermal modality as a complement to the RGB modality. Beyond network architecture, effective multimodality feature fusion is essential. GMNet [Zhou et al., 2021] employs graded-feature extraction and fusion modules at various scales, whereas LASNet [Li et al., 2023] utilizes specialized modules for localization, activation, and sharpening. EAEFNet [Liang et al., 2023] introduces an attentionenhanced fusion module to model shared and unique features, promoting effective feature fusion while minimizing the impact of modality variations. Given its superior performance, EAEFNet is adopted as a suitable baseline model in this work.

2.2 Backdoor Attacks

Most methods execute backdoor attacks via data poisoning [Liao et al., 2018; Shafahi et al., 2018; Tang et al., 2020; Li et al., 2024; Liu et al., 2024b], which insert few modified samples into the training set and embed backdoor in model during training. Some methods manipulate the output by introducing additional modules [Tang et al., 2020; Qi et al., 2022] or by directly modifying the parameters of the model [Chen et al., 2021; Rakin et al., 2020]. Some work also explore physical-world attacks [Jiang et al., 2023; Yin et al., 2024]. Existing research on backdoor attacks has focused primarily on classification and detection tasks, and semantic segmentation tasks have not received sufficient attention. A digital backdoor attack targeted at segmentation models was first introduced, with a manually added black line at the top of all images [Li et al., 2021]. Following this, OFBA [Mao et al., 2023] emerged as another digital attack proposed in this domain. More recently, IBA [Lan et al., 2024], a distinct attack on segmentation models, disrupts the classification of victim class pixels by inserting triggers into non-victim pixels during inference.

3 Method

3.1 Object-level Backdoor Attack Paradigm

We first introduce a novel Object-level Backdoor Attack paradigm in semantic segmentation task. Given a dataset D, the task of semantic segmentation is typically defined at the image-level. Specifically, when an image X ∈ D is provided as input, the benign segmentation model S generates a corresponding semantic map, represented as:

Input: Victim dataset Dvic, Trigger T , Target class ytgt . Output: Modified victim dataset ˆDvic. 1 Initialize ˆDvic ← {}; 2 Calculate threshold δ = 2 ×Area(T ); 3 for (X i , Y i ) in Dvic do 4 Acquire Candidate victim objects Oivic ; 5 Initialize the victim object bank Bivic ← {}; 6 for ok in Oivic do // filter poor candidates 7 if Area(ok ) > δ then // seek centroid coordinate 8 (xk , yk ) ← (m10/m00, m01/m00); 9 Bivic = Bivic ∪ {[ok ; (xk , yk )]}; 10 end 11 end 12 Randomly select [ok ; (xk , yk )] fromBivic ; 13 Generate maskM for ok ; // modify input and label 14 Apply trigger at coordinate (xk , yk ) on X i to Acquire ˆ X i ; 15 Acquire ˆ Y i = (1 -M) × Y i +M× ytgt; 16 ˆDvic = ˆDvic ∪ ( ˆ X i , ˆ Y i ); 17 end

Here, ˜ Y X represents a semantic map with the same size as the input image, and each pixel is classified into the corresponding class of the object at the same position in the image. Now we redefine the segmentation task from the object-level perspective by considering the dataset D as consisting of object sets O of N categories, i.e., D = {O1,O2 . . . ,ON}, where Oi denotes the set of objects belonging to class yi, i.e. Oi = {o1i , o2i , . . . , omi }. In this case, the semantic segmentation task is reformulated as follows: when pick the j-th object oji from set Oi as input, the segmentation model S classifies all the pixels of that object into its corresponding class yi, i.e.,

where Y oji j denotes the semanticmapwith shape and size consistent with the object oji , and the class label yi.

Next, we introduce the difference between existing Classlevel Backdoor Attack(CBA) and our proposed Object-level Backdoor Attack(OBA) based on the definitions above. We denote the model with the injected backdoor as S∗ (·), the victim class as yvic, and the target class as ytgt. Given a image X = {o1 , o2 , . . . , oK }. The image with the trigger applied is denoted as ˆ X, where ˆ X = X ⊕T , and T denotes the trigger. For existing CBA methods, when the backdoor is triggered, the model will misclassify all victim objects in the image to the target class, while other non-victim objects remain correctly classified. This is formalized as:

Here ˜ Y ok tgt denoted a ok -shaped semantic map labed with ytgt. It is worth noting that in existing CBA methods, the trigger is applied at any location in the image. In contrast, our OBA applies the trigger to a specified victim object ol ∈ Ovic, i.e., ˆ X = {o1 , . . . , ˆ ol , . . . , oK }, where ˆ ol = ol ⊕ T . The goal of the OBA attack is that when the backdoor is triggerd, only the designated victim object ˆxl will be misclassified as the target class, while other non-victim objects and victim objects without the trigger applied will still be correctly classified. This is formalized as:

Thus, we have demonstrated the proposed Object-level Backdoor Attack (OBA) paradigm, which enables precise manipulation of attacked objects. Next, we will detail the two key components which facilitate OBA in RGB-T senario: the Cross-Modality Trigger Generation(CMTG) and the Precise Data Poisoning(PDP).

3.2 Cross-Modality Trigger Generation

Suppose we aim to obtain a trigger pair T = {TRGB, TT} with a resolution of a × a. We sample random noise from a uniform distribution U(0, 1) as the initial values (Fig. 4(a). To ensure the effectiveness of the triggers, we allow the triggers to be optimized within the range of 0 to 255 instead of constraining the pixel values within a small range as done in adversarial attacks. Since we design the thermal modality trigger as a black-and-white checkerboard pattern, the difficulty of creating the trigger in physical world increases when a is large. Additionally, in Sec. 5.3, we demonstrate that increasing the scale of the thermal trigger does not lead to significant performance improvements. Therefore we implement it with a b × b grid pattern (b ≤ a) and set b to a smaller value (e.g., b = 8 in the experiments). Hence, we need to develop a differentiable mapping method from continuous texture to discrete pattern. We propose such a mapping function M(·):

Π(·) denotes the projection operation which ensures the value of TT are either 0 or 1. Upsample(·) and sgn(·) denote the interpolation-based upsampling and element wise sign operation, respectively. Thus, we can ensure that TT undergo collaborative optimization with TT during the iterative process.

After obtaining the trigger pair T = {TRGB, TT}, we begin the iterative optimization process. Our goal is to generate a trigger pair that encodes strong semantic information related to the target class, such that when we apply T to the victim object, the model S will be misled by the the trigger and incorrectly classify the corresponding object pixels into the ytgt class. To achieve this, we apply T to victim object o ∈ Ovic and feed it into a pre-trained benign model S(·). Simultaneously, we generate a pseudo-label Y o tgt that has the same shape and size as o, but with the target class label ytgt. We then optimize T using the following objective function:

where Lseg denotes the loss for training the segmentation model. By minimizing the loss between the prediction and the pseudo-label, we inject semantic information with respect to the target class domain from the pre-trained model into the trigger. Subsequently, we update the trigger T iteratively with gradient descent in a manner similar to the PGD method:

where k = 0, 1, . . . ,K - 1, α is the step size for trigger generation. We use Π(·) to constrain TRGB and TT within the value ranges of [0, 1] and [-0.5, 0.5], respectively. The loss function Lseg is the same as the one used during the training process of the segmentation model. In EAEFNet, the overall loss function is defined as follows:

where LDice and LSCE are denoted as Dice loss and Soft Cross Entropy loss.

3.3 Precise Data Poisoning

Figure 4(a) illustrates the details of the proposed PDP. Given a training set D, victim class yvic, and target class ytgt are selected. We select paired RGB-T images that contain victim objects from D, denoted as Dvic = {X 0 , ..., X M ; Y 0 , ..., Y M }. Each RGB-T image pair X i contains K victim objects, denoted as Oivic = {o0 , ..., oK }, as poisoning candidates. Next, we screen the poisoning candidates to avoid complete coverage by the trigger, particularly when the target size is too small. Specifically, we compute the total number of pixels for each poisoning candidate, denoted as Area(ok ) = Count(ok ). When Area(ok ) < δ, the ok is removed from Oivic , where δ = 2 × Count(T ). After that, we obtain a new set of poisoning candidates. We then apply a trigger to each sample pair. To ensure trigger placement on the victim object ok , we position the trigger at its centroid. We seek the centroid coordinates (x, y) through x = m10/m00 and y = m01/m00, where m10 and m01 denote first-order moments, m00 is zero-order moment. This establishes the victim object bank Bivic = [Oivic ; (x, y)]. When randomly designating a victim object o from Oivic , we apply a trigger at its centroid (x, y) to acquire modified RGB-T input through ˆ X i = X i ⊕ T ∗ α, where α represents the transparency of the trigger pattern when applied, with values ranging from [0,1]. When α = 1.0 , it indicates that the trigger pattern is directly pasted onto the image. Then we generate mask M from o where pixels belonging to the object region are set to 1 and the rest to 0. We re-label the region of o as the target class ytgt, which is represented as ˆ Y i = (1-M)×Y i +M×ytgt. The modified labels are illustrated by the near car as shown in Figure 5(c). After completing the aforementioned steps, we acquire the final poisoned dataset Dpoi = ˆDvic ∪ Dnon vic. Refer to Algorithm 1 for the details of our PDP.

4 Experiments

4.1 Experimental Setup

Datasets. We adopt two popular datasets (e.g., MFNet [Ha et al., 2017b] and FMB [Liu et al., 2023a]) to perform the evaluation. MFNet dataset includes 9 classes, in which the background class is labeled 0. Its training, validation and testing sets contain 784, 392 and 393 pairs of images, respectively. FMB dataset describes complex urban street scenes in various severe conditions, e.g., dense fog, heavy rain, and low light. It contains images with 15 categories, and its training and testing sets include 1,220 and 280 pairs, respectively.

Metrics. We adopt three existing metrics ASR, PBA, and CBA from IBA [Lan et al., 2024] with some modifications:

1. Attack Success Rate (ASR). This metric denotes the percentage of victim pixels erroneously classified as the target class due to the trigger's influence. Given the victim object subjected to the trigger containing a total of Nvic pixels and Nsuc pixels successfully misclassified as the target class, the Attack Success Rate is computed as: ASR = Nsuc/Nvic.
2. Poisoned Benign Accuracy (PBA). This metric assesses the model's ability to correctly classify non-victim objects amidst triggered images by calculating the mean Intersection over Union (mIoU) between predictions and labels with triggered victim object excluded.
3. Clean Benign Accuracy (CBA). This metric measures the model's performance on benign testing by computing the mIoU between its predictions and the original labels. For a model affected by poisoning, CBA should closely approximate the results achieved by the model trained on clean data.

Implementation Details. We employ two test sets to validate the effectiveness of our method. The first is the selected victim-poisoned test set containing victim-class objects with trigger applied. The second is the original benign test set without any modification. The training is conducted on a single RTX3090. During training, images are resized to 480 × 640, with a batch size of 8. All other hyperparameters remain consistent with the original paper [Liang et al., 2023].

4.2 Backdoor Attacked Results

Table 1 reports quantitative scores of the backdoor attacks using our precise data poisoning (PDP) technique on the RGB-T semantic segmentation model, across both MFNet and FMB datasets. As we can see, the PBA and CBA metrics consistently maintain stability compared to the scores of our attack paradigm on benign test samples. Intuitively, Figure 6 shows several poisoned RGB-T segmentation cases. These results show that our proposed backdoor attack paradigm incorporating the PDP technique can accurately misclassify the designated victim object (e.g., car) while correctly classifying the non-victim objects.

4.3 Robustness of OBA

We further validate the robustness of the object-level manipulation ability facilitated by the trigger. We first assess the trigger's efficacy on a single object, as depicted in (a) and (b) in Figure 9. It can be seen that when plural victim objects coexist within an image, the application of triggers can lead to the misclassification of any object. Subsequently, we examine the trigger's impact on multiple objects concurrently. The results are illustrated as (c) and (d) in Figure 9. In (c), we apply the triggers to two victim objects simultaneously, resulting in the misclassification of both objects. At this time, the model only misclassified the car. This underscores the efficacy of our OBA in accurately manipulating victim objects.

5 Discussion

5.1 Attacks with Different Target Classes

We test our object-level backdoor attack (OBA) paradigm using different target classes (i.e., "Person", "Vegetation", "Road" and "Sky") when the victim class is defined as 'Car'. The scores of our OBA on the FMB dataset in ASR, PBA, and CBA metrics are presented in Table 3. By comparison, these scores show stability without notable fluctuations. Intuitively, Figure 7 presents the corresponding visual examples. It is evident that as the target class changes, our OBA paradigm causes the segmentation model to misclassify the victim "Car" as "Person", "Vegetation", "Road" and "Sky" respectively. We can conclude that the OBA demonstrates robust attack ability across varied target classes.

5.2 Attacks with Different Trigger Sizes

We test our Object-level Backdoor Attack paradigm (OBA) using the RGB-T poisoned samples with triggers of different sizes (e.g., 15×15, 25×25, 35×35, and 45×45) on the FMB dataset. Poisoning rate is set to 5%. The results in Table 4 indicate that irrespective of changes in trigger size, the ASR, PBA, and CBA metrics maintain nearly consistent scores. Based on these results, we can conclude that the backdoor attack capability of our OBA paradigm is minimally affected by trigger size, indicating its high robustness.

5.3 Attacks with Different Thermal Trigger Scales

We evaluate the impact of different thermal trigger scales on attack performance, and the results are shown in Table 5. As can be seen, the attack success rate does not exhibit a significant improvement with the increase in scale. In fact, when the

• (a) Attack Car-1
• (b) Attack Car-2 (c) Attack Car-1&2

Figure 9: Analysis of robustness of object-level precise attack on single or multiple objects. Note that, the victim class is "Car". The trigger positions are highlighted in the input images. The triggers can induce the misclassification of any victim object (see (a)-(c)).

scale becomes too large (scale = 35), the attack success rate even decreases. However, the increase in scale does lead to an improvement in the model's task performance, as reflected in the higher PBA and CBA scores.

5.4 Attacks with Different Transparency

We conduct tests on different transparency combinations of RGB triggers and thermal triggers on FMB and MFNet, with the experimental results shown in the Figure 8. We test five transparency combinations: [0.1, 0.2, 0.5, 0.7, 1.0]. From this experiment, it can be observed that, during the data poisoning process for injecting backdoors into the model, the texture information in the RGB modality plays a dominant role.

6 Conclusion

This paper presents the first backdoor attack method designed for RGB-T semantic segmentation. Addressing the limitations of existing approaches, we propose OBA - a novel object-level backdoor attack paradigm that enables fine-grained target control. Our framework implements two key innovations: (1) a precision data poisoning technique that selectively manipulates only trigger-designated objects while preserving surrounding segmentation accuracy, and (2) a cross-modality trigger generation method that embeds target class semantics through collaborative optimization of both modalities. Experimental results demonstrate the effectiveness and high attack success rate of OBA.

Acknowledgments

This work was supported by the Fundamental Research Funds for the Central Universities, Sun Yat-sen University (No. 23xkjc010), Shenzhen Science and Technology Program (No.KQTD20221101093559018), National Natural Science Foundation of China (No.62306343) and China Postdoctoral Science Foundation (No.2024M753741).