Physical Adversarial Camouflage Through Gradient Calibration and Regularization

1 Introduction

Object detection is essential for applications like autonomous driving, where robustness and reliability are crucial due to its integration into safety-critical systems. Adversarial attacks pose a serious threat by manipulating object textures to deceive detection algorithms. This can lead to misclassifications or undetected objects, resulting in potential severe real-world consequences like accidents.

∗Corresponding author.

Physical adversarial camouflage presents significant challenges compared to its digital counterparts, as it requires modifying real-world objects in a manner that ensures the attack remains effective when transitioning from the digital domain to uncontrolled physical environments. This process involves optimizing an object's surface texture to deceive detection systems across varying angles and distances, aligning with the inherent variability in how cameras perceive objects under diverse perspectives. However, achieving robust adversarial camouflage necessitates addressing several critical challenges. One key difficulty stems from the localized nature of texture manipulation. Specifically, from any given viewpoint, only the visible portions of the object's texture can be optimized. This viewpoint-specific focus often neglects the broader context of the object's surface, which may lead to suboptimal performance when interactions across different perspectives are considered holistically. Furthermore, inconsistencies in the object's appearance, caused by variations in viewing distance, angle, or environmental conditions, further compound the complexity of this task.

Existing methods typically utilize differentiable renderers to optimize adversarial textures by simulating object appearances across diverse viewpoints and environmental conditions. However, we identify two critical issues within this optimization pipeline that adversely affect attack performance. The first issue arises from distance-dependent sampling density: as the distance between the camera and the object changes, the number of pixels used to render the object varies, as illustrated in Figure 1. This variation leads to inconsistent gradient sparsity during backpropagation. This inconsistency, in turn, results in uneven texture modifications across different distances. The second issue stems from potential redundancy or conflict among gradient directions within a minibatch, as illustrated in Figure 2. Gradients derived from similar viewpoints tend to generate redundant updates, while gradients from distinct viewpoints may contradict and cancel one another, thus impeding effective texture optimization.

To address these identified challenges, we propose a novel adversarial camouflage framework built upon two core strategies: Nearest Gradient Calibration (NGC) and Loss- Prioritized Gradient Decorrelation (LPGD). First, to mitigate the issue of inconsistent gradient sparsity, NGC propagates gradients from sampled points to neighboring unsampled points on the same surface, ensuring local continuity and consistent sparsity in texture updates across varying distances. Second, to resolve the gradient redundancy and conflicts arising from diverse viewpoints within a minibatch, LPGD prioritizes gradients according to their associated adversarial loss and decorrelates them through orthogonalization. This orthogonalization process ensures that each prioritized gradient provides complementary and non-conflicting information, thereby overcoming challenges related to redundancy and cancellation.

We conducted experiments in realistic simulated environments, demonstrating that our adversarial camouflage effectively evades object detection across diverse viewpoints, distances, weather conditions, and multiple detectors. It outperforms state-of-the-art methods by 13.46% in attack effectiveness across distances and 11.03% across angles. Real-world experiments further confirm its practicality and robustness.

Our contributions are as follows:

• We identify two challenges in optimizing adversarial camouflage, including inconsistent gradient sparsity due to distance-dependent sampling density, and redundant or conflicting gradients across viewpoints.
• We propose Nearest Gradient Calibration (NGC) to propagate gradients from sampled to unsampled texture points, ensuring local continuity in gradient updates across varying distances.
• We introduce Loss-Prioritized Gradient Decorrelation (LPGD) to prioritize and decorrelate gradients via orthogonalization, resolving redundancy and conflicts across viewpoints.
• Extensive experiments demonstrate that our approach achieves superior attack performance, enhanced robustness to environmental variations, and improved transferability across diverse object detectors compared to stateof-the-art methods.

2 RelatedWork

2.1 Object Detection

Object detection plays a pivotal role in computer vision, evolving through various strategies. Anchor-based methods include two-stage detectors [Girshick, 2015; He et al., 2017;

Figure 2: Illustration of gradient conflicts from different perspectives

Ren et al., 2015], which use region proposals for classification and bounding box regression, and single-stage models [Liu et al., 2016; Redmon and Farhadi, 2018], which frame detection as a regression problem to achieve real-time performance. Conversely, anchor-free methods [Duan et al., 2019] eliminate predefined anchors by directly predicting object centers and dimensions. Recent developments like DETR [Zhu et al., 2021] introduce an end-to-end detection approach utilizing transformer architectures. Nevertheless, these methods are susceptible to adversarial attacks.

2.2 Adversarial Attacks

Adversarial attacks involve the deliberate alteration of images to mislead deep neural networks [Szegedy et al., 2014], resulting in incorrect predictions. These attacks are divided into digital and physical based on the domain of perturbation. Digital attacks directly apply crafted perturbations to digital images, with techniques such as FGSM [Goodfellow et al., 2014] and PGD [Madry et al., 2017] demonstrating notable success, especially in classification. These strategies have been adapted for object detection but lose effectiveness in real-world scenarios. In contrast, physical attacks [Wei et al., 2024; Zhu et al., 2023] introduce perturbations in the real world, encompassing adversarial patches and camouflage. Patch-based attacks apply perturbations to flat surfaces, effective only from certain viewpoints, while adversarial camouflage covers entire object surfaces, allowing for multi-angle attacks.

2.3 Adversarial Camouflage

Current adversarial camouflage methods often utilize differentiable 3D renderers to project 3D objects as rendered images from multiple angles, optimizing 3D textures end-toend via backpropagation. For instance, Wang et al. [Wang et al., 2022] introduced Full-coverage Camouflage Attack (FCA), optimizing a vehicle's UV textures through a neural renderer for enhanced multi-viewpoint camouflage. Similarly, the Dual Attention Suppression (DAS) attack by Wang et al. [Wang et al., 2021a] reduces visibility to both models and human observers. Another approach involves optimizing 2D texture patterns projected onto vehicle surfaces. Suryanto et al. [Suryanto et al., 2022] proposed the Differentiable Transformer Attack (DTA), employing a neural renderer to simulate realistic effects like shadows. Additionally, ACTIVE [Suryanto et al., 2023] improves camouflage aesthetics with enhanced texture mapping and background color integration. Furthermore, RAUCA [Zhou et al., 2024] utilizes advanced rendering to account for environmental factors such as diverse weather conditions. However, existing techniques may not fully resolve challenges arising from sampling inconsistency and conflicts in gradient updates, which can affect overall attack effectiveness.

3 Method

3.1 ProblemDefinitions

Adversarial camouflage [Wei et al., 2024] aims to optimize the texture of a target object to achieve objectives such as evading detection or inducing misclassification. A 3D object is represented by the tuple (M,T), whereMdenotes the mesh structure and T represents its texture. The object's image, rendered under various camera parameters φ ∈ Φ (e.g., different camera angles and distances), is produced using a differentiable renderer, R. The rendering process first computes the UV coordinates as a function of the camera parameter φ, defined by:

The final image of the target object is obtained by sampling color values from the texture T at the computed UV coordinates:

where FS denotes the sampling function. Since the renderer R does not incorporate background information, a common approach for creating physically realistic images is to overlay the rendered object onto authentic backgrounds. In this study, we utilize background images generated by the CARLA simulator [Dosovitskiy et al., 2017]. The composite image is created by replacing the background in the rendered image using a segmentation mask S:

where IB represents the background image.

To conduct a physical evasion attack, the texture is optimized to mislead the target model, causing detection failure or misclassification. The adversarial texture is derived by solving:

where Fθ is the object detector with parameters θ, y is the ground-truth label covering classification and localization, and L is the suppression loss function. A lower loss implies a reduced probability of predicting the target label correctly. The objective is to hinder the model's ability to accurately predict the target label y.

Existing Obstacles. Achieving effective adversarial camouflage in multi-view settings requires accounting for variations in distance and viewing angles. Current methods [Wang et al., 2022; Zhou et al., 2024] optimize textures by rendering images under diverse conditions using differentiable renderers [Kato et al., 2018]. However, they fail to address two significant challenges: ❶ Varying sampling densities at different distances. This variation causes the gradients used for texture updates to exhibit inconsistent levels of sparsity across distances. ❷ Conflicting texture update directions across different viewing angles. Naively merging these updates undermines the effectiveness of the attack.

3.2 Nearest Gradient Calibration

As illustrated in Equation 2, the rendered image samples colors from the texture T using UV coordinates. The pixel count representing the target object varies with rendering distance: closer objects occupy more pixels, resulting in denser texture sampling, whereas distant objects are represented by fewer pixels, leading to sparser sampling. Typically, only sampled points are updated during texture optimization. Due to varying rendering distances producing different sets of sampled points, inconsistencies in texture updates can emerge across different distances. This issue arises from emphasizing pixellevel optimization while neglecting the continuity required to preserve the integrity of local surface regions.

To address this problem, we propose a simple yet effective method called Nearest Gradient Calibration (NGC). The core idea of NGC is to extend the gradient of each sampled point to its neighboring unsampled points. This propagation ensures that gradient updates over the texture remain locally continuous, thereby preserving the consistency of texture updates across varying distances.

We define the trainable region of the texture T as T′ , which is constrained by a mask K that restricts updates to specific areas, such as the visible outer surface of a vehicle. Within T′ , the sampled subset of points is denoted as TU(φ), determined by the camera parameters φ. For simplicity, we refer to TU(φ) as TU in the following sections. The remaining unsampled points are defined as TF = T′ \ TU. Formally, this relationship is expressed as:

After each iteration of backpropagation, only the gradients of the sampled points, ∇TU , are computed.

Ideally, adjacent points on the same surface should be updated collectively to maintain continuity. However, the renderer's sampling mechanism may result in sparsely distributed gradients, leaving intermediate points without gradients. To address this, we propagate the sparse gradients to their neighboring unsampled points using a nearest neighbor search, implemented via the KD Tree algorithm [Zhou et al., 2008]. For each unsampled point p ∈ TF, we calculate the Euclidean distance to all points in TU to identify the nearest neighbor q. The gradient of the nearest sampled point q ∈ TU is assigned to p provided that ∥p - q∥2 ≤ τ, where τ is the search radius. This ensures that gradients are assigned only to unsampled points within a specified local range, maintaining spatial locality. If no sampled point within the threshold is found, the gradient for p remains zero. Formally, the gradient assignment is defined as:

After assigning gradients to TF, we obtain the final extended gradient ∇T′ . By applying NGC, gradients are smoothly propagated across the texture surface, ensuring local continuity, as illustrated in Figure 3. This approach effectively addresses the inconsistencies caused by varying sampling densities, resulting in more uniform and coherent texture updates across different rendering distances. The detailed procedure for this method is summarized in Algorithm 1.

3.3 Loss-Prioritized Gradient Decorrelation

In multi-view physical adversarial optimization, overlapping texture regions observed from varying azimuth and elevation angles present substantial challenges, primarily due to gradient redundancy and conflicts. Gradient redundancy arises when gradients from different viewpoints exhibit similar update directions, leading to inflated gradient norms that destabilize optimization. Conversely, gradient conflicts occur when gradients act in opposing directions, resulting in partial or complete cancellation of updates and diminishing their

Input : Texture T, Mask K, camera parameter set Φ, Search radius τ Output: Extended gradients ∇T′ Define T′ = T ⊙ K; Initialize KD-Tree K with TU; for each optimization step do Sample camera parameter φ ∈ Φ; Identify sampled points TU(φ); TF ←T′ \TU; Compute gradients ∇TU ; Query nearest neighbors for p ∈ TF using K; for each p ∈ TF do Retrieve nearest q and distance d; if d ≤ τ then Assign ∇TF (p) ← ∇TU (q); else Assign zero gradient: ∇TF (p) ← 0; end end end return ∇T′ ;

effectiveness. If unaddressed, these issues can potentially reduce efficiency of adversarial optimization.

To address these problems, we propose the LossPrioritized Gradient Decorrelation (LPGD) method, which integrates gradient orthogonalization with prioritization based on loss values. This method ensures that the optimization focuses on the most challenging viewpoints and resolves redundancy and conflicts among gradients to produce stable and efficient updates.

In the LPGD method, gradients from k viewpoints, corresponding to the trainable part of the texture T′ in Equation 5, are first sorted according to their loss values, L(φi), such that L(φ1) ≥ L(φ2) ≥ · · · ≥ L(φk). This prioritization ensures that gradients from more challenging viewpoints are processed first. Specifically, the gradients {∇T′ (φ1), ∇T′ (φ2), . . . , ∇T′ (φk)} are ordered in this manner before applying gradient orthogonalization.

Once gradients are prioritized, we proceed with gradient orthogonalization to resolve redundancy and conflicts. This step is crucial to ensure that each gradient contributes unique, non-redundant information to the optimization process. We apply a Schmidt orthogonalization [Leon et al., 2013] procedure, which projects each gradient onto the orthogonal complement of the subspace spanned by the previously processed gradients. Formally, given the i-th gradient ∇T′(φi), its orthogonalized counterpart ∇⊥T′ (φi) is computed as:

This recursive projection removes any components of Algorithm 2: Loss-Prioritized Gradient Decorrela-

∇T′ (φi) that are redundant or conflicting with respect to previously orthogonalized gradients, ensuring that all gradients are mutually decorrelated. As a result, the gradients provide complementary information about the texture updates, rather than conflicting or overlapping contributions.

Once all gradients are orthogonalized, the final update direction is determined by averaging the orthogonalized gradients. The final update direction is computed as:

where ∇⊥T′(φi) are the orthogonalized gradients. By combining orthogonalization with averaging, the method achieves stable updates that respect the contributions of all viewpoints without being dominated by redundant or conflicting gradients. The detailed procedure for this method is summarized in Algorithm 2.

Notably, our proposed NGC and LPGD methods are compatible and can be combined in a straightforward manner. NGC propagates the gradient of points q ∈ TU, i.e., ∇TU , to points p ∈ TF, i.e., ∇TF , to obtain the extended gradient ∇T′ at each iteration. Subsequently, LPGD decorrelates these extended gradients ∇T′ within a minibatch via loss-prioritized orthogonalization to obtain the final gradients ∇∗T′ .

4 Experiment

4.1 Settings

Implementation Details. We utilize the CARLA simulator [Dosovitskiy et al., 2017] to generate datasets. In line with previous studies [Zhou et al., 2024], we capture simulated images to construct a comprehensive training set. Our training set comprises 20,000 images captured from diverse angles and distances to enhance texture generation. We focus on the Audi E-Tron model, as explored in prior research [Wang et al., 2021a; Wang et al., 2022; Suryanto et al., 2022; Suryanto et al., 2023]. For evaluation, adversarial camouflage is applied to the vehicle within CARLA, with images captured at elevation angles of {0◦ , 5◦ , 10◦ , 15◦ , 20◦ , 30◦ , 45◦ , 60◦ }, along with 2-degree azimuth increments for a thorough 360◦ sweep. Extended evaluations encompass distances of {5, 7.5, 10, 12.5, 15} meters and five distinct weather conditions: noon, sunset, night, foggy, and rainy. In real-world tests, the camouflage patterns are printed on 1:24 scale Audi E-Tron models, with images captured from various angles and distances for further analysis. Texture optimization is conducted using the Adam optimizer with a learning rate of 0.1, employing ModernGL [Dombi, 2020] for differentiable rendering and segmentation mask S generation. We optimize the textures over three epochs, with all experiments performed on a single NVIDIA A100 80GB GPU.

Comparison Baselines. Our framework is evaluated against several state-of-the-art adversarial camouflage techniques, including DAS [Wang et al., 2021a], FCA [Wang et al., 2022], DTA [Suryanto et al., 2022], ACTIVE [Suryanto et al., 2023], and RAUCA [Zhou et al., 2024]. For our evaluations, as illustrated in Figure 4, we transform both our optimized textures and those of the baselines into standardized UV textures, which are constrained by a mask K applied to the optimizable regions.

Evaluation Metrics. Following previous studies [Zhou et al., 2024], we assess the attack effectiveness of adversarial camouflage using the AP@0.5 benchmark [Everingham et al., 2015], which is a standard measure capturing both recall and precision at a detection IOU threshold of 0.5.

Target Detection Models. Consistent with prior studies [Zhou et al., 2024], we utilize YOLOv3 [Redmon and Farhadi, 2018] as the white-box target detection model for generating adversarial camouflage. To evaluate the effectiveness of the optimized camouflage, we test it against a range of popular object detection models, treating them as blackbox models, except for YOLOv3. This evaluation encompasses models such as the one-stage detector YOLOX [Ge et al., 2021], two-stage detectors Faster R-CNN (FrRCN) [Ren et al., 2016] and Mask R-CNN (MkRCN) [He et al., 2017], as well as transformer-based detectors DETR [Carion et al., 2020] and PVT [Wang et al., 2021b]. Each model is pretrained on the COCO dataset and implemented using the MMDetection framework [Chen et al., 2019].

4.2 Evaluation in Physical Simulation Settings

In this section, we compare our method to state-of-the-art (SOTA) adversarial camouflage approaches. Comprehensive experiments are conducted in the CARLA simulation platform [Dosovitskiy et al., 2017] to assess attack performance across various viewpoints, distances, weather conditions, and transferability across different object detectors.

Robustness Across Multiple Angles. We evaluated the effectiveness of adversarial attacks across 360◦ azimuth angles and elevation angles from 0◦ to 60◦ . At each elevation, images were captured encircling the target vehicle. Experiments were conducted in a white-box setting with YOLOv3, and results are shown in Table 1. Our findings indicate that attacks are more challenging at lower elevation angles, as evidenced by higher AP@50 scores due to non-camouflaged features like vehicle tires aiding detection. At higher elevations, where such features are less visible and only camouflaged areas remain, detector performance declines, resulting in more successful attacks. Our method consistently achieves the best results, with the lowest AP@50 scores across all angles, demonstrating robust performance across perspectives. Specifically, compared to the state-of-the-art RAUCA [Zhou et al., 2024], our approach improves attack performance by an average of 11.03%.

Robustness AcrossMultiple Distances. We conducted extensive experiments to assess performance across distances from 5 to 15 meters. Using the same protocol as for elevation evaluation, we averaged attack performance over full 360◦ azimuth and 0◦ -60◦ elevation angles. Results in Table 2 reveal two key insights. First, although detection capability weakens with distance, attack performance doesn't necessarily improve due to loss of texture details at greater ranges, which can diminish camouflage effectiveness. Second, our method consistently achieves high attack success rates across all distances, outperforming existing approaches by 13.46%, highlighting the robustness and adaptability of the proposed NGC strategy.

Robustness Under Different Weather Conditions. Following previous studies [Zhou et al., 2024], we evaluated our method under varying weather conditions, including Noon (high illumination), Night (low illumination), Sunset (moderate illumination), as well as Foggy and Rainy conditions, which reduce visibility. The results, summarized in Table 3, show that most existing methods are sensitive to these conditions. Extreme illumination (Noon and Night) leads to performance degradation due to overexposure or reduced visibility. Moreover, adverse weather conditions like Foggy and Rainy also impact performance by obscuring critical visual adversarial features. In contrast, our method demonstrates significantly improved robustness across all tested conditions. Notably, compared to state-of-the-art (SOTA) method RAUCA [Zhou et al., 2024], our method achieves a consistent improvement, with an average gain of 19.38% across all weather scenarios. Despite this, high illumination (Noon) still poses challenges due to overexposure, providing an avenue for further enhancement.

Transferability Across Different Object Detectors. To evaluate the ability of our method to transfer across various object detectors, we carried out comprehensive experiments on detectors with diverse architectures, including singlestage, two-stage, and transformer-based models. These experiments compared our approach with several state-of-theart (SOTA) methods. The outcomes, detailed in Table 4, reveal notable variations in method performance across different detector types. Notably, our method not only surpasses existing SOTA approaches in the white-box scenario but also achieves consistently stronger results in most black-box tests, highlighting its superior transferability.

4.3 Evaluation in Real-World Settings

In this section, we evaluate our method in real-world settings by printing and applying the optimized texture onto a car, as shown in Figure 5. The evaluation considers varying perspectives and distances. For perspectives, we tested three elevation angles and averaged results over a full 360◦ azimuth. For distances, we assessed three distinct ranges. Results in Figure 6 show that real-world conditions are more challenging than digital simulations. Lower elevation angles yield weaker performance, aligning with simulation trends. As distance increases, AP drops, indicating improved attack effectiveness-likely due to degraded detector performance at longer ranges. Additionally, while camouflage detail fades with distance, even robust features like uncovered tires become less detectable.

4.4 Computational Complexity

The NGC module has a complexity of O(M logN), where M = |TF | and N = |TU| denote the numbers of unsampled and sampled points, respectively. This stems from KDTree-based nearest neighbor search. The LPGD component introduces a complexity of O(m2k), due to gradient orthogonalization over a texture of size m2 across k optimization steps. In runtime tests, NGC introduces an 18.0% overhead (+0.0209s) per iteration, while LPGD adds 4.6% (+0.0054s), compared to the base forward and backward time (0.1163s).

4.5 Ablation Studies

In this section, we perform ablation studies to analyze the impact of our proposed NGC and LPGD strategies. As shown in Table 6, both components individually improve attack performance, with their combination yielding the best results. For NGC, we further examine the effect of the search radius τ, which determines the furthest neighbor allowed for gradient propagation. For LPGD, we explore how the batch size used for gradient orthogonalization influences performance. To isolate the effects, we set the other component's effectiveness to zero during each study.

Effect of Search Radius τ. We evaluate our NGC strategy across a range of search radius values τ from 0 to 16. The results in Figure 7(a) show that attack performance initially improves with increasing τ, reaches an optimal point, and then declines. This validates the effectiveness of our method compared to the case where τ = 0. However, excessively large τ allows gradients to propagate beyond the target surface, introducing noise to irrelevant textures. Thus, selecting an appropriate τ is critical for balancing local continuity and noise control.

Effect of Batch Size k. We assess the LPGD strategy by varying the batch size from 1 to 40. The results, depicted in Figure 7 (b), show a trend of increasing attack performance with larger k. This validates our method's effectiveness in reducing conflicts by considering a broader global context and decorrelating redundancy and conflict between gradients.

5 Conclusion

In this study, we identify two key challenges in physical adversarial camouflage: inconsistency in gradient sparsity and conflicting gradient updates. We introduce a novel framework that incorporates Nearest Gradient Calibration (NGC) and Loss-Prioritized Gradient Decorrelation (LPGD). NGC promotes gradient propagation from sampled to nearby unsampled texture points, ensuring local continuity across varying distances. LPGD prioritizes and orthogonalizes gradients to resolve redundancy and conflicts for gradients derived from different viewpoints. Our approach significantly enhances attack performance, robustness across diverse environments, and transferability across different detectors.

Acknowledgements

This work was supported by the Fundamental Research Funds for the Central Universities, Sun Yat-sen University under Grants No. 23xkjc010; the Shenzhen Science and Technology Program (No.KQTD20221101093559018).