RoboFusion: Towards Robust Multi-Modal 3D Object Detection via SAM

1 Introduction

Multi-modal 3D object detection plays a pivotal role in autonomous driving (AD) [Wang et al., 2023a; Song et al., 2024a]. Different modalities often provide complementary information. For instance, images contain richer semantic representations, yet lack depth information. In contrast, point clouds offer geometric and depth details, but they are sparse and lack semantic information. Therefore, effectively leveraging the advantages of multi-model while mitigating their limitations contributes to enhancing the robustness and accuracy of perception systems [Song et al., 2023].

∗Corresponding author

With the emergence of AD datasets [Geiger et al., 2012; Caesar et al., 2020; Zhang et al., 2023c], state-of-theart (SOTA) methods [Liu et al., 2023; Bai et al., 2022; Chen et al., 2022; Huang et al., 2020; Li et al., 2023; Song et al., 2024b] on 'clean' datasets [Geiger et al., 2012; Caesar et al., 2020] have achieved record-breaking performance. However, they overlook the exploration of robustness and generalization in out-of-distribution (OOD) scenarios [Dong et al., 2023]. For example, the KITTI dataset [Geiger et al., 2012] lacks severe weather conditions. When SOTA methods [Chen et al., 2022; Li et al., 2023; Liu et al., 2023] learn from these sunny weather datasets, can they truly generalize and maintain robustness in severe weather conditions like snow and fog?

The answer is 'No', as shown in Fig. 1 and verified in Table 3. People often utilize domain adaptation (DA) techniques to address these challenges [Wang et al., 2023b; Tsai et al., 2023; Peng et al., 2023; Hu et al., 2023]. Although DA techniques improve the robustness of 3D object detection and reduce the need for annotated data, they have some profound drawbacks, including domain shift limitations, label shift issues, and overfitting risks [Oza et al., 2023]. For instance, DA techniques may be constrained if the differences between two domains are significant, leading to performance degradation on the target domain.

Recently, both Natural Language Processing (NLP) and Computer Vision (CV) have witnessed the appearance and the power of a series of foundation models [Kirillov et al., 2023; OpenAI, 2023; Zhao et al., 2023; Zhang et al., 2023a], resulting in the emergence of new paradigms in deep learning. For example, a series of novel visual foundation models (VFMs) [Kirillov et al., 2023; Zhao et al., 2023; Zhang et al., 2023a] have been developed. Thanks to their extensive training on huge datasets, these models exhibit powerful generalization capabilities. These developments have inspired new ideas, leveraging the robustness and generalization abilities of VFMs to achieve generalization in OOD noisy scenarios, much like how adults generalize knowledge when encountering new situations, without relying on DA techniques [Wang et al., 2023b; Tsai et al., 2023].

Inspired by the success of VFMs in CV tasks, in this work, we intend to use these models to tackle the challenges of multi-modal 3D object detectors in OOD noise scenarios. Therefore, we propose a robust framework, RoboFusion, which leverages VFMs like SAM to adapt a 3D multi-modal object detector from clean scenarios to OOD noise scenarios. In particular, the adaptation strategies for SAM are as follows. 1) We utilize features extracted from SAM rather than inference segmentation results. 2) We propose SAM-AD, which is a pre-trained SAM for AD scenarios. 3) We introduce a novel AD-FPN to address the issue of feature upsampling for aligning VFMs with multi-modal 3D object detector. 4) To further reduce noise interference and retain essential signal features, we design a Depth-Guided Wavelet Attention (DGWA) module that effectively attenuates both high-frequency and low-frequency noises. 5) After fusing point cloud features and image features, we propose Adaptive Fusion to further enhance feature robustness and noise resistance through selfattention to re-weight the fused features adaptively. We validate RoboFusion's robustness against OOD noise scenarios in KITTI-C and nuScenes-C datasets [Dong et al., 2023], achieving SOTA performance amid noise, as shown in Fig. 1.

2 RelatedWork

2.1 Multi-Modal 3DObject Detection

Currently, multi-modal 3D object detection has received considerable attention on popular datasets [Geiger et al., 2012; Caesar et al., 2020]. BEVFusion [Liu et al., 2023] fuse multimodal representations in a unified 3D or BEV space. Trans- Fusion [Bai et al., 2022] builds a two-stage pipeline where proposals are generated based on LiDAR features and further refined using query image features. DeepInteraction [Yang et al., 2022] and SparseFusion [Xie et al., 2023] further optimize the camera branch on top of TransFusion. Previous methods are highly optimized to achieve the best performance on clean datasets. However, they ignore common factors in the real world (e.g., bad weather and sensor noise). In this work, we consider a real-world robustness perspective and design a robust multi-modal 3D perception framework, RoboFusion.

2.2 Visual FoundationModels for 3DObject Detection

Motivated by the success of Large Language Models (LLMs) [OpenAI, 2023], VFMs start to be explored in CV community. SAM [Kirillov et al., 2023] leverages ViT [Dosovitskiy et al., 2020] to train on the huge SA-1B dataset, containing 11 million samples, which enables SAM to be generalized to many scenes. Currently, there have been a few research endeavors aiming at integrating 3D object detectors with SAM. For instance, SAM3D [Zhang et al., 2023b], as a LiDAR-only method, solely transforms LiDAR's 3D perspective into a BEV (Bird's Eye View) 2D space to harness the generalization capabilities of SAM, yielding sub-optimal performance on 'clean' datasets. Another in progress work, 3D-Box-Segment-Anything 1 , tries to utilize SAM for 3D object detection. This indicates the highly attention of SAM like foundation models in 3D scenes in the literature. Our RoboFusion, as a multi-modal method, gives clear strategies to leverage the generalization capabilities of VFMs to address the OOD noise challenges inherent in existing 3D multimodal object detection methods.

1 https://github.com/dvlab-research/3D-Box-Segment-Anything

3 RoboFusion

In this section, we present RoboFusion, a framework that harnesses the robustness and generalization capabilities of VFMs such as SAM [Kirillov et al., 2023] for multi-modal 3D object detection. The overall architecture is depicted in Fig. 2

and comprises the following components: 1) SAM-AD & AD-FPN module which obtains robust multi-scale image features, 2) Depth-Guided Wavelet Attention (DGWA) module which employs wavelet decomposition to denoise depthguided image features, 3) Adaptive Fusion module which adaptively fuses point cloud features with image features.

3.1 SAM-AD&AD-FPN

Preliminaries

SAM [Kirillov et al., 2023], a VFM, achieves generalization across diverse scenes due to its extensive training on the large-scale SA-1B dataset-with over 11 million samples and 1 billion high-quality masks. Currently, SAM family [Kirillov et al., 2023; Zhao et al., 2023; Zhang et al., 2023a] primarily support 2D tasks. However, directly extending VFMs like SAM to 3D tasks presents a gap. To address this, we combine SAM with multi-modal 3D models, merging 2D robust feature representations with 3D point cloud features to achieve robust fused features.

SAM-AD

To further adapt SAM with AD (autonomous driving) scenarios, we perform pre-training on SAM to obtain SAM-AD. Specifically, we curate an extensive collection of image samples from well-established datasets (i.e., KITTI [Geiger et al., 2012] and nuScenes [Caesar et al., 2020]), forming the foundational AD dataset. Following DMAE [Wu et al., 2023], we perform pre-training on SAM to obtain SAM-AD in AD scenarios, as shown in Fig. 3. We denote x as a clean image from the AD dataset (i.e. KITTI [Geiger et al., 2012] and nuScenes [Caesar et al., 2020]) and η as a set of noise images generated by [Dong et al., 2023] based on x. And the noise type and the severity are randomly chosen from the four weather (i.e., rain, snow, fog, and strong sunlight) and the five severities from 1 to 5, respectively. We employ the image encoder of SAM [Kirillov et al., 2023] , MobileSAM [Zhang et al., 2023a] as our encoder while the decoder and the reconstruction loss are the same as DMAE [Wu et al., 2023]. For FastSAM [Zhao et al., 2023], we adopt YOLOv8 2 to pretrain FastSAM on the AD dataset. To avoid overfitting, we use random resizing and cropping as data augmentation. We also set the mask ratio as 0.75 and have trained 400 epochs on 8 NVIDIA A100 GPUs.

AD-FPN

As a promptable segmentation model, SAM has three components: image encoder, prompt encoder and mask decoder. Generally, the image encoder can provide high-quality and highly robust image embedding for downstream models, while the mask decoder is only designed to provide decoding services for semantic segmentation. Furthermore, what we require are robust image features rather than the processing of prompting information by the prompt encoder. Therefore, we employ SAM's image encoder to extract robust image features. However, SAM utilizes the ViT series [Dosovitskiy et al., 2020] as its image encoder, which excludes multi-scale features and provides only high-dimensional low-resolution features. To generate the multi-scale features required for object detection, inspired by [Li et al., 2022a], we design an AD-FPN that offers ViT-based multi-scale features. Specifically, leveraging height-dimensional image embedding with stride 16 (scale=1/16) provided by SAM, we produce a series of multi-scale features Fms with stride of {32, 16, 8, 4}. Sequentially, we acquire multi-scale feature Fi ∈ R H4 × W4 ×Ci by integrate Fms in a bottom-up manner similar to FPN [Lin et al., 2017].

3.2 Depth-GuidedWavelet Attention

Although SAM-AD or SAM has the capability to extract robust image features, the gap between 2D and 3D domains still persists and cameras lacking geometric information in a corrupted environment often amplify noise and give rise to negative migration issues. To mitigate this problem, we propose the Depth-Guided Wavelet Attention (DGWA) module, which can be split into two steps. 1) A depth-guided network is designed, that adds geometry prior to image features by combining image features and depth features from a point cloud. 2) The features of an image are decomposed into four wavelet subbands using the Haar wavelet transform [Liu et al., 2020a], then attention mechanism allows to denoise informative features in the subbands.

2 https://github.com/ultralytics/ultralytics

Formally, given image features Fi ∈ R H4 × W4 ×Ci and raw points P ∈ RN,Cp as input. We project P onto the image plane to acquire a sparse depth map S ∈ RH×W×2 . Next, we feed S into the depth encoder DE(·), which consists of several convolution and max pooling blocks, to acquire depth features Fd ∈ R H4 × W4 ×Ci . Afterward, we leverage convolution encode (Fi, Fd) to acquire depth-guided image features ˆ Fi ∈ R H4 × W4 ×16 , given by

Subsequently, we employ discrete wavelet transform (DWT), a reversible operator, to partition the input ˆ Fi into four subbands. Specifically, we encode the rows and columns of the input separately into one low-frequency band ˜ f LL i ∈ R H8 × W8 ×4 and three high-frequency bands ( ˜ f LH i , ˜ f HL i , ˜ f HH i ) ∈ R H8 , W8 ,4 , with the low-filter fL = ( 1 √2 , 1 √2 ) and the high-filter fH = ( 1 √2 ,- 1 √2 ). In this state, the low-frequency band retains coarse-grained information while the high-frequency band retains fine-grained information. In other words, it is easier to capture the mutation signal, so as to filter the noise information. We concatenate the foursubband features along channel dimension to acquire wavelet features ˜ Fi = [ ˆ f LL i , ˆ f LH i , ˆ f HL i , ˆ f HH i ] ∈ R H8 × W8 ×16 . Next, we perform wave-attention Attω to query informative features in the wavelet features. Concretely, we employ ˆ Fi as a Query and ˜ Fi as a Key/Value given by

Finally, we leverage the IDWT (inverse DWT) to convert ˜ Fi back to ˆ Fi and integrate this converted ˆ Fi and Fatt to obtain denoise features Fout ∈ R H 16 × W16 ×16 by

where Fout preserves informative features and restrains redundant mutation noise in the frequency domain.

3.3 Adaptive Fusion

Following the incorporation of image depth features within the DGWA module, we propose the Adaptive Fusion technique to combine point cloud attributes with robust image features enriched with depth information. Specifically, different types of noise affect LiDAR and images to different degrees, which raises a corruption imbalance problem. Therefore, considering the distinct influences of various noises on LiDAR and camera, we employ self-attention to re-weight the fused features adaptively as shown in Fig. 4. The corruption degree of modality-specificity is dynamic, and self-attention mechanism allows adaptive re-weighting features to enhance informative features and suppress redundant noise.

4 Experiments

4.1 Datasets

We perform experiments on both the clean public benchmarks (KITTI [Geiger et al., 2012] and nuScenes [Caesar et al., 2020]) and the noisy public benchmarks (KITTI-C[Dong et al., 2023] and nuScenes-C [Dong et al., 2023]).

Clean Datasets

The KITTI dataset provides synchronized LiDAR point clouds and front-view camera images, consists of 3,712 training samples, 3,769 validation samples and 7,518 test samples. The standard evaluation metric for object detection is the mean Average Precision (mAP), computed using recall at 40 positions (R40).

The nuScenes dataset is a large-scale 3D detection benchmark consisting of 700 training scenes, 150 validation scenes, and 150 testing scenes. The data are collected using six multiview cameras and a 32-channel LiDAR sensor. It includes 360-degree object annotations for 10 object classes. To evaluate the detection performance, the primary metrics used are the mean Average Precision (mAP) and the nuScenes detection score (NDS).

Noisy Datasets

In terms of data robustness, [Dong et al., 2023] has designed 27 types of common corruptions for both LiDAR and camera, with the aim of benchmarking the corruption robustness of existing 3D object detectors. [Dong et al., 2023] has established corruption robustness benchmarks 3 , including KITTI-C and nuScenes-C, by synthesizing corruptions on public datasets. Specifically, we utilize KITTI-C and nuScenes-C in our work. It is worth noting that Ref. [Dong et al., 2023] has only added noise to the validation dataset and kept the train and test datasets clear.

4.2 Experimental Settings

Network Architecture

Our RoboFusion consists of three variants: RoboFusion-L, RoboFusion-B, and RoboFusion-T, which utilize the models SAM-B [Kirillov et al., 2023], FastSAM [Zhao et al., 2023], and MobileSAM [Zhang et al., 2023a], respectively. It is noteworthy that due to the convolutional operations of FastSAM in RoboFusion-B which is capable of generating multi-scale features, the AD-FPN module is not employed. Since KITTI and nuScenes are distinct datasets with varying evaluation metrics and characteristics, we provide a detailed description of our RoboFusion settings for each dataset.

3 https://github.com/thu-ml/3D Corruptions AD

We validate our RoboFusion on the KITTI dataset using Focals Conv [Chen et al., 2022] as the baseline. The input voxel size is set to (0.05m, 0.05m, 0.1m), with anchor sizes for cars at [3.9, 1.6, 1.56] and anchor rotations at [0, 1.57]. We adopt the same data augmentation solution as Focals Conv-F.

We validate our RoboFusion on the nuScenes dataset using TransFusion [Bai et al., 2022] as the baseline. The detection range for the X and Y axis is set at [-54m, 54m] and [-5m, 3m] for the Z axis. The input voxel size is set at (0.075m, 0.075m, 0.2m), and the maximum number of point clouds contained in each voxel is set to 10. It is noteworthy that the Adaptive Fusion module is applied exclusively to Focals Conv rather than TransFusion, while TransFusion uses its own fusion module.

Training and Testing Details

Our RoboFusion is meticulously trained from scratch using the Adam optimizer and incorporates several foundation models as image encoders including SAM, FastSAM and MobileSAM. To enable effective training on the KITTI and nuScenes datasets, we utilize 8 NVIDIA A100 GPUs for network training. Additionally, the runtime is evaluated on an NVIDIA A100 GPU. Specifically, for KITTI, our RoboFusion based on Focals Conv[Chen et al., 2022] involves training for 80 epochs. For nuScenes, our RoboFusion based on TransFusion [Bai et al., 2022] has 20 epochs of training. During the model inference stage, we employ a non-maximal suppression (NMS) operation in the Region Proposal Network (RPN) with an IoU threshold of 0.7. We select the top 100 region proposals to serve as inputs for the detection head. After refinement, we apply NMS again with an IoU threshold of 0.1 to eliminate redundant predictions. For additional details regarding our method, please refer to OpenPCDet 4 .

4.3 Comparing with State-of-the-art

We conduct evaluations on the clean datasets KITTI and nuScenes, as well as the noisy datasets KITTI-C and nuScenes-C. While SOTA methods are primarily focused on achieving high accuracy, we place greater emphasis on the robustness and generalization of the methods. These factors are crucial for the practical deployment of 3D object detection in AD scenarios, making the evaluation on the noisy datasets more important in our perspective.

4 https://github.com/open-mmlab/OpenPCDet

Results on the Clean Benchmark

As shown in Table 1, we compare our RoboFusion with SOTA methods, including Voxel R-CNN [Deng et al., 2021], VFF [Li et al., 2022b], CAT-Det[Zhang et al., 2022] , Focals Conv-F [Chen et al., 2022], and LoGoNet [Li et al., 2023] on the KITTI validation and test sets. As shown in Table 2, we also compare our RoboFusion with SOTA methods, including FUTR3D [Chen et al., 2023], TransFusion [Bai et al., 2022], BEVFusion [Liu et al., 2023], DeeepInteraction [Yang et al., 2022], CMT [Yan et al., 2023] and SparseFusion [Xie et al., 2023], on the nuScenes test and validation sets. Our RoboFusion has achieved SOTA performance on the clean benchmarks (KITTI and nuScenes).

Results on the Noisy Benchmark

In the real-world AD scenarios, the distribution of data often differs from that of training or testing data, as shown in Fig. 1 (a). Specifically, Ref. [Dong et al., 2023] provides a novel noisy benchmark that includes KITTI-C and nuScenesC, which we primarily use to evaluate the weather and sensor noise corruptions, including rain, snow, fog, and strong sunlight, density, cutout, and so on. In addition, comparisons of our RoboFusion with SOTA methods in other settings are presented in the Appendix 5 .

As shown in Table 3, SOTA methods, including SECOND [Yan et al., 2018], PointPillars [Lang et al., 2019], PointRCNN [Shi et al., 2019], PV-RCNN [Shi et al., 2020], SMOKE [Liu et al., 2020b], ImVoxelNet [Rukhovich et al., 2022], EpNet[Huang et al., 2020], Focals Conv-F[Chen et al., 2022], and LoGoNet[Li et al., 2023], experience a significant decrease in performance on the noisy scenarios, particularly for weather conditions such as snow and rain. It can be attributed to the fact that the 'clean' KITTI dataset does not include examples in snowy or rainy weather. On the other hand, VFMs like SAM-AD have been trained on a diverse range of data and exhibit robustness and generalization to OOD scenarios, leading to higher performance on our RoboFusion metric. Furthermore, multi-modal methods like LoGoNet, and Focals Conv-F demonstrate better robustness and generalization in sensor noise scenarios, while LiDAR-only methods like PVRCNN [Shi et al., 2020] are more robust in weather noise scenarios. This observation motivates our research on adaptive fusion schemes for point cloud and image features. Overall, in the KITTI-C [Dong et al., 2023] dataset, our RoboFusion's performance is nearly on par with the clean scene, indicating high level of robustness and generalization.

As shown in Table 4, SOTA methods including PointPillars [Lang et al., 2019], SSN [Zhu et al., 2020], CenterPoint [Yin et al., 2021], FCOS3D [Wang et al., 2021], PGD [Wang et al., 2022a], DETR3D [Wang et al., 2022b], BEVFormer [Li et al., 2022c], FUTR3D [Chen et al., 2023], TransFusion [Bai et al., 2022], BEVFusion[Liu et al., 2023], DeepInteraction [Yang et al., 2022] and CMT [Yan et al., 2023] in nuScenesC show relatively higher robustness than in KITTI-C when faced with weather noise. However, BEVFusion performs well in the presence of snow, rain, and strong sunlight noise but experiences a significant performance drop in foggy scenarios. In contrast, our method exhibits strong robustness and generalization in both weather and sensor noise scenarios in nuScenes-C.

5 https://arxiv.org/abs/2401.03907

4.4 Ablation Study

Performance of Different VFMs on RoboFusion

In order to analyze the noise robustness and FPS performance of different-sized VFMs, SAM, FastSAM and MobileSAM, we conduct comparative experiments of RoboFusionL, RoboFusion-B and RoboFusion-T with SOTA methods, DeepInteraction [Yang et al., 2022] and TransFusion [Bai et al., 2022], on the nuScenes-C [Dong et al., 2023] validation set, as shown in Table 5. Specifically, our RoboFusion exhibits remarkable robustness to weather noise scenarios. Furthermore, our RoboFusion-T has a similar FPS to TransFusion [Bai et al., 2022]. Overall, we have presented a viable application of SAM in 3D object detection tasks.

Impacts of Different SAM Usages

As shown in Table 6, our RoboFusion-L is experimented upon. Specifically, the first row is the offline usage, which involves loading pre-saved image features during training. It implies that certain online data augmentation cannot be utilized. The second (No optim) and the third (Optim) rows are online usages, where the former omits fine-tuning and keeps the model parameters fixed, the latter follows fine-tuning and updating. Therefore, offline usage perform worse than online usages. Additionally, fine-tuning the weights of SAM has demonstrated superior performance, resulting in a performance improvement in the presence of snow, rain, and fog noise scenarios.

Influence of Pre-training on SAM

As shown in Table 7, to investigate the scientific value of pretrained VFMs like SAM, FastSAM, and MobileSAM in AD scenarios, we conduct our RoboFusion-L with SAM evaluation on SAM and SAM-AD. Through pre-training, SAM-AD has gained a better understanding of AD scenarios than the original SAM. The pre-training strategy effectively improves the performance of our RoboFusion, demonstrating a significant improvement in the snow, rain, and fog noise scenarios.

Roles of Different Modules in RoboFusion

As shown in Table 8, we present ablation experiments for different modules of our RoboFusion-L, built upon SAM-AD, including AD-FPN, DGWA, and Adaptive Fusion. Leveraging the strong capabilities of SAM-AD in AD scenarios, SAM-AD has a significant improvement from baseline Focals Conv [Chen et al., 2022] (34.77%, 41.30%, 44.55%, 80.97%) to (80.68%, 81.68%, 81.67%, 83.48%). Subsequently, ADFPN, DGWA, and Adaptive Fusion achieve even higher performance on the foundation of SAM-AD. This further highlights the substantial contributions of diverse modules within our RoboFusion framework in addressing OOD noise scenarios in AD.

5 Conclusions

In this work, we propose a robust framework RoboFusion to enhance the robustness and generalization of multi-modal 3D object detectors using VFMs like SAM, FastSAM, and MobileSAM. Specifically, we pre-train SAM for AD scenarios, yielding SAM-AD. To align SAM or SAM-AD with multimodal 3D object detectors, we introduce AD-FPN for feature upsampling. To further mitigate noise and weather interference, we apply wavelet decomposition for depth-guided image denoising. Subsequently, we utilize self-attention mechanisms to adaptively reweight fused features, enhancing informative attributes and suppressing excess noises. Extensive experiments demonstrate that our RoboFusion effectively integrates VFMs to boost feature robustness and address OOD noise challenges. We anticipate this work to lay a strong foundation for future research on building robust and dependable foundation AD models.

Limitation and Future Work

First, RoboFusion has a heavy reliance on the representation capability of VFMs. This raises the baseline models' generalization ability, but increases their complexities. Second, the inference speed of RoboFusion-L and RoboFusion-B is relatively slow due to the limitations of SAM and FastSAM. However, the inference speed of RoboFusion-T is competitive with some SOTA methods (e.g. TransFusion) without VFMs. In the future, for improving the real-time application ability of VFMs, we will attempt to incorporate SAM only in the training phase to guide a fast-speed student model, meanwhile explore more noise scenarios.

Acknowledgments

This work was supported by the Fundamental Research Funds for the Central Universities (2023YJS019), the National Key R&D Program of China (2018AAA0100302).