Boost Embodied AI Models with Robust Compression Boundary

1 Introduction

As of 2025, the field of embodied AI is witnessing significant advancements, driven by innovations in technology and a growing understanding of how to integrate AI with physical systems. For example, various classification [Yang , 2023b], object detection [Jiang et al., 2023], and semantic segmentation [Liu et al., 2023b] models help autonomous vehicles perceive the surrounding environments better. Visual SLAM and point cloud models further improve autonomous vehicles' positioning and navigation capabilities in a dynamic environment.

Another good example is introducing the vision and language foundation models [Touvron et al., 2023] into the control strategies learning for robotic manipulation. The enormous robot manipulation datasets [Khazatsky et al., 2024] only have 100K to 1M examples, while the foundation models are trained with Internet-scale pretraining datasets [Laurenc¸on et al., 2022]. By fully utilizing the generalization capabilities of the foundation models, the robots may be able to extrapolate behaviors to different surrounding conditions (e.g., manipulation positions [Brohan et al., 2022], lighting [Chi et al., 2023], scene distractors [Xie et al., 2024]) and unseen situations (e.g., novel objects [Mees et al., 2024], new task instruction [Walke et al., 2023]). Following this paradigm, a series of the vision-language- action (VLA) models [Kim et al., 2024] are explored to enhance the control and manipulation of robotics.

∗Tao Chen and Zhongxue Gan are corresponding authors.

'Rain'

'Snow'

The rapid development of large-scale neural models has broadly and profoundly influenced the capabilities of embodied AI. Meanwhile, large-scale neural models have drawbacks [Yu, 2021], such as high computational cost and energy consumption, limitations in response-sensitive scenarios, and high memory requirements in the edge platforms of embodied AI. Considering the application characters and actual costs, the computational core devices on embodied AI platforms tend to use embedded systems with low power consumption, limited computational power, and memory [Tinchev et al., 2019] [Wan et al., 2021]. So, model compression [Abbasi et al., 2022] [Li et al., 2022] is essential to alleviate the gaps between the increasing scale and considerable resource consumption of the models and their efficient deployment on the limited-capacity embodied AI hardware platforms.

Considering the safety and robustness concerns in the embodied AI application, e.g., autonomous driving area, when we want to compress an autonomous driving model into compressed form, we need to check further whether this compressed autonomous driving model is a qualified substitute facing corruptions. In traffic, the autonomous driving models are expected to resist natural corruption caused by real-world events shown in Figure 1 such as blur, darkness, weather conditions, and even the adversarial corruption to further guarantee the safety of the compressed embodied AI models.

In this paper, we first review the prior arts of model compression and their potential flaws when applied to embodied AI models, which require good accuracy, efficient deployment, safety, and robustness for corruption. Based on the analysis of past efforts, we explore and propose a novel paradigm for embodied AI models' compression. We push the robust compression boundary beyond the upper limitations that come from the original dense model. Finally, we evaluate our method on typical embodied AI models and benchmarking tasks to show the efficacy of our method for finding the optimal balance between accuracy, efficiency, and robustness in real-world conditions.

2 RelatedWork

2.1 Model Compression Against Corruption

When we try to use a compressed embodied AI model as the substitute for an original dense model, the typical strategy is applying the compression technique to ensure this obtained compressed model has a similar model accuracy and detection results with the original dense model on benign input images, shown in the upper part of the Figure 2. In this model compression process, only the weight parameters of the target compressed embodied AI model will be updated. The weight parameters of the original dense model are used in initialization or some knowledge-distillation-based compression techniques [Li et al., 2023] [Yu et al., 2023b] as the golden reference, so they are frozen.

After verifying the compressed models' quality on the benign dataset, the extra step is needed to evaluate whether the compressed is a qualified substitute when facing corruption, shown in the lower part of the Figure 2. Prior arts [Hnewa and Radha, 2020] will collect the real corrupted images or generate the simulated corrupted images to cover natural [Chen et al., 2018] and adversarial corruption [Shen et al., 2021] cases. Then, these corrupted input images are used to evaluate the compressed models' accuracy and detection results. In this case, the weight parameters of the target compressed embodied AI model will also be frozen. Some prior arts [Shen et al., 2021] [Diffenderfer et al., 2021] try to improve the robustness capabilities of the compressed embodied AI model, so they divide the subset of the collected corrupted images and unfrozen the weight parameters of the target compressed embodied AI model for against-corruption finetuning.

Some prior works further explore the relationship between model compression and robustness against corruption [Gui et al., 2019] [Wang et al., 2023]. Based on their exploration, the previous workflow shown in Figure 2 disentangles the model compression process with the against-corruption finetuning process while simplifying the complexity of the workflow but introducing flaws. The apparent flaw is that further finetuning on the corrupted samples may lead to the accuracy regression on the benign samples. Compensation efforts [Yan et al., 2018] [Wang et al., 2019] by recurrent training on individual benign and corrupted datasets or joint training on mixed benign and corrupted samples are proposed, but also raising concerns about more training cost and non-trivial efforts in balancing the accuracy and robustness. The more efficient workflow [Yu et al., 2023a] introduces contrast learning into the robust model compression process, as shown in the Figure 3. During the normal model compression process, the workflow will check the detection results of the dense embodied AI model and the compressed counterpart on the corrupted images. If they have different behaviors, we need to tune the compressed model and finally guarantee they have similar behaviors on the corrupted samples, as well as on the benign samples.

2.2 Typical Embodied AI Models and Applications

Embodied AI represents a significant advancement in artificial intelligence, focusing on systems integrating cognitive abilities with physical actions to interact with and learn from their environments. Embodied AI has revolutionized autonomous vehicles by enabling them to navigate complex environments with human-like decision-making capabilities. Autonomous vehicles can adapt to changing road conditions and obstacles using advanced multi-modal perception systems. Robots equipped with embodied AI are used in manufacturing for assembly and material transport tasks. They can learn from their environment and adapt to changes in real time, improving efficiency. The system called Robotic Avatars [Luo et al., 2022] allow users to operate robots remotely, capturing environmental data for telepresence applications. This technology has potential uses in telemedicine and hazardous environment operations.

With comparable and even superior effectiveness than the traditional convolution neural models, more transformerbased models [Liu et al., 2023b] [Yang et al., 2023a] [Liu et al., 2023a] are explored and widely adopted in embodied AI typical applications with state-of-the-art performance. However, large-scale transformer-based models are computationintensive and memory-intensive [Yu et al., 2023b], placing an increasing burden on deployment efficiency.

3 Boost Robust Compression Boundary

By analyzing the prior model compression paradigms with a against-corruption setting, we find there are two main shortcomings in prior approaches, include:

• The model compression techniques do not fully dig into and utilize the difference between the benign and corrupted training samples.
• The compressed model cannot exceed the robustness boundary of the dense model because both the model

compression and contrast finetuning processes apply the dense model as the golden reference.

We intend to improve these two shortcomings and propose a new algorithm to compress the embodied AI models to push their robustness boundary and efficiency. The new algorithm is called Boost Robust Compression Boundary (BRCB).

3.1 Against-CorruptionMechanism

Traditional studies regard corrupted samples as domain transformation. So, they usually use the corrupted samples for domain adaptation or mix them with benign ones in adversarial training. However, these prior solutions ignore whether such a difference between the corrupted and benign samples can be integrated and used for improving the model compression process. Inspired by this point, we design the againstcorruption mechanism for BRCB workflow.

When we get the natural or adversarial corrupted samples, we first use them to calibrate the dense embodied AI model. Because the corrupted samples can be regarded as benign samples that concatenate the differences, the calibration, i.e., tiny tuning for the original dense embodied AI model, can usually work well with the corrupted samples. In other words, the differences between the original and the calibrated dense weight tensors should be marginal. So, we can disassemble the calibrated dense weight tensors as the similar weight part and the diff weight part, e.g., rainy diff, snowy diff, dark diff, and adversarial diff in the left side of Figure 4.

Then, the BRCB algorithm will concatenate all the diff weight tensors to generate a concentrated diff weight. All the elements in this concentrated diff weights are corruptionsensitive. With this corruptions-sensitive diff weight, we can transfer it as a corruptions-sensitive mask, using the bool elements to indicate the positions of the corruptions-sensitive elements. Because the corruptions-sensitive mask is in the same shape as the original dense model's weight, we can attach them to filter out a new weight tensor that contains all the corruptions-stable elements.

Because the input sample corruptions will less influence the elements in the corruptions-stable weight tensors, we can apply the combined sparse pruning and quantization techniques as normal for weight compression. And we call it the corruption-stable model compression path in the BRCB algorithm. Usually, the corruptions-stable weight tensors are more friendly to compression without out-of-range outliers. For the corruptions-sensitive diff weight, we will use the collected corrupted samples to execute the corruptionssensitive model compression, with more details in the following section. Finally, the BRCB algorithm combines the compressed weights from corruptions-stable and corruptionssensitive paths to obtain the final compressed embodied AI model. The detailed against-corruption mechanism is shown in the left side of Figure 4.

The key to BRCB's against-corruption mechanism design is a fine-granularity division for the corruptions-stable and corruptions-sensitive elements from the original dense model. For the corruptions-stable elements, which are friendly for model compression, BRCB can compress them into a more compact form. For the corruptions-sensitive elements, BRCB fully utilizes the given corrupted training samples during their compression process.

3.2 Push the Limitation of Robustness Boundary

In prior model compression techniques, the dense model's behavior is used as the golden reference, and the target compressed model is designed to mimic the dense model's inference results and feature maps. Suppose the dense model makes the wrong inference result due to the perturbation caused by changing from benign to adversarial samples. In that case, the compressed model will have a high possibility of making the same mistake. This principle design paradigm limits the compressed model, which cannot exceed the robustness boundary of dense model when facing corruption.

BRCB breaks the robustness boundary in the corruptionsensitive model compression path. Instead of mimicking the dense model as the golden reference, it tries to solve this problem from the principle view. The main reason that the corrupted samples bring misleading behavior to the embodied AI model is that the normal feature maps (i.e., the output activation tensors), when feeding the model with the benign inputs, have the out-of-range or out-of-distribution outliers introduced by the corruption. So, the BRCB algorithm attaches a BRCB gate layer before each compression target layer, like the query, key, value, linear projection, and feed-forward layers in the transformer structure. The primary purpose of the BRCB gate layer is to randomly mask out some elements in the feature maps before feeding them into the compression target layers during the corruption-sensitive model compression tuning process. With such regulation during training, the final obtained compressed model can properly adjust when meeting the outliers in inference with the corrupted samples.

We can find that the BRCB gate introduced in the corruption-sensitive model compression process has no reliance on the original dense model as a golden reference. In contrast, it enhances the robustness of the corruption for the compressed model from the principle. In the following experiments, we can see that the BRCB-compressed models can even exceed the robustness boundary of the dense model.

4 Experiments and Results

For the experiments in this section, we choose PyTorch with version 1.9.0 as the framework to train all baseline and efficient neural models. All of the training experimental results are obtained with A100 GPU clusters. All the accuracy results reported for our proposed method use INT8 as the default data type. All the reference algorithms use the default data type provided in public repositories.

4.1 Compression Efficacy for Autonomous Driving

To evaluate the compression efficacy of BRCB and make the comparison with prior arts on the autonomous driving, BEVFormer & BEVFormer v2 [Yang et al., 2023a] 1 and BEVFusion [Liu et al., 2023b] 2 are chosen as the experiment target models for 3D object detection task. BEVFusion is chosen as the experiment target model for bird's-eye view (BEV) map segmentation task. For the state-of-the-art transformerbased compression methods, we choose the MiniViT [Zhang et al., 2022] and UVC [Yu et al., 2022] as the reference sparse pruning techniques, we choose the FQ-ViT [Lin et al., 2022] and PTQ-ViT [Liu et al., 2021b] as the reference quantization techniques. We also choose the GPUSQ-ViT [Yu et al.,

1 https://github.com/fundamentalvision/BEVFormer

2 https://github.com/mit-han-lab/bevfusion

2023b] as the reference mixed-compression technique. The comparison results for 3D object detection task and BEV map segmentation task are shown in Tables 1 and 2.

The comparison results shown in Tables 1 and 2 show that BRCB can steadily provide a smaller accuracy drop for the compressed models on benign samples than both sparse pruning and quantization prior arts. And the accuracy drop compared with the original dense baseline model is negligible in most cases. There is an apparent accuracy drop when natural corruption like rain, snow, and darkness exists for the original dense models. Not to mention, the compressed models were obtained using the prior model compression techniques. However, the BRCB compressed counterparts have an apparent lower accuracy drop than other compressed models, even against the original dense model. This phenomenon proves the effectiveness of the against-corruption mechanism designed in the BRCB method. Moreover, it confirms that the compressed model can break the robustness boundary of the dense model.

In Table 3, we also evaluate the effectiveness of the BRCB compression technique and compare it with the prior arts when facing adversarial corruption. Because adversarial corruption is manually intended to perturb the judgment of the embodied AI models, we can find the accuracy drops are more evident than facing natural corruption. The comparison results shown in Table 3 show that the BRCB compressed models can still steadily provide a smaller accuracy drop for the compressed models than prior arts, even against the original dense model. This experiment further proves the effectiveness of the against-corruption mechanism designed in the BRCB method.

4.2 Compression Efficacy for Robotics

To evaluate the compression efficacy of BRCB on the robotics, OpenVLA [Kim et al., 2024] 3 is chosen as the experiment target model for various tasks. For the state-of-theart transformer-based compression method, we choose the GPUSQ-ViT [Yu et al., 2023b]. The comparison results for Robotics task are shown in Tables 4.

3 https://github.com/openvla/openvla

We evaluate on a comprehensive set of evaluation tasks in each environment that covers various axes of generalization, such as visual (unseen backgrounds, distractor objects, colors/appearances of objects); motion (unseen object positions/orientations); physical (unseen object sizes/shapes); and semantic (unseen target objects, instructions, and concepts from the Internet) generalization. The comparison results shown in Table 4 show that the BRCB compressed models can still steadily provide better accuracy than the prior arts compressed models. And when facing the unseen situations, the generalization capabilities are even against the original dense model. This experiment further proves the effectiveness of the prposed BRCB method.

4.3 Visualization Experiments

For autonomous vehicle, real tasks are multidimensional and more complicated. We also verify BRCB workflow and optimizing the deployment efficiency of UniverseNet [Shinya, 2021] on object detection task, CondLaneNet [Liu et al., 2021a] on lane detection task, Mask2Former [Cheng et al., 2022] on segmentation task, and StereoDNN [Smolyanskiy et al., 2018] on depth estimation task.

For the deployment efficiency, the BRCB-compressed UniverseNet model, the BRCB-compressed CondLaneNet model, the BRCB-compressed Mask2Former model and the BRCB-compressed StereoDNN model can achieve 2.31×, 2.55×, 1.97× and 1.66× acceleration against their corresponding dense counterparts on NVIDIA DRIVE AGX Orin platform. The corresponding visualization results are shown in Figure 5, Figure 6, and Figure 7. We can find all these autonomous driving models keep their effectiveness while the deployment efficiency is obviously improved.

5 Conclusions

In this paper, we have introduced a novel approach to enhance the efficiency and robustness of embodied AI models through the Boost Robust Compression Boundary (BRCB) algorithm. Our findings demonstrate that traditional model compression techniques often fall short when applied to the unique challenges faced by embodied AI systems, particularly in scenarios requiring rapid responses and high reliability.

The BRCB algorithm addresses these shortcomings by employing an innovative against-corruption mechanism that distinguishes between corruptions-stable and corruptionssensitive model components. This fine-grained approach allows for more effective compression while maintaining the integrity of model performance across diverse conditions. Our experimental results indicate that models compressed using BRCB not only achieve comparable accuracy on benign inputs but also exhibit significantly improved resilience against various natural and adversarial corruptions.

Moreover, the BRCB method has shown a remarkable ability to push beyond the robustness boundaries established by dense models, enabling compressed models to perform better

Object Detection

Lane Detection

Urban Street

Downtown Crossroad

Parking Garage

Downtown Street

Tunnel

than their uncompressed counterparts in challenging environments. This advancement is particularly relevant for applications in autonomous driving and robotics, where safety and reliability are paramount. We also evaluate the efficacy of our method on BEV, object detection, lane detection, segmentation, and depth estimation tasks with the real deployment efficiency on the DRIVE AGX Orin autonomous vehicle platform.

Ethical Statement

This research does not involve human or animal subjects, and therefore does not present any ethical issues related to participant consent or welfare. The methodologies employed in this study focus solely on computational models and algorithms, ensuring compliance with ethical standards in artificial intelligence research.

Acknowledgements

This work is supported by Shanghai Natural Science Foundation (No. 23ZR1402900), National Key Research and Development Program of China (No. 2022ZD0160101), Shanghai Science and Technology Commission Explorer Program Project (24TS1401300), Shanghai Municipal Science and Technology Major Project (No.2021SHZDZX0103).

The computations in this research were performed using the CFFF platform of Fudan University.