Do You Remember the Future? Weak-to-Strong Generalization in 3D Object Detection

1 Introduction

In the rapidly advancing fields of computer vision and autonomous driving, 3D object detection is crucial for safe vehicle navigation and interaction with the environment. LiDAR, with its detailed 3D environmental data capture, stands out among sensing technologies. However, LiDAR data faces challenges like sparsity and occlusion, affecting 3D detection efficiency. Sparsity results from LiDAR's point cloud data, which, despite its detail, often misses the continuous coverage seen in camera images or self-supervised learning based reconstructed dense depth maps [Karpov and Makarov, 2022; Indyk and Makarov, 2023; Luginov and Makarov, 2023]. Occlusions further complicate detection, as objects can be hidden by obstacles.

Our X-Ray Teacher framework [Gambashidze et al., 2024] offers a novel solution by utilizing the temporal dimension of LiDAR data to construct Object-Complete frames from multiple viewpoints and use it for weak-to-strong knowledge distillation, presented on Figure 1. This approach effectively mitigates sparsity and occlusion, enabling the distillation of comprehensive object knowledge to our detection system.

Central to our innovation is the Teacher-Student framework, where the Teacher model, trained on Object-Complete frames, imparts its enhanced environmental understanding to the Student model. This knowledge transfer significantly improves detection performance, especially since real-time generation of Object-Complete frames is impractical due to the need for future viewpoints.

Validated across leading autonomous driving datasets, our method integrates seamlessly with any model, consistently boosting performance of supervised models. In this demonstration, we highlight the Teacher model's precise predictions and the X-Ray Student's ability to surpass original base models, setting new accuracy and reliability standards in 3D object detection.

2 RelatedWorks

In the realm of 3D object detection, the field has evolved from directly processing point clouds with foundational models like PointNet and PointNet++ [Qi et al., 2017a; Qi et al., 2017b], to adopting voxel-based representations for efficiency, leveraging 3D sparse convolutions [Yang et al., 2018; Yin et al., 2021; Zhou et al., 2022; Xu et al., 2022], and incorporating advanced techniques like modified self-attention layers for enhanced performance [Wang et al., 2023].

The concept of Knowledge Distillation (KD), introduced by Hinton et al. [Geoffrey Hinton, 2015], employs a teacherstudent model to transfer knowledge, typically focusing on logits/regression distillation and feature map distillation. Unlike conventional KD applications in 3D detection that prioritize efficiency [Cho et al., 2023; Yang et al., 2022], our X-Ray Teacher model adopts a hybrid distillation approach to enhance detection accuracy, setting a new precedent for leveraging KD to surpass state-of-the-art performance.

3 Methodology

Our framework, designed for easy integration with any object detection model, leverages sequential LiDAR data for innovative enhancements. We hypothesized that training on object-complete data-aggregating points from all sequence frames-would significantly improve predictions and supervised distillation. Our X-Ray Teacher, a weaker network, effectively guides a more robust student model to higher accuracy. Validation on the NuScenes dataset showed the objectcomplete model achieving 79.5% mAP (trained and validated on object-complete frames), while baseline model has 59.2% mAP (trained and validated on original clouds).

3.1 Object Complete Frames

To make point clouds much simpler and address the occlusion and sparsity problems, we need to aggregate all possible information about each object in each point cloud from all frames in a sequence. In the supervised setting the task is quite clear: all objects have their own instance IDs and well annotated boxes, so we only need to iterate over all frames and find all appearances of an object that we currently have processed.

3.2 X-Ray Teacher

It is evident that a model trained on original point clouds will not perform better on object-complete frames. In the case of the NuScenes dataset, we validated our original CenterPoint model on an object-complete validation set and achieved only a 31.6% mAP score, which confirms the necessity of training X-Ray Teachers either from scratch or by fine-tuning them from an original pretrained checkpoint. This underscores the critical importance of the X-Ray Teacher stage where we train teacher models from scratch. This step transforms the Teacher model into a weaker one that guides the stronger (student) model to make even better predictions.

3.3 Knowledge Distillation in Supervised Setting

Having prepared object-complete point clouds, we proceeded to train the baseline model (Student) to minimize Knowledge Distillation losses, aligning it with the X-Ray Teacher's insights on complete data. Distillation involves matching the Teacher's and Student's outputs across several dimensions: backbone encoder embeddings, bounding box regression labels, class distributions for classification tasks, and intermediate features from regression and classification heads before label assignment. The pipeline is shown on Figure 2.

The distillation losses are defined as follows:

Here, T and S represent the Teacher and Student model outputs, respectively, with S processing the original frame F and T handling the Object-Complete Frame ˜ F. The terms Sback and Tback denote the backbone outputs, while Sreg, Treg, and Scls, Tcls correspond to the regression and classification outputs. ˜ Tboxes are the Teacher's predicted boxes, and Spreds is the Student's overall output. The parameters α1, α2 are weights for the loss components, and ϕ and ω adjust feature map dimensions and flexibility, respectively. This approach ensures the Student model learns to extract and utilize rich information from less detailed data, mirroring the Teacher's advanced detection capabilities.

4 Experiments

4.1 Implementation Details

In this demonstration we show results on Waymo Open Dataset [Sun et al., 2020] and NuScenes [Caesar et al., 2020].

For the Waymo dataset, we have refined model architectures due to its longer sequences, which resulted in highly dense complete objects post our object completion procedure, see example in Figure 4. Given that the data for the X-Ray Teacher is significantly simpler, we increased the complexity of the student network to better match the teacher's feature maps distribution. The same principle does not hold for NuScenes as it has less informative object-complete clouds. We used default configuration files for all runs.

4.2 Results

In this demo, we introduce our novel plug-and-play framework for LiDAR-based 3D object detection. Our method demonstrates extreme robustness and improves all models we have tested so far, including the previous state-of-the-art, DSVT [Wang et al., 2023]. We validated our method on the two most popular datasets: Waymo and NuScenes, see results in Table 1 and Table 2, respectively.

We also show visual side-by-side comparison of the framework performance on NuScenes at Figure 3.

5 Conclusion

In this demonstration, we highlight the capability of combining object points from different frames to create complete object frames. This approach seamlessly integrates knowledge distillation into any 3D object detection pipeline, functioning as a plug-and-play solution. Our results reveal that a Teacher model, trained and validated on object-complete frames delivers highly accurate predictions. The Teacher model's behavior enables the student model to focus on occluded objects and gain a better understanding of sparsely represented ones.

Ethical Statement

Our research on LiDAR-based 3D object detection presents an improvement in detection metrics by 2-3%. While significant within our domain, this advancement requires extensive hyperparameter tuning, leading to increased computational demands and potential environmental impacts.

We utilized open datasets from Waymo and NuScenes, with all privacy concerns managed by the dataset providers. This approach ensures our adherence to data privacy standards and contributes to the reproducibility of our work.

A notable consideration is the impact of autonomous driving technologies on employment within the transportation sector. As these technologies advance, particularly through enhancements like ours, the demand for human drivers may decrease, posing socioeconomic challenges.

In summary, our work seeks to push forward the capabilities of autonomous driving technologies while acknowledging the ethical considerations of increased computation and the potential societal impacts on employment. We emphasize the importance of responsible technological advancement and the need for a balanced approach to innovation.

Acknowledgements

This research was supported in part through computational resources of HPC facilities at HSE University.

The work of I. Makarov on 3D object detection related work was supported by a grant for research centers in the field of artificial intelligence, provided by the Analytical Center in accordance with the subsidy agreement (agreement identifier 000000D730321P5Q0002) and the agreement with the Ivannikov Institute for System Programming of dated November 2, 2021 No. 70-2021-00142.