DAMO-StreamNet: Optimizing Streaming Perception in Autonomous Driving

1 Introduction

The rapid development of autonomous vehicles necessitates robust and efficient traffic environment perception systems. Crucial to this is streaming perception, which concurrently detects and tracks objects in a video stream, and directly influences autonomous driving decisions. Challenges, however, arise from the swiftly fluctuating scales of traffic objects due to vehicle motion, leading to conflicts in the receptive field when identifying both large and small objects. Furthermore, real-time perception is a complex issue largely reliant on motion consistency context and historical data. The two primary hurdles in real-time perception are: (1) the adaptive management of quickly shifting object scales, and (2) the accurate and efficient learning of long-termmotion consistency.

Despite previous research on temporal aggregation techniques [Wang et al., 2018; Chen et al., 2018; Lin et al., 2020; Sun et al., 2021; Huang et al., 2022] has primarily focused on offline settings and is unsuitable for online real-time perception. Furthermore, enhancing the base detector has not been thoroughly investigated in the context of real-time perception. To address these limitations, we propose DAMOStreamNet, a practical real-time perception pipeline that improves the model in four key aspects:

1. To augment the performance of the base detector, we introduce an effective feature aggregation scheme named Dynamic Receptive Field FPN. Leveraging connections and deformable convolution networks, this scheme mitigates receptive field conflicts and strengthens feature alignment capacity. We also implement a state-of-theart detection technique known as Re-parameterization to boost the network's performance without adding extra inference costs. These improvements result in superior detection accuracy and quicker inference times.
2. To capture long-term spatial-temporal correlations, we construct a dual-path structure temporal fusion module. Utilizing a two-stream architecture, this module separates spatial and temporal information, enabling the precise and efficient capture of long-term correlations.

1. To address the complexities of learning long-term motion consistency, we devise an Asymmetric Knowledge Distillation (AK-Distillation) framework. This framework applies a teacher-student learning strategy, wherein student networks are supervised by transferring the generalized knowledge captured by large-scale teacher networks. This method enforces long-term motion consistency of the feature representations between the teacher-student pair, leading to improved performance.
2. To meet the demand for real-time forecasting, we update the support frame features with the current frame before the subsequent prediction in the inference phase. Furthermore, the support frame features are updated by the current frame in preparation for the next prediction in the inference phase to fulfill the real-time forecasting requirement. This process enables the pipeline to handle real-time streaming perception and make timely predictions.

In a nutshell, DAMO-StreamNet presents a cutting-edge solution for real-time perception in autonomous driving. We also introduce a novel evaluation metric, the K-Step Streaming Metric, which takes into account the temporal interval to assess real-time perception. Our experiments demonstrate that DAMO-StreamNet surpasses existing SOTA methods, achieving 37.8% (normal size (600, 960)) and 43.3% (large size (1200, 1920)) sAP without utilizing any extra data. Our work not only sets a new standard for real-time perception but also contributes meaningful insights for future research in this field. Furthermore, DAMO-StreamNet can be adapted to a variety of autonomous systems, such as drones and robots, to provide accurate and real-time environmental perception, thereby enhancing their safety and efficiency.

2 RelatedWork

2.1 Image Object Detection

State-of-the-art Detectors. The field of image object detection has seen significant progress in recent years due to the development of advanced detectors [Ge et al., 2021b; Wang et al., 2022], with techniques focusing on backbone design [Wang et al., 2021; Ding et al., 2021a; Ding et al., 2021b; Ding et al., 2019; Vasu et al., 2022], feature aggregation [Lin et al., 2017; Ghiasi et al., 2019; Jiang et al., 2022; Tan et al., 2020; Cheng et al., 2022; Tu et al., 2023; Cheng et al., 2017a; Cheng et al., 2019b], and label assignment [Ge et al., 2021a; Kim and Lee, 2020; Carion et al., 2020].

Feature Aggregation. Feature aggregation, especially with FPN [Lin et al., 2017] and PAFPN [Liu et al., 2018], plays a key role in object detection. More recently, the Neural Architecture Search (NAS) methodology has been incorporated into this area [Ghiasi et al., 2019; Cheng et al., 2018; Huang et al., 2018]. GiraffeDet [Jiang et al., 2022; Chen et al., 2023; Zhou et al., 2022] further innovates by using a lightweight backbone and a heavy neck for feature learning.

2.2 Video Object Detection

Temporal Learning. Temporal learning often involves feature aggregation across nearby frames [Wang et al., 2018; Chen et al., 2018; Lin et al., 2020; Sun et al., 2021; Lan et al., 2022; Cheng et al., 2017a]. This has been implemented in DeepFlow [Zhu et al., 2017b] and FGFA [Zhu et al., 2017a] through optic flow, and in MANet [Wang et al., 2018] through pixel-level calibration.

Temporal Linking. Despite the success of temporal learning, video object detection often requires complex temporal modeling components, such as optical flow models [Zhu et al., 2017b], recurrent neural networks [Lin et al., 2020; He et al., 2021], and relation networks [Gao et al., 2021; Cheng et al., 2017b]. Simpler alternatives include temporal linking modules like Seq-NMS [Han et al., 2016], Tubelet rescoring [Kang et al., 2016], and Seq-Bbox Matching [Belhassen et al., 2019; Lan et al., 2022].

2.3 Knowledge Distillation

Knowledge distillation [Hinton et al., 2015] aims to transfer feature representation from a teacher network to a student network. This approach has been adapted in various ways, such as with intermediate-sized teacher-assistant networks [Mirzadeh et al., 2020] and hint learning [Chen et al., 2017]. Other efforts have focused on leveraging different intermediate representations [Heo et al., 2019; Chen et al., 2021; Cheng et al., 2018; Cheng et al., 2019b] or learning data sample or layer relations [Yao et al., 2021; Liu et al., 2020; Yang et al., 2022b; Cheng et al., 2019a]. DAMO-StreamNet is the first work to use knowledge distillation for the streaming perception task, employing the knowledge distillation module to enhance the accuracy of "predicting the next frame" [Yang et al., 2022a] by mirroring the features of the "next frame."

2.4 Streaming Perception

Streaming perception is a relatively new field with limited research focus. Existing methods [Li et al., 2020] are based on object detection and use temporal modeling techniques to improve performance. However, state-of-the-art work such as StreamYOLO [Yang et al., 2022a] does not fully utilize the semantics and motion in video streams. Meanwhile, other recent efforts [Yang et al., 2022a; Li et al., 2022] build on the YOLOX-based detector. Our work with DAMO-StreamNet addresses these issues by re-engineering the base detector and integrating feature aggregation and knowledge distillation. As a result, our method presents a more comprehensive solution for the streaming perception task, outperforming existing methods and setting a new standard for future research.

3 DAMO-StreamNet

The overall framework is illustrated in Fig. 3. Initially, a video frame sequence passes through DAMO-StreamNet to extract spatiotemporal features and generate the final output feature. Subsequently, the Asymmetric Knowledge Distillation module (AK-Distillation) takes the output logit features of the teacher and student networks as inputs, transferring the semantics and spatial position of the future frame extracted by the teacher to the student network.

Given a video frame sequence S = {It, . . . It-Nδt}, where N and δt represent the number and step size of the frame sequence, respectively. DAMO-StreamNet can be defined as,

where W denotes the network weights, and T represents the collection of final output feature maps. T can be further decoded using Decode(T ) to obtain the result R, which includes the score, category, and location of the objects.

In the training phase, the student network can be represented as,

Besides the student network, the teacher network takes the t + 1 frame as input to generate the future result, represented by,

where Wstu and Wtea denote the weights of the student and teacher networks, respectively. Then, AK-Distillation leverages Tstu and Ttea as inputs to perform knowledge distillation AKDM(Tstu, Ttea). More details are elaborated in the following subsections.

3.1 Network Architecture

The network is composed of three elements: the backbone, neck, and head. It can be formulated as,

where Gb, Gn, and Gh stand for the backbone, neck, and head components respectively, while Wb, Wn, and Wh symbolize their corresponding weights. Previous studies [Jiang et al., 2022] highlighted the neck structure's critical role in feature fusion and representation learning for detection tasks. Consequently, we introduce the Dynamic Receptive Field FPN (DRFPN), which employs a learnable receptive field approach for enhanced feature fusion. To benchmark against the current state-of-the-art (SOTA), we apply the same settings for Gn, Gh, and StreamYOLO [Yang et al., 2022a], leveraging CSPDarknet-53 [Ge et al., 2021b] and TALHead [Yang et al., 2022a] to build the network. Given the proven efficacy of long-term temporal information by the existing LongShortNet [Li et al., 2022], we also integrate a dual-path architectural module for spatial-temporal feature extraction.

Dynamic Receptive Field FPN. Recent object detection studies, including StreamYOLO [Yang et al., 2022a] and LongShortNet [Li et al., 2022], have utilized YOLOX as their fundamental detector. YOLOX's limitation is its fixed spatial receptive field that cannot synchronize features temporally, thus impacting its performance. To address this, we propose the Dynamic Receptive Field FPN (DRFPN) with a learnable receptive field strategy and an optimized fusion mechanism.

Specifically, Fig.4 contrasts PAFPN and DRFPN. PAFPN employs sequential top-down and bottom-up fusion operations to amplify feature representation. However, conventional convolution with a static kernel size fails to align features effectively. As a solution, we amalgamate the DRM module and Bottom-up Auxiliary Connect (BuAC) with PAFPN to create DRFPN. We introduce three notable modifications compared to PAFPN's CSP module (Fig.4):(1) We integrate deformable convolution layers into the DRFPN module to provide the network with learnable receptive fields;(2) To enhance feature representation, we adopt re-parameterized convolutional layers [Ding et al., 2021b];(3) ELAN [Wang et al., 2022] and Bottom-up Auxiliary Connect bridge the semantic gap between low and high-level features, ensuring effective detection of objects at diverse scales.

Dual-Path Architecture. The existing StreamYOLO [Yang et al., 2022a] relies on a single historical frame in conjunction with the current frame to learn short-term motion consistency. While this suffices for ideal uniform linear motion, it falls short in handling complex motion, such as non-uniform motion (e.g., accelerating vehicles), non-linear motion (e.g., rotation of objects or camera), and scene occlusions (e.g., billboard or oncoming car occlusion).

To remedy this, we integrate the dual-path architecture [Li et al., 2022] with a reimagined base detector, enabling the capture of long-term temporal motion while calibrating it with short-term spatial semantics. The original backbone and neck can be represented formally as,

where Gfuse represents the LSFM-Lf-Dil of LongShortNet. G short n+b and G long n+b denote the ShortPath and LongPath of LongShortNet, which are used for feature extraction of the current and historical feature, respectively. Note that their weights are shared.

Finally, the dual-path network is formulated as,

where the proposed dual-path architecture effectively addresses complex motion scenarios and offers a sophisticated solution for object detection in video sequences.

3.2 Asymmetric Knowledge Distillation

The ability to retain long-term spatiotemporal knowledge through fused features lends strength to forecasting, yet achieving streaming perception remains a daunting task. Drawing inspiration from knowledge distillation, we've fashioned an asymmetric distillation strategy, transferring "future knowledge" to the present frame. This assists the model in honing its accuracy in streaming perception without the burden of additional inference costs.

Given the asymmetric input nature of the teacher and student networks, a sizable gap emerges in their feature distributions, thus impairing the effectiveness of distillation at the feature level. Logits-based distillation primarily garners performance improvements by harmonizing the teacher model's response-based knowledge, which aligns knowledge distribution at the semantic level. This simplifies the optimization process for asymmetric distillation. As a result, we've engineered a distillation module to convey rich semantic and localization knowledge from the teacher (the future) to the student (the present).

The asymmetric distillation is depicted in Fig. 3. The teacher model is a still image detector that takes It+1 as input and produces logits for It+1. The student model is a standard streaming perception pipeline that uses historical frames It-1, . . . , It-N and the current frame It as input to forecast the results of the arriving frame It+1. The logits produced by the teacher and student are represented by Tstu = {F cls stu , F reg stu , F obj stu }, and Ttea = {F cls tea , F reg tea , F obj tea }, where F cls · , F reg · , and F obj · correspond to the classification, objectness, and regression logits features, respectively. The Asymmetric Knowledge Distillation, AKDM(·), is mathematically formulated as,

where Lcls(·) and Lobj(·) are Mean Square Error (MSE) loss functions, and Lreg(·) is the GIoU loss [Rezatofighi et al., 2019]. ˆ F reg stu and ˆ F reg tea represent the positive samples of the regression logit features, filtered using the OTA assignment method as in YOLOX [Ge et al., 2021b]. It is worth noting that location knowledge distillation is only performed on positive samples to avoid noise from negative ones.

3.3 K-step StreamingMetric

The Streaming Average Precision (sAP) metric is a prevalent tool used to gauge the precision of Streaming Perception systems [Li et al., 2020]. This metric gauges precision by juxtaposing real-world ground truth with system-generated results, factoring in process latency.

Two primary methodologies exist in this domain: non-realtime and real-time. For non-real-time methods, as depicted in Fig.5(a), the sAP metric calculates precision by comparing the current frame It results with the ground truth of the following frame It+2, post processing of frame It. Conversely, real-time methods, as demonstrated in Fig. 5(b), conclude the processing of the current frame It prior to the next frame It+1 arrival. Our proposed method, DAMO-StreamNet, is a realtime method, adhering to the pipeline outlined in Fig. 5(b).

Though the sAP metric effectively evaluates the short-term forecasting capability of algorithms, it falls short in assessing their long-term forecasting prowess- a critical factor in real-world autonomous driving scenarios. In response, we introduce the K-step Streaming metric, an expansion of the sAP metric, specifically tailored to evaluate long-term performance. As depicted in Fig. 5(c), the algorithm projects the results of the upcoming two frames, and the cycle continues. The projection of the next K frames is represented as "KsAP", as shown in Fig. 5(d). Consequently, the standard sAP metric translates to 1-sAP in the K-step metric context.

4 Experiments

4.1 Dataset andMetric

Dataset. We utilized the Argoverse-HD dataset, which comprises various urban outdoor scenes from two US cities. The dataset contains detection annotations and center RGB camera images, which were used in our experiments. We adhered to the train/validation split proposed by Li et al. [Li et al., 2020], with the validation set consisting of 15k frames.

Evaluation Metrics. We employed the streaming Average Precision (sAP) metric to evaluate performance. The sAP metric calculates the average mAP over Intersection over Union (IoU) thresholds ranging from 0.5 to 0.95, as well as APs, APm, and APl for small, medium, and large objects, respectively. This metric has been widely used in object detection, including in previous works such as [Li et al., 2020; Yang et al., 2022a].

4.2 Implementation Details

We pretrained the base detector of our DAMO-StreamNet on the COCO dataset [Lin et al., 2014], following the methodology of StreamYOLO [Yang et al., 2022a]. We then trained DAMO-StreamNet on the Argoverse-HD dataset for 8 epochs with a batch size of 32, using 4 V100 GPUs. For convenient comparison with recent state-of-the-art models [Yang et al., 2022a; Li et al., 2022], we designed small, medium, and large networks (i.e., DAMO-StreamNet-S, DAMO-StreamNet-M, and DAMO-StreamNet-L). The normal input resolution (600, 960) was utilized unless specified otherwise. We maintained consistency with other hyperparameters from previous works [Yang et al., 2022a; Li et al., 2022]. AK-Distillation is an auxiliary loss for DAMO-StreamNet training, with the weight of the loss set to 0.2/0.2/0.1 for DAMO-StreamNetS/M/L, respectively.

4.3 Comparison with State-of-the-art Methods

We compared our proposed approach with state-of-the-art methods to evaluate its performance. In this subsection, we directly copied the reported performance from their original papers as their results. The performance comparison was conducted on the Argoverse-HD dataset [Li et al., 2020]. An overview of the results reveals that our proposed DAMOStreamNet with an input resolution of 600 × 960 achieves 37.8% sAP, outperforming the current state-of-the-art methods by a significant margin. For the large-resolution input of 1200 × 1920, our DAMO-StreamNet attains 43.3% sAP without extra training data, surpassing the state-of-the-art work StreamYOLO, which was trained with large-scale auxiliary datasets. This clearly demonstrates the effectiveness of the systematic improvements in DAMO-StreamNet.

Compared to StreamYOLO and LongShortNet, DAMOStreamNet-L achieves absolute improvements of 3.6% and 2.4% under the sAPL metric, respectively. This also provides substantial evidence that the features produced by DRFPN offer a self-adaptive and sufficient size of the receptive field for large-sized objects. It is worth noting that DAMO-StreamNet experiences a slight decline compared to LongShortNet under the stricter metric sAP75. This observation suggests that although the dynamic receptive field achieves a sufficient receptive field for different scales of objects, it is not as accurate as fixed kernel-size ConvNets. The offset prediction in the deformable convolution layer may not be precise enough for high-precision scenarios. In other words, better performance could be achieved if this issue is addressed, and we leave this for future work.

4.4 Ablation Study

Investigation of DRFPN. To verify the effectiveness of DRFPN, we use StreamYOLO [Yang et al., 2022a] and LongShortNet [Li et al., 2022] as baselines and integrate themwith the proposed DRFPN, respectively. The experimental results are listed in Table 2. It is evident that DRFPN significantly improves the feature aggregation capability of the baselines. Particularly, the small-scale baseline models equipped with DRFPN achieve improvements of 1.9% and 1.7%, separately. This also demonstrates that the dynamic receptive field is crucial for the stream perception task. More importantly, DRFPN enhances the performance of LongShortNet, which suggests that the temporal feature alignment capacity is also augmented by the dynamic receptive field mechanism.

Investigation of Temporal Range. To isolate the influence of temporal range, we conduct an ablation study on N and δt, as listed in Table 3. (0, -) represents the model utilizing only the current frame as input. It is evident that increasing the number of input frames can enhance the model's performance, with the best results obtained when N is equal to 2, 2, and 3 for DAMO-StreamNet-S/M/L, respectively. However, as the number of input frames continues to increase, the performance experiences significant declines. Intuitively, longer temporal information should be more conducive to forecasting, but the effective utilization of long-term temporal information remains a critical challenge worth investigating.

Investigation of AK-Distillation. AK-Distillation is a cost-free approach for enhancing the streaming perception pipeline, and we examine its impact. We perform AKDistillation with various lengths of temporal modeling and scales of DAMO-StreamNet. As the results listed in Table 4 indicate, AK-Distillation yields improvements of 0.2% to 0.9% for the DAMO-StreamNet configured with N = 1 short-term temporal modeling. This demonstrates that AKDistillation can effectively transfer "future knowledge" from the teacher to the student. For the DAMO-StreamNet with the setting of N = 3, AK-Distillation improves DAMOStreamNet-S/L by only 0.3%, but results in a slight decline for the medium-scale model. The limited improvement for long-term DAMO-StreamNet is due to the narrow performance gap between the teacher and student, and the relatively high precision is difficult to further enhance.

Investigation of K-step Streaming Metric. We evaluate DAMO-StreamNet with settings N = 1 and N = 2/3 under the new metric sAPk, where k ranges from 1 to 6. The results are listed in Table 5. It is clear that the performance progressively declines as k increases, which also highlights the challenge of long-term forecasting. Another observation is that the longer time-series information leads to better performance under the new metric.

Inference Efficiency Analysis. Although the proposed DRFPN has a more complex structure compared to PAFPN, DAMO-StreamNet still maintains real-time streaming perception capabilities. For long-term fusion, we adopt the buffer mechanism from StreamYOLO [Yang et al., 2022a], which incurs only minimal additional computational cost for multiframe feature fusion.

5 Conclusion

Our research presents DAMO-StreamNet, a novel and robust framework integrating cutting-edge technologies from the YOLO series. Key innovations include (1) a robust neck structure using deformable convolution, (2) a dual-branch design for enhanced time-series data analysis, (3) logit-level distillation, and (4) a dynamic real-time prediction mechanism. Comparison with existing methods on the ArgoverseHD dataset clearly shows DAMO-StreamNet's superiority.

Acknowledgments

Zhi-Qi Cheng's research in this project was supported by the US Department of Transportation, Office of the Assistant Secretary for Research and Technology, under the University Transportation Center Program (Federal Grant Number 69A3551747111), as well as Intel and IBM Fellowships.

Contribution Statement

Jun-Yan He, Zhi-Qi Cheng, Chenyang Li, and Wangmeng Xiang contributed equally as co-first authors to this work, in random order. They were involved in the study design, experiments, manuscript writing, and discussions. The manuscript underwent review by Binghui Chen, Bin Luo, Yifeng Geng, and Xuansong Xie. Zhi-Qi Cheng, as the corresponding author, managed the entire project. Parts of the work were completed through Jun-Yan He's remote collaboration with Carnegie Mellon University.