CF-Deformable DETR: An End-to-End Alignment-Free Model for Weakly Aligned Visible-Infrared Object Detection

1 Introduction

Object detection based on visible sensors has been widely explored and extensively applied in fields like autonomous driving [Li et al., 2022b], video surveillance [Zhang et al., 2022], and robotics [Feng et al., 2022]. Despite the success, it is usually susceptible to a variety of environmental factors like illumination and darkness, resulting in a sharp performance drop [Li et al., 2022a]. Consequently, there is a growing interest in enhancing detection performance by incorporating additional sensors, where infrared sensors demonstrate sufficient complementarity and compatibility with visible sensors,

∗Corresponding author1

†Corresponding author2

• (d) Our motivation

Figure 1: An example to illustrate weakly aligned visible-infrared images (subfigure (a)) with different alignment strategies, where subfigure (b) adopts global pixel-to-pixel alignment, subfigure (c) employs local ROI jittering alignment, and subfigure (d) shows our motivation by using cross-modal alignment-free detection.

thereby promoting the increasing attention on visible-infrared object detection [Cui et al., 2023].

Most existing visible-infrared object detection approaches receive a strictly aligned visible (V) and infrared (IR) image pair and then effectively fuse them for performance boosting [Wang et al., 2022]. Technically, there exist three main fusion strategies including pixel-level fusion [Chen et al., 2022a], feature-level fusion [Yang et al., 2022], and decision-level fusion [Chen et al., 2022b]. No matter which paradigm is adopted, it is assumed that the input image pair is strictly aligned pixel-to-pixel as Figure 1 (a) shows. Unfortunately, in real applications, even visible and infrared sensors are fixed in a carrier and could well cooperate on image capturing, it is inevitable that the capturing V and IR images have slight pixel offsets due to the unstable moving of the carrier. As a result, the general fusion strategies usually fail on weakly aligned visible-infrared data [Zhang et al., 2019].

To tackle this problem, the simple strategy first conducts a global pixel-to-pixel alignment between V and IR images (see Figure 1 (b)) before detection. This pixel-to-pixel alignment inevitably brings unbearable computation costs, thereby significantly affecting the detection speed. Another strategy incorporates the alignment process into a two-stage detector by performing a local alignment, which randomly jitters an ROI region on one modality to well match that on the other modality (see Figure 1 (c)) [Zhang et al., 2019]. Despite improving efficiency, this method still cannot meet the efficiency requirement for real-world applications.

In this article, we explore whether is it possible to skip the alignment for weakly aligned visible-infrared object detection. Inspired by DETR in vision object detection as Figure 1 (d) shows, we discover that an object reference point integrating feature information from several predicted key points (see the green arrows) could offer robust point features for vision object detection. This point-based strategy leverages point features instead of region features for object detection, which offers a tip for us. If we can implement a cross-modal point-based object detection, where a reference point on one modality could automatically search and integrate relative key points from the other modality (see the red arrows), the alignment between V and IR images could be discarded.

Towards this end, this paper first proposes an end-toend alignment-free architecture Cross-modal Fusion Deformable DEtection TRansformer ("CF-Deformable DETR") for weakly aligned visible-infrared object detection. Different from deformable DETR [Zhu et al., 2020] with a single-branch decoder for single-modal object detection, CFDeformable DETR designs a bimodal decoder with two branches, where one branch adopts single-modal key point search as same as Deformable DETR, and the other one executes cross-modal key point prediction. Concretely, given a reference point on one modality, we propose a Cross-modal Deformable Attention Mechanism (CDAM) to automatically search for relative key points on the other modality to well match the reference point. As a result, there are two features (visible and IR) generated from the two branches for each point. To guide the learning of CDAM, we design a Point-level Feature Consistency (PFC) loss, which aims to bridge the cross-modal representation gap by using contrastive learning on paired bimodal features. Differently, the PFC loss is implemented on hyperbolic space instead of Euclidean space, yielding better performance. We conduct extensive experiments on three visible-infrared object detection datasets "KAIST ", "FLIR", and "LLVIP", and the experimental results demonstrate that CF-Deformable DETR achieves promising detection accuracy with high efficiency, and could keep stable performance under varying degrees of weakly aligned data. The main contributions of this paper are summarized as follows:

• We propose an end-to-end architecture Cross-modal Fusion Deformable DEtection TRansformer for weakly aligned visible-infrared object detection. As far as our knowledge, this is the first alignment-free point-based detection network on this topic, providing a promising solution for future research.
• We propose a novel Cross-modal Deformable Attention Mechanism (CDAM) to implement cross-modal point mapping. By building a cross-modal bridging mecha-
• nism, our approach could automatically connect points between visible and IR modalities and thus avoid the cumbersome alignment with high efficiency.
• We design a Point-level Feature Consistency (PFC) loss to bridge the inter-modality representation gap, which could effectively guide the learning of CDAM, generating well-aligned visible and infrared feature pairs for performance boosting.

2 RelatedWork

2.1 Visible-Infrared Object Detection

The high complementarity between visible and IR representations promotes the development of visible-infrared object detection, where numerous researchers are devoted to crossmodal fusion strategies including pixel-level fusion [Cao et al., 2022], feature-level fusion [Kim et al., 2022; Ding et al., 2021; Jain et al., 2023] and decision-level fusion [Chen et al., 2022b]. Among them, feature-level fusion demonstrates superior detection efficiency and effectiveness due to the deep exploration of complementary features. For example, Kinjal [Dasgupta et al., 2022] enhanced the detector performance by fusing the features with contextual information extracted from four RNN networks. Liu [Liu et al., 2021] introduced a deep cross-modalities representation learning-based pedestrian detection network (DCRL-PDN). Besides the aforementioned convolution-based methods, Xing [Xing et al., 2023] extended the Transformer framework to multi-modal detection. By balancing and optimizing visible, infrared, and fusion detection heads, they enhanced the detector's adaptability to varying environments. This advancement brings new possibilities to the realm of visible-infrared object detection. However, it is noted that all of these efforts heavily rely on tightly aligned visible-infrared image pairs, generating huge alignment costs for real-world applications.

2.2 Weakly Aligned Visible-Infrared Object Detection

For weakly aligned visible-infrared object detection, the direct feature fusion usually suffers from a sharp performance drop. To tackle this problem, the simple solution first performs global pixel-to-pixel cross-modal alignment followed by object detection on aligned data, bringing high alignment costs [Deng and Ma, 2022]. To reduce costs, Zhang [Zhang et al., 2019] integrated both alignment and detection into an end-to-end network by jittering regions of interest (ROIs), obtaining aligned ROI features to enhance detection accuracy. On this basis, Zhang [Zhang et al., 2021] further extended this method to tasks involving multiple modalities like visible depth. Zhou [Zhou et al., 2020] designed an alignment module that first predicts the offset for each pixel, and then obtains the final pixel value from neighboring pixels. However, the presence of information deviations has an impact on the performance of the detector. Chen [Chen et al., 2022b] aggregates the detection results from multiple models on different modalities and then selects the final bounding box through post-processing, thereby avoiding the impact of weakly aligned data.

Beyond the previous studies, our work aims at completely abandoning the existing alignment strategies for weakly aligned visible-infrared object detection. Inspired by DETR, we propose an end-to-end alignment-free model CFDeformable DETR to automatically implement cross-modal point mapping learning, supporting robust object detection with high efficiency.

3 Methodology

3.1 Architecture Overview

Figure 2 illustrates the architecture of a Cross-modal Fusion Deformable DEtection TRansformer, consisting of three parts: an encoder for extracting both visible and IR features, a bimodal decoder for generating and fusing point-level visible and IR features, and a prediction head for object prediction. Different from the single-modal Deformable DETR, CF-Deformable DETR supports receiving a weakly aligned visible and IR image pair to predict objects on both modalities. Technically, the proposed bimodal decoder contains a Single-modal Deformable Attention Module (SDAM), a Cross-modal Deformable Attention Module (CDAM), and a Bimodal Adaptive Fusion Module (BAFM). Given a reference point on a visible image, SDAM aims at predicting several relative key points to generate an aggregated visible feature which is the same as Deformable DETR. Differently, CDAM attempts to search for cross-modal key points on the IR image to generate an IR feature. We design a novel Pointlevel Feature Consistency (PFC) loss to guide the learning of CDAM, and the generated bimodal features are then passed to BAFM for effective feature fusion for object detection. Next, we elaborate on CDAM, PFCloss, and PFFM, respectively, followed by the implementation of our model.

3.2 Cross-Modal Deformable AttentionModule

Given a reference point xmq (m ∈ {V, IR}) on the modality m with the point feature ym q ∈ RC and position embedding pmq ∈ RC where C is the channel dimension, crossmodal deformable attention module (CDAM) aims to search for relative key points on the other modality m to aggregate their features around xmq . The traditional single-modal deformable attention module predicts point offsets and attention weights on the same modality to generate an aggregated feature f m q ∈ RC of xmq based on ym q and pmq . Differently, CDAM requires exploring the cross-modal feature correlations and thus we further add ym q to help predict point offsets and attention weights on the other modality m. Specifically, our method consists of two steps: First, we concatenate the two modality features ym q and ym q , followed by a mapping function to capture the complex correlations between them:

where [, ] represents the concatenation operation at the channel level, φ(·) is a linear layer to capture the inter-modal correlations, and ˆ yq ∈ RC is the generated fusion feature. Second, to generate cross-modal offsets concerning xmq , we incorporate the positional embedding pmq into the fusion feature ˆ yq and design a cross-modal offset predictor as follows:

where ϕ(·) represents the re-scales operation, and △pmq represents the predicted cross-modal offsets on the modality m concerning xmq . This cross-modal offset predictor could build a position mapping bridge between the two modalities, which automatically maps a reference point on one modality into several relative key points on the other modality, thereby effectively avoiding the pixel-level alignment problem.

Next, CDAM follows the single-modal deformable attention module to generate K key points with the corresponding attention weights by considering multiple heads, yielding an aggregated feature f m q on the modality m concerning xmq which is expressed as follows:

where H = 8 and K = 4 denote the indexes of the attention head and the key point, respectively. W m h ∈ RC×Cv and W ′m h ∈ RCv×C are the learnable weights (Cv = C/H). Amqhk is the scalar attention weight of the k-th key point on the h-th attention head, which is obtained via a linear projection as same as the single-modal deformable attention module. ym(.) returns the key point m feature according to position information.

Injected by visible and IR feature maps, CDAM automatically builds a point mapping mechanism between the two modalities, which could generate well-aligned visible and IR features for each reference point. Therefore, our method no longer requires point-to-point image alignment operations, thereby saving amounts of computational costs.

3.3 Point-Level Feature Consistency Loss in Hyperbolic Geometry

Considering the unsupervised mapping by CDAM may suffer from the cross-modal representation gap stemming from two aspects: the inherent representation difference caused by visible and infrared modalities, and the inaccurate mapping difference as the left two images in Figure 3 shows, where the corresponding visible and infrared points focus on different components of the car. Therefore, it is necessary to bridge the cross-modal representation gap. Motivated by this, we design a Point-level Feature Consistency (PFC) loss, which aims at reducing the inherent representation difference as well as guiding the cross-modal mapping by CDAM. Since visible and infrared features inherently possess substantial representation differences, existing Euclidean-space-based metric methods cannot accurately measure the similarity between them (see Figure 3). In contrast, hyperbolic space Dn , as a Riemannian manifold with constant negative curvature [Yaras et al., 2022], gives rise to geometric properties such as curvature, symmetry, and non-linearity [Khrulkov et al., 2020], making it better suited to capture domain-invariant features. Inspired by its successful application in image retrieval, we incorporate hyperbolic embedding into the proposed PFC loss, bridging the representation gap between visible-infrared feature pairs in hyperbolic geometry as well as guiding the cross-modal mapping by CDAM (see the right part in Figure 3).

Given a Euclidean vector a, the exponential bijective mapping expcz : TpRn → Dn of the Poincar´e ball modal Dn = {a ∈ Rn |c ∥a∥2 < 1, c ≥ 1} with the curvature parameter c projects a into hyperbolic space, denoted as:

where ⊕c is the differentiable M¨obius addition, z is the reference point empirically set to 0 for simplified computation, c and λcz = 2 1-c∥z∥2 denote the conformal factor that scales the local distance. Specifically, we first follow Eq. 4 to obtain the hyperbolic representations ˆ f m q and ˆ f m q by applying the exponential mapping to f m q and f m q extracted from SDAM and CDAM. Then, Our PFC loss Lpfc explicitly optimizes the features of two modalities to bridge the cross-modal representation gap. Inspired by contrastive learning [Chen et al., 2020], Lpfc encourages the cross-modal feature pair ˆ f m q and ˆ f m q concerning the same point xmq to be as similar as possible, which is formulated as:

where σ(·) is the sigmoid function, and Q is the number of input points. The introduction of Lpfc has two advantages: First, it aligns the key point features from two heterogeneous modalities, reducing their cross-modal representation differences in a high-dimensional feature space. Second, it effectively guides the learning of CDAM, implementing accurate cross-modal point mapping.

3.4 Point-Level Feature FusionModule

To better leverage the complementarity between visible and infrared features, we adopt a point-level feature fusion module to adaptively adjust the correlation weights for feature fusion. Concretely, we first construct the cross-modal correlation matrix Cm ∈ RQ×Q by performing matrix multiplication and softmax function softmax(·) as follows:

where f m ∈ RQ×C is a point feature matrix in the modality m and c·pq describes the inter-modality feature correlation between xq and xp. Similarly, we can obtain Cm ∈ RQ×Q. By mining the cross-modal feature correlation, we can adaptively query effective information from the complementary modality when the point features of a modality cannot accurately capture an object. Then, we multiply Cm with f m, followed by performing residual calculations to obtain the correlationweighted features CFm:

Similarly, we obtain CFm in the same manner, and the two correlation-weighted features are combined as the fused feature FFm by point-wise addition:

The fusion feature FFm is obtained by integrating two heterogeneous modal features at the point level, offering richer object information for object detection.

3.5 Implementation

The above fusion module provides two fusion features FFv and FFir to the two prediction heads to generate detection results on visible and IR images, respectively. For each branch, we adopt end-to-end training by using a composite loss function consisting of two components: the detection loss Ldet and the PFC loss Lpfc, denoted as:

where λ is a balancing weight. Here, Ldet measures the deviation between ground truth and prediction results. More details of Ldet could be referred to [Zhu et al., 2020].

4 Experiments

4.1 Datasets

We adopt three datasets KAIST [Hwang et al., 2015], FLIR [FLIR, 2018] and LLVIP [Jia et al., 2021] as the baseline to construct three weakly aligned datasets as follows:

• KAIST: This dataset is a challenging dataset for pedestrian detection. We adopt the improved dataset by [Liu et al., 2016] containing 7, 601 pairs for training and 2, 252 pairs for testing.
• FLIR: We use a cleaned version in [Zhang et al., 2020] including 4, 129 and 1, 013 visible/IR image pairs for training and testing with the image size 640 × 512.
• LLVIP: This dataset is captured in low-light environments for pedestrian detection. There are a total of 15, 488 visible/IR image pairs (1280 × 1024) divided into 12, 025 pairs for training and 3, 463 for testing.
• Weakly Aligned Dataset Generation: We create weakly aligned image pairs (KAISTw", FLIRw," "LLVIPw") by introducing random offsets on IR images while fixing visible images. Specifically, we use the horizontal and vertical offset with [-10, 10] pixels for FLIR, and [-20, 20] pixels for LLVIP as considering their resolution difference. Differently, we construct the KAISTw dataset following [Zhang et al., 2019], which adds two more offset directions including 45◦ and 135◦ .

4.2 Experimental Setups

All the experiments are deployed on NVIDIA RTX A6000 GPUs. Specifically, we adopt Resnet50 as the backbone and the transformer encoder with the same configuration of Deformable DETR. For the decoder, we set the number of reference points Q to 300. The weight λ of Lpfc is set to 0.05. To train our model, we execute a total of 60 epochs with the batch size 4 for all the datasets by using the Adam optimizer. The initial learning rate is 0.0001 and decayed by a factor of 0.1 at the 40-th epoch. All the testing results are reported by mAP at IOU = 0.5 (mAP@50), which is widely used in visible-infrared object detection [Chen et al., 2022b].

4.3 Evaluation on CDAM

We compare CDAM with two different cross-modal point mapping mechanisms as shown in Table 1, where SDAM directly uses the coordinates of key points in the single-modal branch to select key points in the cross-modal branch, and CDAM- aims at mapping a reference point in the singlemodal branch to a corresponding point in the cross-modal branch. Compared to CDAM, SDAM declines about 6% mAPs since selecting the two cross-modal points with the same coordinate for object detection cannot overcome the pixel offset problem on weakly aligned data, resulting in unpaired visible and IR feature generation. Moreover, CDAMachieves the worst performance because it adopts a two-stage point mapping including searching for a cross-modal reference point followed by searching for cross-modal key points, resulting in the error accumulation problem in point selection.

We further generate weakly aligned datasets with different offsets to verify the effectiveness of CDAM as shown in Table 2. On FLIRw dataset, the performance remains stable when the offset is less than 10 pixels. As we further increase the offset, the performance suffers from a drop but still exceeds 60% mAP values. Comparatively, LLVIP has a larger resolution to remain stable performance when the offset is less than 20 pixels, which proves the effectiveness of CDAM on weakly aligned data. However, we also discovered too large offsets still have a negative effect due to the increasing difficulty of cross-modal key point search. Figure 4 lists two examples to visualize the generated cross-modal key points concerning a reference point. It is shown that the corresponding reference points on the weakly aligned visible images have a certain positional deviation from that on the IR image. In such cases, CDAM is still able to identify relevant key points. Moreover, as the offset increases, the quality of key points has deteriorated with more points deviating from the objects, which is the reason for the performance degradation in Table 2.

4.4 Evaluation on PFC Loss

This experiment explores the effectiveness of the Point-level Feature Consistency loss Lpfc in Equation 5. As listed in Table 3, we compare the performance of Lpfc in Euclidean space with that in hyperbolic space. Obviously, using the loss in hyperbolic space achieves better performance, with 0.6% mAP improvement. This is because hyperbolic space effectively reduces the representation gap across modalities, thereby better guiding the learning of CDAM to offer accurate bimodal feature pairs for object detection. We further test the effect of different weights of Lpfc in Equation 9 as shown in Table 4. The best performance arrives at λ = 0.05 with mAPs of 77.0% and 77.1% on visible and IR images, respectively. The smaller values of λ overshadow the utility of Lpfc, which is not conducive to feature alignment. Comparatively, too large values of λ affect the detection loss Ldet, resulting in performance decline.

4.5 ComparisonWith State-of-the-Art Methods

We compare CF-Deformable DETR with several state-of-theart methods on KAIST, FLIR, and LLVIP datasets, and the corresponding results are shown in Tables 5, 6, and 7, respectively, where the first row part refers to strictly aligned detection methods, and the second part is weakly aligned detection methods. All the results in Table 5 are cited from the corresponding references, and those in Tables 6, and 7 are reproduced by using published codes. Overall, the strictly aligned detection methods suffer from a significant performance drop on weakly aligned data since they adopt weakly aligned bimodal features for fusion, thereby degrading the performance. Comparatively, weakly aligned detection methods alleviate this performance drop since they perform crossmodal image alignment before detection. Among all the weakly aligned methods, Our method performs the best and hardly leads to any performance degradation. This superior performance stems from the effective cross-modal deformable attention mechanism incorporating the feature consistency loss, which could implement accurate feature alignment to support weakly aligned object detection. Table 8 further lists the testing times of weakly aligned methods. Although our approach employs the transformer-based structure, the testing time is quite good as compared to the other CNN-based methods since our approach does not conduct cross-modal image alignment, saving a number of computational costs.

Figure 5 shows the detection results by different methods on three examples. Our approach could well detect all the objects on the two modalities, while MBNet and ProbEN3 miss several partially occluded objects. This result reflects the advantages of our cross-modal point-based detection on weakly aligned data, which explores multi-modal point features to better distinguish overlapping objects.

5 Conclusions

In this work, we delve into weakly aligned visible-infrared object detection and first propose an end-to-end alignmentfree model CF-Deformable DETR. Abandoning the existing alignment strategies, CF-Deformable DETR directly implements cross-modal point feature extraction and fusion conditioned on weakly aligned data, eliminating the alignment overhead. Technically, CF-Deformable DETR proposes a cross-modal deformable attention mechanism to automatically search for matching points as well as a point-level feature consistency loss to effectively guide the learning, offering well-aligned bimodal features for detection. Extensive experiments on three benchmark datasets confirm the effectiveness and efficiency of the proposed approach, yielding promising and stable performance on weakly aligned data. We hope our work can provide a new paradigm for solving weakly aligned object detection, promoting technological progress in this issue.

Acknowledgments

This work was partially supported by National Natural Science Foundation of China (No.U21A20518, No.U23A20341, No.62272157).