UniCT Depth: Event-Image Fusion Based Monocular Depth Estimation with Convolution-Compensated ViT Dual SA Block

1 Introduction

Depth estimation is crucial for understanding 3D scene structures, with broad applications in areas like autonomous driving and medical imaging [2022; 2024a]. While image-based methods have achieved significant success, they encounter limitations under extreme lighting conditions, where critical scene details may be lost. Event cameras, which respond to pixel-level intensity changes, provide advantages such as wide dynamic range and high temporal resolution, making them well-suited for challenging scenarios [2014]. However, the asynchronous and sparse nature of event data poses significant challenges in generating dense predictions from sparse features. To address these challenges, recent studies have explored event and image modality fusion to combine their strengths [2021; 2023b; 2024]. These fusion-based methods utilize the dynamic range and temporal resolution of event cameras alongside the detailed scene information from intensity cameras, achieving more accurate and robust depth estimation in challenging scenarios.

∗ Corresponding Author

CNNs are widely used for event-frame fusion in depth estimation tasks [2021; 2023b; 2023]. While effective at capturing local features, they struggle with modeling long-range dependencies, leading to poor performance in scenes with occlusions or significant depth disparities, such as the occlusion between the street lamp and the building shown in Figure 1. Some approaches use CNNs with recurrent structures for temporal modeling but face challenges such as gradient vanishing and exploding [2011]. Transformers excel at capturing long-range dependencies. Some methods design event representations suitable for Transformer, but they process modality features independently, limiting cross-modal interaction[2022; 2023; 2023]. SRFNet [2024] introduces an attention-based interactive fusion module to merge modality features with spatial priors, but it neglects channel-wise contribution, limiting its effectiveness in fully capturing crossmodal dependencies. Recently proposed Transformer-based [2024] uses a single Transformer encoder to capture crossmodal dependencies. However, it rely on standard selfattention over concatenated modality tokens, resulting in high computational costs and coarse modality fusion, which is susceptible to interference from long-range noise.

In this paper, we propose an event-image fusion-based depth estimation method, UniCT Depth, which adopts a unified conv-transformer architecture to collaboratively model local spatial features and global dependencies. In the encoder, we adopt a unified feature extraction and fusion design to reduce redundancy and repetitive computation in traditional separated designs, enhancing cross-modal interaction efficiency and joint representation expressiveness. We introduce the Convolution-compensated ViT Dual SA (CcViTDA) Block as the core unit in the encoder. This block integrates Context Modeling Self-Attention (CMSA) branch and Modal Fusion Self-Attention (MFSA) branch to optimize cross-modal collaborative representation. Spatially, CMSA captures contextual dependencies to improve depth estimation in complex scenes, while channel-wise, MFSA captures global dependencies and establishes correlations between modalities, facilitating effective cross-modal fusion. To further enhance the CcViT-DA Block, the tailored Detail Compensation Convolution (DCC) block refines local features, improves texture detail extraction, and emphasizes enhanced edge information in multimodal features. The contributions of this work are as follows:

• We propose UniCT Depth, an event-image fusion depth estimation with a CNN-Transformer architecture that combines local and global feature modeling. Its unified feature extraction and fusion design reduces redundancy and enhances cross-modal interaction for robust depth estimation.
• We design CcViT-DA Block to optimize cross-modal representation, integrating CMSA for spatial dependencies and MFSA for modal fusion. The tailored DCC block refines local features, improving depth details and edge representations of concatenated modalities.
• We conduct experiments using both public real-world datasets and simulated datasets. The results demonstrate that our method outperforms monocular depth estimation algorithms based on events, images, and their fusion, delivering better performance on key metrics.

2 RelatedWork

2.1 Image-based Depth Estimation

Early image-based methods employed probabilistic and feature-based methods. Yet, these approaches often performed poorly in non-aligned settings, especially when horizontal alignment conditions were not met. Learning-based methods, primarily using CNNs, have made significant progress and achieved outstanding results in depth estimation. Eigen et al. [2014] used a multi-scale convolutional neural network for monocular depth estimation, showing the feasibility of neural networks and inspiring further research into complex models [2015; 2016], loss functions [2018; 2020a; 2019], and auxiliary information [2020b; 2020] for improved accuracy. Recently, Transformers have achieved impressive results in computer vision [2020]. Ranftl et al. [2021] proposed dense prediction transformer (DPT), demonstrating the efficacy of Vision Transformers in dense vision tasks. Since then, transformer-based methods have been widely explored [2022; 2024]. While image-based methods excel in static or slowly evolving environments, they face considerable challenges under extreme lighting conditions or in the presence of rapidly moving objects.

2.2 Event-based Depth Estimation

Model-based methods jointly optimize pose and mapping by solving nonlinear optimization problems, yet typically produce only semi-dense depth [2016; 2018; 2018]. Learningbased methods significantly improve the performance of event-based monocular depth estimation, exhibiting strong generalization [2019; 2019; 2020; 2023a]. Zhu et al. [2019] employed a feed-forward neural network to jointly predict camera position and pixel disparity, but it produces only semi-dense depth estimates by applying a mask only to pixels where events occur. Tulyakov et al. [2019] generated dense metric depth maps by fusing data from stereo setups, while this method still depends on stereo setups and standard feed-forward structures. Hidalgo-Carri´o et al. [2020] proposed a recurrent neural network with temporal consistency supervision, achieving real-time monocular dense depth estimation. Shi et al. [2023a] improved the accuracy of eventbased monocular dense depth estimation by utilizing optical flow information between consecutive event frames. However, event-based methods struggle with high-resolution texture information due to the inherent sparsity of asynchronous event streams and their limited ability to capture scene details.

2.3 Event-image Fusion Depth Estimation

Due to the complementary nature of event and image frames, researchers have developed methods to fuse the two modalities. CNNs exhibit generalizability in learning local semantics [2024b], which leads to their widespread adoption for depth estimation in event-image fusion. Gehrig et al. [2021] proposed a fully convolutional recurrent asynchronous multimodal network for depth estimation that can process images and event data. Zhu et al. [2023] proposed a selfsupervised event-based estimation using cross-modal consistency between aligned frames and events for training. Shi et al. [2023b] proposed a three-stage monocular depth estimation framework with a low-light enhancement module, suitable for challenging nighttime conditions. CNNs-based methods rely on convolutional networks with limited receptive fields, leading to poor performance in multi-scale or occluded scenes. Recently, transformer-based methods have gained significant attention in event-image fusion for depth estimation. Sabater et al. [2022; 2023] proposed patch-based event representation for transformer architectures. Hamaguchi et al. [2023] employed a multi-level memory hierarchy to process event streams, designing an attention-based representation to encode event data. However, these methods treat each modality independently, leading to suboptimal modality fusion and reduced depth estimation accuracy. Pan et al. [2024] proposed a Spatial Reliability-oriented Fusion, which features an attention-based interactive fusion module that learns consensus regions to guide feature fusion, but it overlooks channel-wise information. The most relevant work to ours is Transformer-based [2024], which uses a single Transformer to fuse event and image data. However, it employs a basic self-attention mechanism, which is computationally expensive and lacks targeted modality fusion. Furthermore, it relies on ConvLSTM to process events, which is susceptibility to gradient when dealing with long-range dependencies.

In this paper, we propose a CNN-Transformer architecture that combines the strengths of local feature representation and global context modeling, effectively handling multi-scale and occluded complex scenes. In the encoder, we introduce ViT Dual SA block that analyzes the spatial and channel dimensions of modalities. This block not only balances long-range modeling performance and computational efficiency, but also promotes modality fusion by adaptively weighting channels. Besides, we design a DDC Block to enhance the local feature extraction capability, improving the model's ability to capture high-texture objects in complex scenes.

3 Method

In this section, we present our method for estimating dense depth maps from the given image and asynchronous event streams. We begin by transforming the event stream into an image-like representation. Then we present our network architecture and discuss the loss function used for training the network.

3.1 Event Representation

Given the high temporal resolution of event cameras, numerous events can occur in a short span, resulting in a sparse stream. We encode the event stream as a spatio-temporal voxel grid, discretizing the time domain to better preserve temporal information and reduce motion blur [Zhu et al., 2019]. Specifically, for an event 𝑒 = (𝑥, 𝑦, 𝑡, 𝑝), (𝑥, 𝑦) denotes the pixel location, 𝑡 the timestamp, and 𝑝 the polarity. The voxel grid is defined as a 3D tensor V ∈ R 𝐻×𝑊×𝐵 , where 𝐻 and 𝑊 are the height and width of the grid, respectively, and 𝐵 is the number of time bins. For every time bin, the timestamps of occurred events 𝐸𝑘 = {𝑒𝑖} 𝑀-1 𝑖=0 are scaled to the range [0, 𝐵 - 1]. Events are then accumulated to the corresponding voxel grid by bilinear interpolation. The voxel grid V𝑘 (𝑥, 𝑦, 𝑡) is defined as follows:

Here, 𝑡 ∗ 𝑖 = 𝐵-1 Δ𝑇 (𝑡𝑖 - 𝑡0). We set the height and width of the grid to match the resolution of the image.

3.2 Network Architecture

As shown in Figure 2, our method utilizes a U-Net-like architecture [Ronneberger et al., 2015] comprising a preprocessor, encoder, and decoder. The preprocessor conducts convolution on both event and image separately, extracting features from each modality to produce full-resolution feature maps. These maps are then concatenated and further convolved to merge features, yielding the full-resolution feature map.

The encoder comprises two residual blocks and four Convolution-compensated ViT Dual Self-Attention (CcViTDA) modules, enabling efficient communication across multiscale feature representations. To address the computational load of Transformers with high-resolution images, residual blocks are used to downsample the feature maps, producing half-resolution features. The proposed CcViT-DA modules then serve as the core units to build the encoder. CcViT-DA block downsamples the feature map by a patch embedding layer, which passes the input feature map through a convolution layer with a 3 × 3 kernel and stride of 2, generating a feature map with the resolution halved. In the encoder, the output feature maps have resolutions relative to the input image of { 1 2 , 1 4 , 1 8 , 1 16 , 1 32 }, respectively.

The decoder comprises five decoder blocks. Each decoder block includes a deconvolution layer with a kernel size of 3 and a stride of 2, used to double the size of the input features. In the decoder, the output features have resolutions relative to the input image of { 1 16 , 1 8 , 1 4 , 1 2 , 1 1 }, respectively. The network is designed with skip connections that concatenate features across layers, enhancing the representation of models.

CCViT-DA Block. The CcViT-DA block comprises a patch embedding layer, a ViT Dual self-Attention (ViT Dual SA) block, and a Detail Compensation Convolution (DCC) block. Figure 2 (a)illustrates the architecture of the CcViTDA block. We first apply patch embedding and positional embedding to the input feature maps. The patch embedding divides the input into non-overlapping patches, which are linearly transformed into tokens. Positional embedding encodes spatial information. The outputs of both embeddings are then combined to form the input for subsequent processing.

We introduce the ViT Dual SA block with two parallel branches: the Context Modeling Self-Attention (CMSA) branch for contextual dependencies and the Modal Fusion Self-Attention (MFSA) branch for global modal correlations. These branches jointly enhance cross-modal fusion by refining joint representations along spatial and channel dimensions. To further improve local feature representation, we design a DCC Block tailored for the ViT Dual SA block. Specifically, the outputs from both branches are dot-multiplied with the DCC Block outputs, concatenated, and passed through a merge layer to produce the final feature representation.

Context Modeling Self-Attention. The CMSA branch is designed to capture contextual dependencies within the spatial dimension and adapt to local depth variations. By dividing the image into non-overlapping windows, the network focuses on relevant regions within each window, helping to handle depth variations in challenging scenarios like occlusions. As shown in Figure 2(b), the CMSA block [Ding et al., 2022] calculates multi-head attention within localized windows. The image is partitioned into 𝑁𝑤 non-overlapping windows, each containing 𝑃𝑤 patches, such that the total number of patches 𝑃 is 𝑃 = 𝑃𝑤 × 𝑁𝑤. The attention operation within each window is computed as follows:

where 𝑄𝑖, 𝐾𝑖,𝑉𝑖 ∈ R 𝑃𝑤×𝐶ℎ denote the query, key, and value of the 𝑗-th attention head in the 𝑖-th window, respectively. 𝐶ℎ denotes the number of channels for each attention head.

Modal Fusion Self-Attention. The MFSA branch adaptively models cross-modal feature relationships along the channel dimension, capturing dependencies between modalities. It enhances the contributions of reliable modalities while suppressing noisy features, thereby improving modality fusion and model robustness. As shown in Figure 2(c), MFSA block [Ding et al., 2022] applies the self-attention on a patch-level transposed token, capturing global information along the spatial dimensions by setting the number of attention heads to 1. Each transposed token abstracts the global information. Channels are grouped, and self-attention is applied within these groups to reduce computational complexity. Let 𝑁𝑔 denote the number of groups, and 𝐶𝑔 denote the number of channels per group, thus 𝐶 = 𝑁𝑔 ×𝐶𝑔. The channel attention mechanism is defined as follows:

where 𝑄𝑖, 𝐾𝑖,𝑉𝑖 ∈ R 𝑃×𝐶𝑔 are grouped channel-wise imagelevel queries, keys, and values, respectively.

Unlike the standard self-attention mechanism with quadratic complexity [Cui et al., 2023], the proposed CMSA block and MFSA blocks reduce the computational complexity from 𝑂(𝑃 2 𝑑) to 𝑂(𝑃𝑃𝑤𝑑) and 𝑂(𝑃𝐶𝐶𝑔) respectively, significantly improving computational efficiency.

Detail Compensation Conv Block. We introduce a DCC block that enables the CcViT-DA block to consider both global and local information about the scene. We design a weighted module to facilitate the integration of the convolution and ViT self-attention. Figure 2 (d) illustrates the architecture of DCC Block, which aggregates the channel dimensions of the input feature maps to encode spatial regions for emphasis or suppression. Specially, the channel information of the input features is aggregated using channel-based global maximum pooling and global average pooling, and the two feature maps are concatenated in the channel dimension. Then, they are transformed into a single channel using a convolution layer, and the initial spatial attention map is generated after a sigmoid activation function. The initial spatial attention map are dot product to the input feature map to enhance the input feature space representation. Finally, the feature map is generated as a weighted output with the shape of 𝐻 ×𝑊 × 1, processed through two convolutional layers and activation functions.

3.3 Loss Function

The network is trained in a supervised manner, utilizing a loss function that incorporates both L1 and L2 losses. Throughout the training process, the network aims to minimize the loss function at each timestep. Given a sequence of predicted depth maps denoted as {𝐷𝑘}, we define the discrepancy 𝑅𝑘 = 𝐷 ∗ 𝑘 - 𝐷𝑘, where 𝐷 ∗ 𝑘 and 𝐷𝑘 represent the ground truth and predicted depth values, respectively. The loss function is defined as: 𝐿 = 1 𝑛 � 𝑢(𝑅𝑘 (𝑥, 𝑦) + 𝑅𝑘 (𝑥, 𝑦) 2 ), where 𝑛 represents the number of valid pixels in the ground truth depth values.

4 Experiments

4.1 Datasets and Evaluation Protocol

Due to the absence of event data in traditional image datasets like KITTI and NYU, we followed prior work and utilized event camera public datasets for our experiments. Specifically, we conducted primary experiments on the MVSEC dataset [Zhu et al., 2018] to evaluate our method in real-world daytime and challenging nighttime conditions. Besides, we also conducted experiments on the simulated DENSE dataset [Hidalgo-Carri´o et al., 2020] to verify the generalization of methods.

Our model was implemented in PyTorch, utilizing two NVIDIA GeForce RTX 3090 GPUs. We choose a learning rate of 0.0002 for MVSEC and 0.002 for DENSE. The ADAMW optimizer and MultiStepLR learning scheduler was used for training, with a batch size of 16. The model was trained on the dataset over 50 epochs, with the learning rate being reduced by a factor of 0.5 at the 10th, 20th and 30th epochs. The input image size was configured to 224 × 224. The number of voxel grid time bins was set to 5, which we found to be a good balance between temporal resolution and computational cost. The weights for the L1 and L2 loss functions were set to 1.

4.2 Comparison with SOTAMethods

MVSEC dataset. We compare our method with image-based methods, event-based methods, and fusion-based methods on the MVSEC dataset. Following previous work, we evaluate the average absolute depth errors (Avg. Error) of the methods at 10m, 20m, 30m cutoff distances. In Table 1, our method achieves the best performance across all cutoff distances on sequence average. In Table 2, we provide a detailed comparison of our method with state-of-the-art approaches on the MVSEC dataset. We employ commonly used metrics in depth estimation, including absolute relative error (Abs. Rel.), logarithmic mean squared error (RMSE log), and accuracy 𝛿𝑛 (𝛿<1.25 𝑛 , 𝑛 = 1, 2, 3). Our method achieves the best results in 10 out of 12 scores and remains highly competitive in the remaining two. On the most valuable metric, Abs. Rel., our method improves by 5.56% and 7.16% relative to the second-best method, SRFNet, on day and night scenes, respectively. Figure 3 presents a qualitative comparison on the MVSEC dataset. In low-light conditions, imagebased methods like DPT tend to lose objects, such as trees and cars in the third and fourth rows. Event-based methods lack sufficient texture information, resulting in poor performance in both detail prediction and overall depth estimation of the scene. Compared to the fusion method, RAMNet, our approach effectively merges the two modalities, enabling more accurate depth estimation even for information present in only one modality (e.g., cars in the first row and trees in the third row). Besides, our method outperforms all others in separating foreground and background, especially under occlusions.

DENSE dataset. Table 3 shows that our method achieves the best results on all cut-off Avg. Error, improving by 13.3% at 10m, 30% at 20m, and 10.9% at 30m compared to the second-best values. Furthermore, our method performs comparably to the best results in Abs. Rel metric. Figure 4 shows the comparison of the qualitative results of different methods on the DENSE dataset. Compared to the baseline, our method provides a more complete estimation of scene information, demonstrating the effectiveness of model fusion. Additionally, in regions with occlusions, our method more accurately estimates the depth differences between foreground objects and the background. For objects within the scene, such as trees, utility poles, and traffic lights, our method offers more precise depth estimations.

4.3 Ablation Study

We conducted an ablation study on the proposed CcViT-DA block. Table 4 compares different configurations of the ViT Dual SA and DCC block, with (1) to (5) examining the ViT Dual SA block and (6) to (8) assessing the DCC block. Furthermore, an additional ablation study on the input modalities is presented in Table 5.

ViT Dual Self-Attention Block. (1), employing a fully convolutional network, achieves the highest frame rate but also the highest error. (2), which employs traditional selfattention, lowers errors but incurs the lowest frame rate due to significant computation overhead. (3) and (4) improve the frame rate compared to (2). The combination of the MFSA and CMSA branches in (5) results in the lowest errors among configurations (1) to (5) while maintaining an acceptable speed, demonstrating the efficacy of our designed block. Detail CompensationConvolutionBlock. (8) outperformed (6) and (7) in terms of Avg. Error across all cut-off depth distances in both day and night scenes, demonstrating the effectiveness of the DCC block. Compared to all other configurations, our method (8) achieved the best depth estimation results while maintaining an acceptable real-time speed. Specifically, compared to (2), the Mean Avg. Error of (8) was reduced by 9.27%, 7.04%, and 7.43% at 10m, 20m, and 30m cut-off distances, respectively.

Modalities. Table 5 demonstrates that, when used independently, the event modality outperforms the image modality in night scenes, while the image modality achieves superior performance in daytime scenes. However, the combination of both modalities yields the best results in both types of scenes.

5 Conclusion

In this paper, we introduce UniCT Depth, a novel monocular depth estimation model that integrates asynchronous event data with images for improved depth accuracy in challenging lighting conditions. Our method combines ViT with CNNs to overcome the shortcomings of traditional CNNs methods, especially in complex multi-scale and occlusion scenarios. It features a dual self-attention block with spatial and channelwise branches for enhanced pixel relationship modeling and data interaction, significantly enhancing performance. Additionally, a detail compensation convolution boosts local feature extraction, improving the detection of high-texture objects. Our comprehensive experiments on public datasets show that UniCT Depth outperforms existing methods in key metrics. This work improves depth estimation and introduces a new strategy for data fusion across different modalities.

Acknowledgments

This work is supported in part by the National Natural Science Foundation of China under Grant 42201513, and in part by the China Postdoctoral Science Foundation under Grant 2022M723902 and Grant 2023T160789.