Self-supervised End-to-end ToF Imaging Based on RGB-D Cross-modal Dependency

1 Introduction

Depth sensing techniques are playing an increasingly important role in many fields, such as 3D reconstruction, understanding, and interaction. Over the past decade, various depth sensing techniques have emerged, including stereo vision, structured light and Time-of-Flight (ToF). Due to low cost, high accuracy, and lightweight design, ToF cameras have attracted increasing attention in commercial applications. However, ToF cameras typically suffer from a variety of noise, including phase wrapping, multi-path interference, motion artifact, and shot noise. Therefore, the ToF imaging method for producing high-quality depth images is of significant value.

∗Corresponding author

To improve the ToF imaging quality, some methods use a multi-stage pipeline and solve these problems in isolation, e.g., phase unwrapping [J¨aremo Lawin et al., 2016; Wang et al., 2021], multi-path removal [Marco et al., 2017; Agresti and Zanuttigh, 2018], and motion artifact suppression [Chen et al., 2020]. However, the cascaded imaging method suffers from error accumulation. Moreover, most of these methods rely on prior assumptions or pre-trained models, which usually deviate from real-world data. The domain gap between prior assumptions and real-world data results in limited performance.

To mitigate these issues, end-to-end learning frameworks have been proposed to recover the depth from raw correlation measurements directly in [Su et al., 2018; Zheng et al., 2021; Wang et al., 2023a; Wang et al., 2023b], which jointly realize multi-path interference removal, phase unwrapping, and de-noising. However, such frameworks also encounters the problem that the collection of real-world raw data paired with ground-truth depth images is difficult and even impractical. Supervision signals containing reconstruction error and misalignment also lead to poor imaging performance.

To address these challenges, this paper proposes a selfsupervised end-to-end learning method which does not require any noisy-clean depth image pairs for training. We exploit the cross-modal dependency between the RGB and depth modalities as supervision information, which is inspired by the nature that RGB images also contain rich geometric information of the scene.

In summary, the contributions are as follows:

• We propose a self-supervised learning framework for end-to-end ToF imaging without the requirement of any noisy-clean depth image pairs. It exploits the crossmodal dependency between the RGB and depth modalities as implicit supervision to suppress noise and preserve fidelity.
• We design a hybrid loss function incorporating the statistical characteristics of raw measurement data, which enhances robustness against noise and artifacts.
• We conduct extensive real-world end-to-end ToF imag-

ing experiments on four different off-the-shelf ToF cameras. The results demonstrate that the proposed method achieves competitive performance compared to supervised methods. Furthermore, our method consistently delivers strong performance across all evaluated cameras, highlighting its generalization capabilities.

2 RelatedWork

2.1 Cascaded ToF Imaging ProcessingMethods

Phase Unwrapping. Phase wrapping is a fundamental problem in many applications, such as synthetic aperture radar (SAR) [Pritt, 1996; Chen and Zebker, 2002], as well as in ToF imaging. Classical phase unwrapping methods estimate the phase wrap numbers with only a single depth image. These methods follow man-made assumptions, including the relationship between amplitude and depth [McClure et al., 2010; Cho et al., 2012], and the phase jump between adjacent pixels [Frey et al., 2001; Droeschel et al., 2010b].

Multi-shot methods take two depth images at two different frequencies to solve depth ambiguity. The common one is Chinese remainder theorem, that is effective in the easy cases with the absence of other errors. To address the hard cases with the presence of MPI or noise, multiple image priors are proposed, such as amplitude constraints [J¨aremo Lawin et al., 2016; Wang et al., 2021], and frequency constraints [Droeschel et al., 2010a].

Multi-path Interference Removal. Due to the complexity of multi-path effect, traditional methods usually take multiple measurements at different modulation frequencies. These methods recast the multi-path as a sparse estimation problem and solve from different perspectives, including optimization [Bhandari et al., 2014; Freedman et al., 2014], spectral estimation [Feigin et al., 2015; Kirmani et al., 2013], and compressive sensing (CS) [Xuan et al., 2016]. Hardware-based methods introduce an extra projector [Naik et al., 2015] or custom coding [Kadambi et al., 2013].

Recently, learning based methods for MPI removal have been proposed [Son et al., 2016; Marco et al., 2017; Agresti and Zanuttigh, 2018; Qiu et al., 2019; Gutierrez-Barragan et al., 2021]. However, these learning based methods have an unsatisfactory performance in real-world data.

2.2 End-to-end ToF Imaging ProcessingMethods

The multi-stage pipeline often suffers from cumulative errors and information loss. To address these issues, the end-toend framework is widely adopted, leveraging noisy-clean image pairs for ToF imaging processing in [Su et al., 2018; Yan et al., 2020; Zheng et al., 2021; Gao et al., 2021; Jung et al., 2021; Li et al., 2022; Meng et al., 2024; Tang et al., 2024]. Su et al. [Su et al., 2018] firstly propose a GANbased architecture for ToF imaging from dual-frequency, raw correlation measurements. Zheng et al. [Zheng et al., 2021] design a multi-stage iterative CNN for I-ToF depth error removal. Jung et al. [Jung et al., 2021] take an fusion of RGB and raw correlation data as input and focus on real-world scenarios with strong ambient light and far distances.

However, existing methods rely on either 3D reconstruction or other commercial depth cameras as ground truth, which is not sufficiently reliable. Zheng et al. [Zheng et al., 2021] propose to use 3D reconstruction results as clean depth image, but this method suffers from reconstruction errors and misalignment. Su et al. [Su et al., 2018] propose to generate noisy data by adding synthesized noise and degradation to clean data. However, real-world noise and degradation are complicated and difficult to simulate accurately. Therefore, the performance of these methods is limited due to the domain gap between real-world data and generated noisy-clean pairs.

2.3 Unsupervised LearningMethods for Image Enhancement

To address the challenges in collecting clean image for supervision, unsupervised and self-supervised learning methods without noisy-clean pairs have recently gained increasing popularity in image enhancement. For example, recent methods use noisy-noisy pairs [Lehtinen et al., 2018], noisier-noisy pairs [Moran et al., 2020; Krull et al., 2019], noisy images only [Batson and Royer, 2019; Xu et al., 2020; Wang et al., 2023a] instead of noisy-clean pairs for image denoising. Most of these methods present a specific prior model on the noise signal, which may lead to unsatisfactory in real-world scenarios. Wang et al. [Wang et al., 2023b] propose an optimal transport based method that shows remarkable superiority in raw depth image denoising without any prior models, but it cannot handle other error sources. Similarly, Agresti et al. [Agresti et al., 2019] propose an unsupervised method for multipath interference removal based on pixel-level domain adaptation, while it still needs clean-noisy pairs for training.

3 Background

Typical ToF imaging systems consist of two main components, namely an active illumination module (emitter) and an image sensor (receiver), as shown in Figure 1. The distance d is derived from the phase shift ∆ϕ between emitted signal g(t) and reflected signal s(t), which can be written as

where ∆ϕ is the phase shift, λ is the wavelength, fm is the modulation frequency, and c is the speed of light. The problem is to solve for the phase shift.

The image sensor cannot measure the phase shift ∆ϕ directly, but measures the integral of the raw correlation function instead as

with

where r(t) is the reference signal usually equal to the emitted signal g(t), α is the amplitude attenuation coefficient, and β represents the intensity of the ambient light. Then, the raw correlation measurement in (2) can be written as

In practice, with four sampling points of iπ/2 for i ∈ {0, 1, 2, 3} in a period, we can obtain four raw correlation measurements h0 to h3 and calculate the phase shift ∆ϕ with these four raw correlation measurements, further the depth d and amplitude value α as

The above method is most commonly used in ToF cameras. However, due to the imperfection of ToF measurements, some geometric information would be lost, while undesired noise may be introduced, resulting in inaccurate depth estimation, such as those caused by multi-path interference, phase wrapping, etc.

4 Methodology

Grounded in information theory, we derive a self-supervised formulation and implement it using WGAN-based adversarial learning, combined with MAE fidelity and smoothness losses, to guide training without the need for clean GT pairs. By maximizing the cross-modal mutual information, the model learns to generate high-quality depth from raw-data input.

4.1 Theoretical Analysis

Taking raw correlation measurements C as input, the ToF imaging model can be denoted as

where ˆX is the estimated depth image of the model.

It is relatively easy to train an imaging model with the ground truth depth image X as supervision. However, the challenge lies in collecting noisy-clean pairs ( ˆX,X). To overcome this challenge, we use the mutual information between the RGB image R and the depth image X, i.e. I(X; R) to supervise the learning of an imaging model. To achieve this, we consider the following formulation

where d(·, ·) measures the distance between distributions, such as the KL divergence or Wasserstein distance. p ˆX,R and pX,R are the joint distributions of ( ˆX, R) and (X, R), respectively.

According to information theory, mutual information I( ˆX;R) is equivalent to the KL divergence DKL(p ˆX,R∥p ˆX pR), represented as

Generally, transition from an actual joint distribution to a joint distribution under the assumption of independence will either remain or increase the KL divergence. In other words, the KL divergence between p ˆX,R and p ˆX pR is no less than that between p ˆX,R and pX,R, which can be written as

The mutual information I( ˆX;R) is upper bounded by I(X; R) based on information theory, denoted as

Therefore, minimizing the divergence between the two joint distributions p ˆX,R and pX,R in (10) would maximize the mutual information I( ˆX;R). The above analysis motivates us to design a self-supervised end-to-end framework leveraging RGB-D cross-modal dependency as supervision for training a ToF imaging model.

4.2 Network Architecture

The proposed formulation (10) is implemented based on a WGAN architecture, as shown in Figure 2. The generator G estimates the de-noised depth image while preserving the geometry information of raw measurements. The discriminator D forces the model to learn the common RGB-D cross-modal dependency between ˆX and R, which can further suppress noise and improve fidelity.

The proposed generator consists of a multi-branch feature extraction module and a U-Net module. The feature extraction module includes a raw correlation branch, a depth branch, and a depth-amplitude branch. We use the Residual Channel Attention Block (RCAB) [Zhang et al., 2018] as backbone, which can focus on more informative features. The features from the three branches are concatenated as input to the U-Net architecture for depth image generation.

The discriminator is designed to determine whether the generated depth image and the paired RGB image ( ˆX,R)

maintain consistent cross-modal dependencies with the synthesized RGB-D image pair (Xs ,Rs ), which guides the model to generate a depth image with correct geometry and high fidelity relative to the paired RGB image. Note that the generated depth image ˆX and the synthesized depth image Xs in the discriminator are not paired, which relaxes the requirement for paired noise-clean depth images in supervision.

4.3 Loss Functions

As shown in Figure 2, the proposed framework is implemented based on WGAN and we propose a hybrid loss to train the end-to-end ToF imaging model as

where λ1, λ2, and λ3 are hyperparameters.

In (14), the first term represents cross-modal adversarial learning, which guides the model to suppress noise and improve fidelity based on the geometric information implied in the RGB image. Besides, the loss function takes into account the statistical characteristics of both depth data and raw correlation data. The last three terms correspond to three branches of the generator. The details are given as follows.

Ldep minimizes the mean absolute error between the output generated depth image ˆX and the original depth image X0 calculated from raw correlation measurements C as

Lcorr minimizes the mean absolute error between the reconstructed correlation measurements ˆ C and input raw correlation measurements C as

where ˆ C is reconstructed based on ˆX and amplitude Z at two modulation frequencies f1 and f2 in the following form as

Lsmooth is used to ensure the local smoothness of generated depth image. Especially, we use a total variation loss [Su et al., 2018] can be written as

5 Experimental Results

In this section, we evaluate the proposed method on both synthetic and real-world ToF raw data captured with four different off-the-shelf ToF cameras to evaluated the performance and generalization capabilities.

5.1 Experimental Settings

In our method, the discriminator is trained by paired RGB-D data {(ˆxi, ri)} and {(xsi , rs i )}, while the generator takes raw correlation data ci, directly calculated noisy depth data ˆ xi and amplitude data zi as input. The U-Net module of generator consists of two down-sampling CNN layers in the encoder and two up-sampling CNN layers in the decoder. The synthesized RGB-D pairs {(xsi , rs i )} for discriminator are from the synthetic dataset [Zheng et al., 2021], which is generated based on simulating the imaging system of the LUCID Helios camera. Table 1 shows the hyperparameter tuning procedure for the loss function. Using PSNR as the evaluation metric, we set λ1 = 10 and λ3 = 20. λ2 is empirically set to 5. The initial learning rate is 0.1, and the optimizer is RMSprop. The maximum number of epochs is set to 200. The code is implemented in PyTorch and run on Nvidia 3090Ti. The comparison of computational complexity is shown in Table 2, which shows that our method has medium parameter amount and computation cost.

5.2 Experiments on Synthetic Datasets

In this experiment, we evaluate the proposed method on a synthetic dataset [Zheng et al., 2021] under the scenario with combined corruptions. Table 3 shows the quantitative results of the compared methods in terms of mean absolute error (MAE), relative error (Rela.), and PSNR. Without using noisy-clean pairs as supervision, our self-supervised method achieves competitive performance compared to the supervised method [Zheng et al., 2021] which is trained using noisy-clean pairs.

Moreover, Figure 3 compares the visual quality along with quantitative scores on samples from the synthetic dataset. The depth images are shown in pseudo-color, with a red-to-blue color scheme representing distances from near to far. The first one is a bathroom case with high reflectivity mirror and a corner. In the second case, the main degradation is lowintensity noise caused by a black curtain. It can be observed that generated depth images of ours can suppress noise while preserving geometry information, which achieve competitive performance with supervised method [Zheng et al., 2021].

5.3 Experiments on Real-world Datasets

In this experiment, we evaluate the proposed method on realworld data captured by four off-the-shelf ToF depth cameras, respectively LUCID, TI, TCS, and TCE. The basic parameters of four cameras are listed in Table 4. The first two datasets are captured by [Zheng et al., 2021] and [Su et al., 2018]. We test on the data of two additional cameras to further prove the generalization of our proposed method.

Comparison on real-world data captured with TI. Figure 4 illustrates two cases of the daily scenes from the realworld dataset [Su et al., 2018]. The raw measurements are captured by a TI OPT8241-CDK-EVM camera, including noise, invalid pixels, and complex reflectance. It can be observed that the supervised method [Zheng et al., 2021] does not generalize well to the real-world data captured by a different depth sensor. Compared with [Su et al., 2018], our proposed method can preserve detail information while effectively removing noise. For example, in the first sample, the potted leaves are blended into the background in [Su et al., 2018], but our method retains the details.

Comparison on real-world data captured with LUCID. Figure 5 compares the visual quality on two samples from the real-world dataset captured by Lucid. It can be observed that the method [Su et al., 2018] does not perform well in this dataset, due to the difference of the noise model. The supervised method [Zheng et al., 2021] recovers smooth depth image, but it is trained using data captured by the LUCID camera. However, the method [Zheng et al., 2021] changes the shape of the object, such as the distorted edges in the second scene. In comparison, our proposed method can recover a cleaner depth image with more reasonable 3D geometry.

Comparison on real-world data captured with TCS. Figure 6 illustrates two cases of the daily scenes in the office, captured with TCS. The first scene is a far-distance scene with a maximum range of 7 meters, while the second one is a close-range scene with complex reflectance and shape. The raw ToF measurements are captured at two high frequencies, respectively 80M and 100M. Therefore, the depth images suffer from phase unwrapping and the pseudo-color visualization of the raw depth image exhibits periodic repetition. As shown in Figure 6, the complex noise leads to failure of [Su et al., 2018] and [Zheng et al., 2021]. Compared with these two methods, the proposed method can restore the depth image with explicit edges and less noise.

Comparison on real-world data captured with TCE. Figure 7 illustrates two cases of the daily scenes in the room. The first case is a full-view of room. The second one is a corner case. Both of the methods can remove random noise, but the restored depth image by [Zheng et al., 2021] is oversmoothed, and errors occur at the boundary of the image. For an intuitive comparison of multi-path removal, we randomly select a scanline (blue line in Amplitude). As shown in Figure 7, the scanline of the raw depth and restored depth by [Zheng et al., 2021] is far from the ideal structure of the corner, as the scanline is curved and messy. Note that Su et al. requires dual-frequency raw ToF correlation measurements as input to recover depth. However, the TCE camera provides only single-frequency measurements. Therefore, the results of Su et al. are not included in Fig. 7.

5.4 Ablation Experiments

In this section, we provide ablation study to evaluate the components of the proposed loss functions. The ablation experiments are conduct on a small dataset which contains only a quarter of the full synthetic dataset [Zheng et al., 2021].

As shown in Table 5, the model performs best when all the components are used as the loss function, which demonstrate that the hybrid loss function incorporating statistic characteristics of raw measurements can improve the performance of the imaging model.

6 Conclusion

In this paper, we propose a self-supervised learning framework for end-to-end ToF imaging, which does not require any noisy-clean pairs yet generalizes well across various off-theshelf cameras. The proposed self-supervised framework utilizes the cross-modal dependency between RGB and depth data as implicit supervision to suppress noise and preserve fidelity. The loss function incorporates the statistical characteristics of rawmeasurement data, improving robustness against noise and artifacts. Experimental results on both synthetic and real-world data demonstrate that our proposed method can suppress noise and preserve fidelity, which achieves competitive performance with supervised methods. Furthermore, our method consistently delivers strong performance across all evaluated cameras, highlighting its generalization capabilities across various scenarios.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (NSFC) under Grant 62271314.