Frequency-Aware Deep Depth from Focus

1 Introduction

Obtaining three-dimensional (3D) shape information of a scene using visual cues is an important problem in the field of computer vision. Current methods for 3D shape reconstruction are primarily based on optical imaging and can be broadly categorized into active optical reconstruction and passive optical reconstruction. Active optical methods require additional equipment to assist in obtaining the 3D structure of the scene. Typical methods include laser confocal microscopy and structured light 3D measurement. However, the former suffers from low scanning efficiency and a limited measurement range, making it unsuitable for scenes with large depth variations and high real-time constraints. On the other hand, the latter is easily affected by high reflectivity and lighting conditions during the reconstruction process, which can lead to decreased reconstruction accuracy [Yan et al., 2020b]. Passive optical reconstruction primarily recovers the 3D structure of a scene from 2D images. Due to the low cost and convenience of acquiring scene images, many different types of methods have emerged, such as multiview stereo vision [Wei et al., 2023] and monocular depth estimation [Yang et al., 2024] methods. However, multiview stereo vision heavily relies on the accuracy of left-right view matching, monocular depth estimation requires all-in-focus images as a prerequisite for accurate depth estimation. For scenes requiring high-resolution imaging, a large aperture means shallow depth of field (DoF), making it difficult to obtain all-infocus images in a single shot.

To address the aforementioned challenges, Depth from Focus (DFF) technology [Nayar and Nakagawa, 1994] holds promise for efficiently and cost-effectively obtaining highprecision 3D shape information of a scene. DFF works by adjusting the distance between the camera and the scene at regular intervals, capturing a sequence of images that cover the entire depth range of the scene. Depth information is then obtained by analyzing the focus changes within the multifocus image sequence. This is particularly important for scenes requiring high-resolution reconstruction. For instance, DFF technology is used to achieve high-precision 3D shape reconstruction of industrial microscopic scenes [Yan et al., 2020a]. Additionally, some Nikon commercial cameras also feature focal length variation shooting capabilities [Mandl et al., 2024], which can subsequently be utilized to achieve high quality DoF synthesis using DFF technology. Particularly with the emergence of deep neural network models, their powerful ability to represent focus features has shown strong performance in addressing DFF problems [Hazirbas et al., 2019]. Nevertheless, due to the spatial domain operation limitations of convolutional neural networks, existing deep learning-based DFF methods still have room for improvement in terms of preserving fine-grained details and robustness of the scene.

In this work, aiming for finer depth details, we introduce the FAD model from the perspective of frequency domain feature extraction. Unlike existing state-of-the-art DFF methods, our model demonstrates well generalization ability and achieves high-precision scene reconstruction directly from multi-focus image sequences. Our contribution is three-fold:

• A frequency domain feature extraction module is introduced, which effectively preserves the global structural information of the scene, thereby enabling the reconstruction results to achieve more accurate fine-grained details.
• The time-frequency joint module in our FAD model not only enhances the robustness of datasets with varying resolutions but also effectively ensures the consistency of depth information in homogeneous regions and the continuity of depth information in heterogeneous regions within the reconstruction results.
• Compared to state-of-the-art DFF methods, the FAD model exhibits outstanding performance across eight different types of existing datasets. Notably, it can serve as a pre-trained model for rapid adaptation and application in real microscopic scenarios. An experimental evidence is provided in Figure 1.

2 RelatedWork

2.1 Depth fromFocus (DFF)

DFF originated in the field of microscopic imaging and is now widely used in the macro imaging process of commercial cameras [Suwajanakorn et al., 2015]. The commonality between these two applications is the need to shorten the distance between the camera and the scene being measured, which results in a shallower DoF. Additionally, shortening the distance means less light enters the camera, necessitating a larger aperture to allow more light in, which further exacerbates the shallowness of the DoF. These settings lead to only part of the scene being in focus in a single image, with the rest becoming blurred. Therefore, DFF first captures a sequence of images covering the entire DoF by varying the distance between the camera and the scene. It then aggregates the indices of the sharpest pixels from this sequence to derive the depth information of the scene.

2.2 State-of-the-art DFFMethods

Early DFF methods focused on model design, particularly the development of image focus measure operators and depth map refinement. Focus measure operators aim to enhance the signal-to-noise ratio between sharp and defocused pixels, improving the accuracy of focus determination. Representative operators include Laplacian-based measures [Yan et al., 2020a], ring difference filters [Jeon et al., 2020] and transformation-based measures [Yan et al., 2020b]. Depth map refinement uses prior knowledge to correct focus measurement errors, with notable methods such as cost aggregation [Jeon et al., 2020] and regularization [Ali and Mahmood, 2021]. However, these methods often struggle in scenarios with sparse textures and noise interference. In addition, these methods cannot accurately infer complex samples that satisfy similar distributions. To overcome the above issues, deep learning-based DFF methods have emerged [Yang et al., 2022; Maximov et al., 2020; Wang et al., 2021; Fujimura et al., 2024]. These methods design end-to-end convolutional neural networks to directly learn depth information frommulti-focus image sequences. However, a common characteristic of these networks is that they fit the focus measure features of different datasets through convolutional neural networks. Essentially, this approach belongs to the fitting and learning of temporal focus measure operators in model design-based DFF methods. As a result, these methods tend to overlook the global structural information of the scene, leading to blurred depth details in the reconstruction results.

2.3 Applications of Frequency-domain in Vision

Recently, an increasing number of studies have begun incorporating frequency domain methods into the field of deep learning to optimize the performance of deep neural networks [Yang and Soatto, 2020; Xu et al., 2019]. Some studies transform images into the frequency domain to leverage frequency information and enhance the performance of specific tasks [Lee et al., 2018; Rao et al., 2021], others use the convolution theorem to accelerate the computation of convolutional neural networks (CNNs) via fast Fourier transform (FFT) [Li et al., 2020; Ding et al., 2017; Huang et al., 2023]. Inspired by the above works, we believe that introducing a frequency domain processing module in DFF will help enhance the global feature representation of the network, thereby capturing more subtle focal changes.

3 Methodology

Our approach combines spatial and frequency domain information, employing a U-Net architecture to enhance depth estimation accuracy by integrating features from both domains. First, we provide an overview of the proposed FAD method, detailing the overall network process. Next, we describe the extraction of spatial and frequency domain features, including how to effectively integrate these features into the U-Net architecture. Finally, we discuss the implementation details of the network, covering parameter settings, optimization strategies and loss function selection for training to ensure model stability and efficiency.

3.1 Overview

Figure 2 presents an overview of the proposed FAD method, which is based on an encoder-decoder architecture utilizing 3D convolutions instead of 2D convolutions. Unlike 2D convolutions, which can only capture features from a single image frame, 3D convolutions are capable of capturing focus change trends across different frames in a multi-focus image sequence. Feature fusion within the network is achieved through skip connections, which effectively transmit feature information across various levels, thereby enhancing the network's representational capacity. This approach also facilitates gradient propagation and accelerates model convergence.

In the encoder phase, the input multi-focus image sequence undergoes initial feature extraction through a series of 3D convolutional layers. Each layer includes convolution operations, batch normalization and ReLU activation functions to ensure that the extracted features are highly representative and capable of nonlinear expression. Following this, these features are meticulously processed by the clarity area identification (CAI) module, yielding a feature volume that serves as the input for the time-frequency joint module. As the network deepens, the encoder compresses image information into higher-level feature representations while preserving the spatiotemporal relationships within the multi-focus image sequences. Max pooling is employed for downsampling, and to mitigate the loss of depth information, a 3 × 3 convolution is applied to the output of the downsampling process. In the middle phase of the encoder-decoder, separate spatial and frequency domain feature extraction modules are employed to maintain precise focus information in both dimensions. This approach enables the network to better adapt to varying scales and resolutions, thoroughly learning both global structures and local details, thereby improving the accuracy of depth estimation.

In the decoder phase, features from the encoder are decoded layer by layer using a series of transposed convolutions to gradually restore the spatial resolution of the feature maps. These transposed convolutions, combined with symmetrical 3D convolutional layers from the encoder, progressively return the feature maps to the dimensions of the input images. Each layer in the decoder incorporates skip connections from the encoder to enhance information fusion during decoding. This method not only leverages the layer-by-layer recovery capability of the decoder but also effectively integrates the multi-level features extracted during the encoder phase, ensuring the final output image has a higher quality depth estimation.

3.2 Clarity Area Identification

Our initial feature extraction phase consists of 3D convolution and the CIA module. The 3D convolution, with its unique capability, allows for feature exchange with adjacent focal slices, enabling it to detect and identify subtle defocus variations that are often hidden in weak areas with little discernible texture. This ensures high precision and sensitivity in detection. In particular, previous methods utilized global average pooling to simplify the multi-dimensional information of focal slices into a single value, which inevitably led to a significant loss of deep representation information. To overcome this obstacle, we innovatively adopted the CIA module, which uses 3D convolution and ReLU to compute a focal attention score, allowing for the capture of more refined deep representation information.

3.3 Spatial Domain Feature Extraction

The multi-scale feature extraction method can effectively capture details and structural information of different scales in an image. This characteristic is particularly important for reconstructing sparse textures and low-contrast areas in DFF tasks. Therefore, our FAD method utilizes multi-scale feature extraction to enable the model to extract spatial domain features from the scene.

Specifically, given a feature map X, it is first subjected to multi-scale average pooling operations to obtain feature representations Xi at different scales. Let the pooling scales be s1, s2, s3. The multi-scale pooling process can then be represented as:

Next, convolution operations are applied to the pooled feature maps Xi to further extract spatial domain features at multiple scales. Let the convolution kernel sizes at different scales be k1, k2, k3. The convolution process can then be represented as:

The convolution results at different scales are concatenated and fused to obtain the final multi-scale feature representation Fl .

This multi-scale feature extraction method first utilizes average pooling operations to reduce computational load and redundant information while preserving important spatial information. Subsequent convolution operations can further extract and refine features, enhancing the model's ability to perceive spatial structures at different scales. Finally, by fusing feature representations frommultiple scales, the performance of the model in handling complex DFF scenes and tasks at different resolutions is improved.

3.4 Frequency Domain Feature Extraction

Although the spatial domain feature extraction module can capture some global information, the local nature of convolution operations limits its ability to effectively represent global structural information. In contrast, the Fourier transform can process global information in the frequency domain by performing a weighted sum of all image pixels. Therefore, we introduce a frequency domain feature extraction module in the FAD model to more accurately capture subtle focus changes that are difficult to detect in the spatial domain, thereby effectively enhancing the model's sensitivity and accuracy in the depth information extraction process.

Given a feature X as the output of the encoder, we first transform the feature map X into the frequency domain by performing a Fast Fourier Transform (FFT), obtaining its frequency representation Xf . Next, to adaptively select the required frequency information, we learn adaptive weights for Xf to weight the spectrum, obtaining the weighted representation Xω. This process can be expressed as:

where M is a learnable adaptive weight matrix, and ⊙ denotes element-wise matrix multiplication. To implement M, we use a 1 × 1 convolution followed by a ReLU activation function and another linear layer. With respect to the output of the linear layer, the result of this soft thresholding is multiplied with the output of the Fourier transform, yielding the final data representation Xs. This step is designed to simultaneously enhance the sparsity of the output and improve the interpretability of the data, thereby enhancing the accuracy and robustness of deep information extraction.

Finally, we use the inverse Fast Fourier Transform (iFFT) to convert the processed features from the frequency domain back to the spatial domain, obtaining the spatial representation Xf -1 :

To prevent information loss, we combine the features before and after the Fourier transform, X and Xf -1, resulting in a comprehensive feature Fg that ensures the integrity of information during the transformation from the spatial domain to the frequency domain.

By employing a parallel extraction method for both spatial domain features Fl and frequency domain features Fg, the final feature map F is obtained.

The FAD model utilizes frequency domain information to compensate for multi-scale spatial domain features, effectively enhancing the model's performance in fine depth structure estimation.

3.5 Prediction

The feature map F, extracted by the time-frequency joint module, undergoes refinement through 3D CNN to generate four feature maps Fi at different scales. Subsequently, these feature maps undergo an upsampling process sequentially, and a Softplus function is applied for nonlinear normalization, resulting in refined depth attention maps. Immediately following this, the depth attention maps are elementwise multiplied with equally spaced focal positions P, precisely integrating information from each focal position to obtain the final scene depth map Di.

3.6 Implementation Details

We build our network model using the PyTorch framework and train and test it on a single NVIDIA GeForce RTX 4090. During training, we employ the Adam optimizer (β1 = 0.9, β2 = 0.99) with an initial learning rate of 10-3 and a batch size of 4. Additionally, we apply data augmentation techniques such as image flipping, cropping, rotation, and gamma correction. Images are randomly cropped into 256 × 256 patches and fed into the network.

For our depth estimation method, we optimize the entire model by comparing predicted pixel depths to ground truth depths with a multi-scale weighted loss function. The specific loss function is defined as follows:

where || ·||2 denotes the L2 loss, and Dgt represents the ground truth depth map. i ∈ {1, 2, 3, 4} indicates the predicted depth maps at different pyramid-like scales. In this model, ωi is set to 0.3, 0.5, 0.7 and 1, respectively.

4 Experiments

In this section, we detail the evaluation metrics and datasets used in our experiments and compare the proposed FAD method with state-of-the-art DFF methods. Furthermore, we conduct ablation experiments to assess the effectiveness of each component of the proposed network. Finally, we evaluate the model's generalization ability across various synthetic and real microscopic datasets.

4.1 Metrics

For quantitative evaluation, we use the following metrics to evaluate the quantitative results: mean-squared error (MSE), root-mean-squared error (RMSE), log root-mean-squared error (log RMSE), relative-absolute error (AbsRel), relativesquared error (SqRel), accuracy with δi = 1.25i , i = 1, 2, 3 and inference time (Secs.).

4.2 Comparison of Algorithms and Datasets

The comparison includes seven deep learning-based DFF methods (DDFS [Fujimura et al., 2024], DfFW [Won and Jeon, 2022], FV [Yang et al., 2022], DFV [Yang et al., 2022], AiF [Wang et al., 2021], DefocusNet [Maximov et al., 2020], DDFF [Hazirbas et al., 2019]) and two model design-based DFF methods (RDF [Jeon et al., 2020] and RR [Ali and Mahmood, 2021]). The datasets consist of five synthetic datasets (SLFD [Shi et al., 2019], Pov-Ray [Heber and Pock, 2016], DefocusNet[Maximov et al., 2020], 4D Light Field [Honauer et al., 2017], FlyingThings3D [Mayer et al., 2016]) and four real datasets (NYU Depth V2 [Carvalho et al., 2018], DDFF 12-Scene [Hazirbas et al., 2019], Middlebury [Scharstein et al., 2014], Microscopic), among which the microscopic dataset is unlabeled.

4.3 Main Results

The best results among all experimental results are highlighted in bold, and the second-best results are indicated with underline.

Results on DefocusNet Dataset. Table 1 presents the quantitative comparison results for DefocusNet dataset. It can be seen that, except for the AbsRel metric, which is slightly inferior to the DfFW method, our FAD method outperforms other methods in all other metrics. As shown in Figure 3, our FAD method significantly outperforms the DDFS and DfFW methods in inferring the elliptical cavities and spiral details of the objects.

Results on SLFD Dataset. Table 2 presents the quantitative comparison results on the SLFD dataset. Our method has significant improvements in all metrics, although it is slightly slower than DfFW. As shown in Figure 4, our method excels at maintaining the integrity of object edges and fine structures within the scene, including details such as the spokes of a bicycle and the switch of a lamp.

Results on 4D Light Field Dataset. As shown in Table 3, our FAD method significantly outperforms advanced deep learning-based DFF methods across three evaluation metrics.

Due to the introduction of frequency domain features, it can be seen from Figure 5 that our FAD significantly outperforms other methods in terms of the continuity of the light string and the details of the holes in the boxes.

Results on DDFF 12-Scene Dataset. Table 4 presents the quantitative comparison results for DDFF 12-Scene. Compared to other methods, our FAD method achieves the best results across all metrics. As shown in Figure 6, our FAD method significantly outperforms the DFV and DfFW methods in preserving edge details for statues and chairs.

4.4 Ablation Study

To further validate the rationale behind our FAD method and the effectiveness of its components, we conduct ablation experiments on the 4D Light Field dataset and analyze the results based on model performance. In Ablation Experiment 1 (U), we train the model using only the U-shaped backbone network structure. In Ablation Experiment 2 (U + S), we incorporate the spatial domain module to extract finer features. In Ablation Experiment 3 (U + F), we utilize the frequency domain module to capture frequency information and extract global features. In Ablation Experiment 4 (U + S + F), we combine both the spatial and frequency domain modules. As shown in Figure 7, the experimental results indicate that simultaneously utilizing spatial and frequency domain information can complement each other and greatly enhance the model's performance.

4.5 Generalization

Generalization among Different Datasets. To validate the generality of the proposed FAD method, we train it on the FlyingThings3D dataset and test it on three additional datasets: Middlebury, SLFD and Pov-Ray. Comparisons with three state-of-the-art DFF methods reveal that both the quantitative and visual results, presented in Table 5 and Figure 8, demonstrate that our FAD method achieves impressive performance across all tested datasets. Especially in terms of the edges of pipes in the Middlebury sample, the contours of green plants in the SLFD sample and the shapes of small leaves in the Pov-Ray sample, our FAD method shows significant advantages.

Generalization to Real Microscopic Scenes. To validate the FAD method's real-world applicability, we first trained the FAD on the synthetic Pov-Ray dataset. Subsequently, we applied 3D TFT [Yan. et al., 2023] pseudo-labeling to 100 unlabeled microscopic samples. A pre-trained bias correction model [Zaken et al., 2022] was then employed to optimize the network's bias parameters using this pseudo-labeled data, and then directly perform inference on the microscopic data. As shown in Figure 9, due to the data gap between synthetic and real data, the inference results are not ideal. However, after incorporating a small amount of pseudo-labels, the finetuned FAD is able to infer more refined depth details. Especially in Sample 1, the given label cannot capture the subtle scratches, while the FAD method perfectly restores the depth information of those scratches.

5 Conclusion

In this work, we develop a novel frequency-aware deep DFF architecture, referred to as FAD. This network introduces a new frequency domain feature extraction module and achieves a more robust DFF task through a time-frequency feature joint architecture. Experimental results indicate that the proposed FAD network not only achieves optimal performance on public DFF datasets but also generalizes quickly to real-world scenarios through fine-tuning, particularly demonstrating significant advantages in preserving fine details in depth prediction results. In the future, an interesting direction for research is how to achieve mixed training on different types of DFF datasets through multi-task learning, thereby improving the model's generalization capabilities.

Acknowledgments

This work was supported by the Major Program of National Natural Science Foundation of China (T2495250), the Key Program of the National Natural Science Foundation of China (62136005), the National Natural Science Foundation of China (62472268, 62373233, 62476160, 62441239), the Funds for central government-guided local science and technology development (YDZJSX20231C001) and the Fundamental Research Program of Shanxi Province (202403021211226).