LensNet: An End-to-End Learning Framework for Empirical Point Spread Function Modeling and Lensless Imaging Reconstruction

1 Introduction

Conventional imaging systems that rely on lenses are inherently constrained by factors such as cost, weight, and physical dimensions, which limit their potential for miniaturization [Li et al., 2024a; Guo et al., 2025a; Li et al., 2023b; Guo et al., 2024; Guo et al., 2025b; Zhou et al., 2024; Chen et al., 2023a]. In contrast, lensless imaging avoids the use of lenses altogether and instead uses computational algorithms to reconstruct scenes. This lens-free approach not only lowers production costs but also reduces device thickness, supporting the development of ultra-thin imagers. Existing methods frequently employ coded apertures [Asif et al., 2017] or phase masks [Antipa et al., 2017; Boominathan et al., 2020] near image sensors to modulate incoming light. These compact and low-cost solutions are well suited for various applications, including miniature sensors and medical imaging. Without lenses, acquired signals become highly multiplexed and require sophisticated computational techniques [Wiener, 1949] for accurate reconstruction. Central to these techniques is the Point Spread Function (PSF), which describes how the system responds to a point source and significantly influences imaging performance.

Despite its pivotal role, traditional model-based methods [Monakhova et al., 2019; Asif et al., 2015; Tanida et al.,

2001] often rely on static or calibrated PSF models, which struggle to accommodate real-world challenges such as system imperfections and dynamic scene variations. In classic lensless imaging, the PSF is typically pre-calibrated or inferred through fixed models, leading to rigid assumptions that cannot adapt to practical scenarios. Factors such as changes in lighting, sensor noise, and minor misalignments cause deviations between the assumed PSF and the true PSF, ultimately degrading render quality. Furthermore, the large support of the PSF in lensless imaging presents two major challenges. First, the expanded linear equations resulting from a broad PSF footprint are computationally demanding to store and invert, leading to significant memory and processing overhead. Second, this global multiplexing complicates reconstructions as local image details are intertwined with the global signal. This issue becomes especially detrimental under low-light or noisy settings, where the recovery of fine-scale features is critical, yet hard to achieve.

Recent advances in deep learning [Liu et al., 2023a; Zeng and Lam, 2021; Li et al., 2024b; Zhu et al., 2024b; Liu et al., 2023b; Yao et al., 2024; Pan et al., 2024; Chen et al., 2025; Chen et al., 2023b; Luo et al., 2024; Li et al., 2023c; Chen et al., 2024; Yang et al., 2025; Huo et al., 2024; Zhu et al., 2024a] combine traditional imaging physics with data-driven neural networks and have achieved substantial progress in reconstructing scenes from heavily multiplexed signals. For example, FlatNet [Khan et al., 2020] uses paired training data to improve image quality but encounters difficulties in recovering high-frequency details. Similarly, PhoCoLens [Cai et al., 2024] employs a twostage pipeline that shows promising performance, yet may blur high-frequency components during its diffusion procedure.

To address these shortcomings, we propose LensNet, an end-to-end deep learning framework that effectively fuses spatial and frequency domain information for lensless image reconstruction. Our framework adopts an encoder-decoder architecture, where the encoder features a novel Coded Mask Simulator (CMS) module that we design to capture the optical path and raster encoding properties of the lensless imaging process. By modeling the global and multi-scale characteristics of a learnable PSF, this module adapts to diverse imaging conditions without requiring explicit PSF calibration. Furthermore, we incorporate Wiener filtering [Wiener, 1949] into our network to refine reconstructions via frequencydomain deconvolution and PSF correction, thereby enhancing noise suppression and edge fidelity. By integrating these customized modules, LensNet takes advantage of both spatial and frequency domain cues to produce robust, highresolution reconstructions while maintaining global consistency, as showed in Fig. 2. Extensive experiments confirm the efficacy and reliability of our design. Our main contributions are as follows.

• We propose a novel end-to-end deep learning framework for lensless imaging that dynamically captures multiscale features in both spatial and frequency domains, substantially improving the fidelity and accuracy of image reconstructions over conventional methods.
• We introduce a state-of-the-art Coded Mask Simulator that precisely learns the true behavior of the point spread function (PSF), facilitating superior recovery performance across diverse lensless imaging scenarios.
• Through comprehensive quantitative and visual evaluations, we demonstrate that our proposed LensNet outperforms existing approaches on standard benchmark datasets.

2 RelatedWork

2.1 Traditional Lensless Imaging

Traditional lensless imaging methods commonly rely on optical encoding techniques. Coded aperture imaging [Asif et al., 2017] employs a structured mask or aperture to encode incident light, while phase retrieval [Antipa et al., 2017; Boominathan et al., 2020] reconstructs both the amplitude and phase of the light field from intensity measurements. Although these approaches have shown success in specific scenarios, they often require complex optical setups and computationally expensive reconstruction algorithms that are prone to noise and challenging imaging conditions.

In addition to encoding-based methods, iterative optimization methodologies such as FISTA [Beck and Teboulle, 2009a] and ADMM [Wang et al., 2019] are widely adopted to address inverse problems in lensless imaging. FISTA accelerates sparse signal recovery, while ADMM efficiently solves convex optimization problems by decomposing them into simpler sub-problems. Moreover, ADMMwith total variation regularization [Wahlberg et al., 2012] helps to preserve edges and reduce noise. Despite the effectiveness of these methods, they require careful parameter tuning and remain computationally expensive, particularly under noisy measurements or complex optical systems.

2.2 Learning-Based Lensless Imaging

Recent deep learning approaches [Zheng et al., 2024; Liu et al., 2024; Monakhova et al., 2019; Kingshott et al., 2022; Liu et al., 2023c] have capitalized on the representational power of neural networks to model complex forward and reverse imaging processes directly from data. Multiple studies [Cai et al., 2024; Khan et al., 2019; Li et al., 2023a] have validated the effectiveness of these learning-based methods. For example, TikNet and FlatNet [Khan et al., 2020] use paired training data to improve image quality but struggle to preserve high-frequency details. Similarly, PhoCoLens [Cai et al., 2024] employs a two-stage processing framework that achieves favorable results; however, its diffusion process blurs fine structures, requiring iterative refinements. [Rego et al., 2021] proposes a lensless imaging reconstruction method based on GAN, which achieves robust reconstruction of images captured by multiple cameras without requiring explicit PSF knowledge by jointly performing image reconstruction and PSF estimation.

Despite these advances, several challenges remain. Precise recovery of fine-scale image details, accommodating fixed or overly simplistic point spread functions (PSFs), and mitigating noise-induced degradation are significant hurdles. Traditional systems often rely on static PSF models that cannot adapt to variations in the scene or account for sensor imperfections, thus hindering image quality. Preserving highfrequency vision cues, such as sharp edges and detailed textures, remains also difficult. Our proposed imaging framework addresses these limitations by introducing a more flexible PSF modeling mechanism and improving the restoration of high-frequency details.

3 Methodology

In this section, we present a novel lensless imaging framework that jointly exploits both spatial and frequency domain information for high-fidelity image reconstruction. To address the limitations of existing lensless imaging systems, our proposed framework integrates multiple advanced modules specifically designed to overcome inefficiencies and inaccuracies commonly encountered in PSF-based approaches. The remainder of this section describes the overarching network architecture and its constituent modules in detail.

3.1 Preliminary

In optics, the Point Spread Function (PSF), denoted as PSF(x, y), characterizes how an imaging system distributes a point source of light. Technically, PSF(x, y) is a twodimensional or three-dimensional function representing the output signal when a point source is introduced at the input. Coded masks, widely employed as optical encoders, are constructed with specific patterns (e.g., horizontal lines, vertical lines, grids, or random structures) to modulate incident light. By imposing spatial modulation on the optical signal, these masks effectively encode essential information about the scene. In lensless imaging systems, the particular arrangement of the coded mask plays a pivotal role in the definition of PSF(x, y), thereby influencing the quality and fidelity of the reconstructed images. Hence, the design of the coded mask significantly affects the imaging performance of the system, determining how light is distributed and shaping critical attributes such as resolution, contrast, and image clarity.

The forward lensless imaging model can be mathematically represented as a linear transformation [Goodman, 2005]. For an input image, the sensor in a lensless system acquires a blurred observation:

where Iobject(x, y) is the original image, PSF(x, y) denotes the system's point spread function, noise(x, y) is the sensor noise, and Imeasurement(x, y) is the recorded measurement.

By introducing spatial encoding through coded masks, lensless systems modulate the incoming light to create distinctive intensity distributions on the detector. Physically, the coded mask pattern directly affects PSF(x, y), which emerges in the measurement image via the convolution operation:

Hence, the mask pattern governs the distribution of spatial frequencies, the direction of light propagation, and the overall optical properties, all of which are intrinsically manifested in Imeasurement(x, y). Consequently, the choice of coded mask has a profound influence on both the image formation process and the achievable reconstruction quality.

3.2 Overall Architecture

The overall architecture of LensNet is illustrated in Fig. 3. The framework adopts an encoder-decoder structure, where the encoder processes multi-resolution spatial and frequency domain features in the first stage, and the decoder reconstructs the image in the second stage. We begin by outlining the LensNet framework and then provide a detailed description of its individual components.

Given the measurement image Imeasurement(x, y) measured by the sensor in a lensless imaging system, we simplify the analysis by momentarily assuming a noise-free condition. The goal of LensNet is to reconstruct Iobject(x, y) from Imeasurement(x, y) while satisfying the following equation:

In the encoder stage, we incorporate a Spatial Compression Module (SCM), which leverages Reconstruction Blocks (ReCB) as its primary feature transformation mechanism and employs gating strategies to enhance feature fusion. First, Imeasurement(x, y) is processed on multiple scales, extracting spatial context at varying resolutions. These features establish the foundation for reconstructing the spatial representation of the object.

To account for the coded mask distribution associated with the system PSF, we introduce a learnable Coded Mask Simulator (CMS). This module applies a dynamic channel attention mechanism to modulate light intensity across different regions, effectively modeling the coded mask's influence on the system's PSF. By learning the relationship between spatial inputs and the PSF, the CMS dynamically adapts to changing imaging conditions, improving the simulation of light intensity patterns. Its output, a distribution map that encodes highlevel semantic cues, facilitates more accurate reconstruction of structures, especially in complex or dynamic scenarios where traditional fixed PSF methods often fail.

We further extract frequency domain features at each scale via the Wiener Fusion Block (WNFB). This block performs a Fast Fourier Transform (FFT) on both the spatial features and the output of the Coded Mask Simulator, and then computes the Wiener filter transfer function:

where H(u, v) denotes the Wiener filter in the frequency domain, PSF(u, v) is the complex conjugate of the PSF, and | PSF(u, v)|2 is the energy spectrum of the PSF.

Using this transfer function, the encoded frequencydomain signal B(u, v) is preliminarily restored:

In the decoder stage, we propose the Spatial Amplification Module (SAM) to integrate spatial and frequency domain information more comprehensively. The SAM upsamples lowresolution feature maps from deeper layers via bilinear interpolation and aligns them (through zero-padding) with higherresolution encoder features. These feature maps are then concatenated to preserve both fine-grained detail and broader contextual cues. Serving as a multi-scale fusion mechanism, the SAM leverages information from both domains to produce high-quality reconstructions, ensuring that critical spatial details and global consistency are well-preserved.

3.3 CodedMask Simulator

As discussed in Sec. 3.1, the physical level PSF(x, y) corresponds to the coded mask placed within the sensor. Traditional methods commonly adopt static or fixed kernels to approximate PSF(x, y). In contrast, we seek a more flexible approach that adaptively learns these coded patterns, resulting in a learnable PSF and minimizing constraints seen in conventional models.

To address this need, we propose the Coded Mask Simulator (CMS). By employing a channel attention mechanism, the CMS captures the intensity distribution features in Imeasurement(x, y), which encode the coded mask pattern. Consequently, the system's PSF(x, y) can be inferred from these learned mask distributions. This attention mechanism models directional inter-pixel relationships, enabling the network to deconvolve scattering and diffraction artifacts induced by the mask. More concretely, it identifies directional modulations in the image and adaptively recalibrates features according to attention weights, achieving a more robust partial or full recovery of occluded signals.

Formally, let X ∈ RN×C×H×W be the input feature map. We first apply global average pooling to aggregate spatial information, preserving only channel-level statistics:

Subsequently, a pointwise convolution adjusts the resulting channel weights. This operation, which is more efficient than deeper multilayer perceptrons, preserves channelspecific modulation dynamics:

where W ∈ RC′×C′ (with C ′ = C2 ) is the 1 × 1 convolution weight matrix, σ is a (linear) activation function, and sc ∈ RN×C′×1×1 are the channel attention weights.

In summary, the proposed Coded Mask Simulator models the physical coded mask by learning its intensity distribution and directional modulation patterns from the measurement image. This approach implicitly recovers PSF(x, y) and provides valuable semantic cues that guide subsequent reconstruction steps. The learned distribution fosters flexible, adaptive, and comprehensive recovery of scene information, overcoming the drawbacks associated with traditional fixed PSF kernels.

3.4 Objective Function

We train LensNet by jointly minimizing three complementary loss terms: the mean squared error (MSE), the structural similarity index measure (SSIM) and the perceptual image patch similarity (LPIPS). Formally, we define the overall loss function as:

where each loss provides a different perspective on reconstruction fidelity.

1. Mean Squared Error (MSE)

where ̂Ii denotes the output image, and Ii represents the corresponding ground truth. By quantifying pixel-wise intensity differences, MSE imposes strong penalties for large deviations and steadily guides the network toward minimizing low-level reconstruction errors.

2. Structural Similarity Index Measure (SSIM)

This term evaluates perceptual quality by examining local image structures. SSIM is particularly effective for preserving global consistency and fine structural details, ensuring that the reconstructed images align with human perception of image integrity.

3. Learned Perceptual Image Patch Similarity (LPIPS)

LPIPS leverages deep neural network features to evaluate perceptual similarity. Unlike MSE or SSIM, which target lower-level and local structure information, LPIPS operates on higher-level representations, promoting reconstructions that faithfully capture the semantic and perceptual nuances of the scene.

By combining these three loss components, our objective function balances low-level fidelity, structural coherence, and perceptual plausibility. Consequently, this strategy leads to more accurate and visually coherent reconstructions in lensless imaging.

4 Experiments

4.1 Dataset and Preprocessing

The DiffuserCam dataset [Antipa et al., 2017; Monakhova et al., 2019] comprises 25,000 pairs of images captured simultaneously by a standard lensed camera (serving as ground truth) and a diffuser-based lensless camera. The raw images, originally at 1080 × 1920 pixels, are resized to 480 × 270 pixels according to previous methods [Khan et al., 2020; Kingshott et al., 2022; Li et al., 2023a; Rego et al., 2021]. Of these 25,000 paired images, 24,000 are allocated for training and 1,000 for testing.

Similarly, the MWDNs dataset [Li et al., 2023a] consists of 25,000 images displayed on a 27-inch Dell monitor, sourced from the Dogs vs. Cats dataset on Kaggle. Original 1280 × 1280 images undergo the official preprocessing steps (zero-padding and resizing) to obtain 320 × 320 pixels. This standard preprocessing pipeline ensures consistent input dimensions across both datasets, facilitating robust and fair comparisons of lensless imaging strategies.

4.2 Implementation Details

We implemented our method in PyTorch and conducted experiments on 2 NVIDIA RTX 3090 GPUs running Ubuntu. All images were used at their preprocessed resolutions without further downsampling. We set the batch size to 16 for both datasets and applied standard data augmentation strategies, such as random rotation and flipping. For optimization, we employed the Adam optimizer.

4.3 EvaluationMetrics

We evaluated the performance of our lensless imaging approach using three widely adopted image quality metrics: PSNR, SSIM, and LPIPS. PSNR quantifies the level of noise reduction, while SSIM measures the preservation of structural and textural fidelity. LPIPS also assesses perceptual similarity by comparing reconstructed images with the ground truth according to human visual perception. Higher PSNR and SSIM values indicate better image quality and structural integrity, while lower LPIPS values suggest improved perceptual fidelity and preservation of fine details.

4.4 Comparisons with State-of-the-art Methods

In this section, we compare our proposed LensNet with traditional and learning-based methods to assess reconstruction fidelity, robustness, and overall performance in the DiffuserCam and MWDNs datasets.

Quantitative Comparisons

As summarized in Table 1, traditional algorithms such as Wiener, FISTA, ADMM, and APGD produce relatively low numerical performance when applied to highly multiplexed signals observed in lensless imaging. For instance, in the DiffuserCam dataset, these methods yield PSNR values typically between 7.33 dB and 13.27 dB, indicating their limited ability to invert complex degradations. Similarly, iterative optimization algorithms (Vanilla Gradient Descent and Nesterov Gradient Descent) exhibit difficulties in converging to high-quality solutions, particularly in low-light or noisy conditions.

On the other hand, learning-based approaches demonstrate noticeable improvements in reconstruction quality. FlatNet [Khan et al., 2020] achieves PSNR values of 21.16 dB (DiffuserCam) and 19.08 dB (MWDN). Meanwhile, UDN [Kingshott et al., 2022] and TikNet [Khan et al., 2020] tend to preserve structural signals better than traditional methods, with UDN achieving 26.98 dB in PSNR on MWDNs. MWDN [Li et al., 2023a], which uses a multi-scale wavelet decomposition strategy, further improves both PSNR and SSIM.

However, none of the previous methods dominate in every metric. Although UDN excels at preserving edges, it can still miss critical high-frequency texture in challenging illumination patterns. Similarly, though MWDN shows promising fine detail recovery, its susceptibility to high noise levels sometimes results in artifacts or mild oversmoothing under extreme conditions. In contrast, our proposed LensNet consistently outperforms all competing methods across the board, establishing new state-of-the-art results (PSNR: 27.46 dB in DiffuserCam and 33.22 dB in MWDN) alongside substantially lower LPIPS values. And Fig. 1 shows more intuitively.

Qualitative Comparisons

Beyond numerical metrics, we conduct an in-depth qualitative assessment to illustrate how LensNet handles subtle textures, fine details, and complex lighting conditions. Figure 4 presents representative reconstructions from the MWDNs datasets, comparing our method with both traditional and CNN-based baselines, with extended comparisons provided in the Supplementary Material.

As shown in Fig. 4, APGD and other traditional solvers often exhibit visibly blurred regions with pronounced ringing artifacts. FlatNet [Khan et al., 2020] significantly reduces these artifacts, but introduces oversmoothing, obscuring key high-frequency details such as hair strands or fine-textured fabric. LenslessGAN [Rego et al., 2021] improves the realism of certain regions; however, it can occasionally generate hallucinated details or struggle with complex boundaries under minimal SNR settings.

By contrast, LensNet consistently recovers sharper edges and more faithful textures, demonstrating well-preserved contours around objects and a closer match to the true intensity profile. As highlighted in the red and green bounding boxes of Fig. 4, LensNet excels at reconstructing intricate structures like an animal's fur or subtle shading gradients in natural scenes.

4.5 Ablation Study

As shown in Table 2, we conducted a detailed ablation study to verify the contributions of different components in our network. A reduced model with only three downsampling layers (ThreeDown) suffers from inadequate reconstruction quality, leading to notable losses in structural and textural details. When the ReconstructionBlock is removed (w/o RecB), the model still fails to preserve crucial local textures necessary for high-fidelity image reconstruction. Even when using the original point spread function (PSF) method, the performance did not exceed that of our learnable PSF design.

Illustrated in Fig. 5, the visual results underscore the importance of each architectural component. The ThreeDown variant fails to produce structurally coherent images, while omitting RecB degrades local texture preservation. In contrast, our full model-incorporating both the ReconstructionBlock and the learnable PSF-delivers results that are more faithful to the target images, retaining finer details and exhibiting overall higher visual quality. These observations confirm that each proposed module is crucial for achieving highaccuracy lensless imaging reconstructions.

5 Conclusion

In this study, we present LensNet, a novel deep learning framework designed for lensless imaging, which effectively integrates features in both the spatial and frequency domain to address the inherent challenges of lensless image reconstruction. By incorporating a learnable Coded Mask Simulator, LensNet dynamically models optical pathways and the Point Spread Function, enabling the precise and efficient reconstruction of complex scene details, including high-frequency components and fine textures. Our extensive experiments demonstrate that LensNet outperforms state-of-the-art methods, achieving superior image quality across multiple benchmark datasets. Moreover, its robustness in handling noise and variations in the PSF positions LensNet as a promising solution for practical applications in fields such as miniature sensors and medical diagnostics. This work not only enhances the capabilities of lensless imaging systems but also establishes a new standard for image reconstruction in lens-free environments.

Ethical Statement

There are no ethical issues.

Acknowledgments

This work was supported by the Science and Technology Development Fund, Macau SAR, under Grant 0141/2023/RIA2 and 0193/2023/RIA3.

Contribution Statement

This work was a collaborative effort by all contributing authors. Xuhang Chen, serving as the corresponding author, is responsible for all communications related to this manuscript.