A Dataset and Model for Realistic License Plate Deblurring

1 Introduction

Efficient recognition of vehicle license plates is crucial for intelligent traffic management systems, however real-world scenarios often pose a significant challenge due to motion blur. This blur, making license plates unreadable, is especially problematic in situations involving high-speed vehicles or low-light conditions. Such issues are exacerbated during nighttime or in bad weather, resulting in considerable motion blur in captured images. To tackle these issues, our study introduces a comprehensive dataset and a novel model tailored for realistic license plate deblurring.

∗Corresponding author

Image deblurring is a key task in computer vision, focused on restoring blurred images to clear ones for accurate observation and identification. The progress in this field heavily depends on the development of relevant datasets. Current methods for creating deblurring datasets fall into three main categories: (1) synthetic blurring using blur kernels [Sun et al., 2013; K¨ohler et al., 2012; Lai et al., 2016], which leads to a lack of generalization capability for models trained on these synthesized images when applied to real-world images. (2) The generation of blurred images from sharp frames via averaging or fusion [Nah et al., 2017; Shen et al., 2019; Jiang et al., 2020], which doesn't fully mimic real-world overexposure outliers [Chang et al., 2021]. (3) Lastly, beamsplitting systems capture sharp and blurred image pairs via camera shake [Rim et al., 2020], with potential issues in color accuracy and alignment. Each approach contributes to the field but also has inherent limitations impacting the realism and utility of the datasets.

With the advent of deep learning, numerous convolutional neural network (CNN)-based methods have surfaced [Sun et al., 2015; Gong et al., 2017; Tao et al., 2018; Shen et al., 2019; Zhang et al., 2023], playing an essential role in the motion deblurring task. Recently, the proposal of Generative Adversarial Networks (GAN) has also profoundly impacted the image deblurring field [Ramakrishnan et al., 2017; Kupyn et al., 2018; Kupyn et al., 2019; Zhao et al., 2022]. Despite these advancements, deblurring license plate images remains a significant challenge, primarily due to the lack of large-scale, tailored datasets. The complexity of license plate blurring, with its more severe degradation compared to standard motion blur, poses an additional challenge. To better justify the performance of existing image deblurring algorithms on real-world blurred license plate images, we evaluate the performance of several state-of-the-art deblurring algorithms with blurred license plate images. As shown in Figure 1, we can conclude that all these methods fail to performwell in this task. It underscores the necessity for further research specifically targeting real-world vehicle license plate deblurring.

To address these challenges, we present a comprehensive solution consisting of a large-scale paired license plate dataset and a dedicated license plate deblurring model. Our data collection employs a dual-camera setup with different shutter speeds to capture sharp and blurred images simultaneously, eliminating color deviations and enabling post-processing alignment. Our innovative end-to-end model leverages an encoder and latent fusion module for handling multi-scale latent codes, featuring the Swin transformer block ([Liu et al., 2021]) for effective long-range modeling. To enhance letter reconstruction and text legibility, we introduce a partition discriminator assessing per-letter sharpness. Extensive experiments using our LPBlur dataset, including metrics such as L1 loss, Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Perceptual Loss (PerL), and Text Levenshtein Distance (TLD) [Levenshtein and others, 1966], validate its suitability for training and testing. Our proposed model outperforms state-of-the-art motion deblurring methods in real-world license plate deblurring scenarios.

In summary, our main contributions are as follows:

• We present a real-world sharp-blurred license plate dataset, named LPBlur. This dataset consists of 10,288 paired images, meticulously collected under diverse real-road scenarios using our designed dual-camera system, and corrected by a post-processing pipeline.
• We introduce a novel LPDGAN, a license plate deblurring model that leverages multi-scale latent codes as references. It incorporates both a partition discriminator and text reconstruction techniques, which enhance the model's capability to generate high-quality license plate images through spatial architecture and textual information, respectively.
• Extensive experiments demonstrate that our dataset LPBlur is highly effective for model training and evaluation. Compared to other state-of-the-art (SOTA) deblurring models, our proposed LPDGAN can achieve 21.24% license plate recognition accuracy improvement.

2 RelatedWork

2.1 Image Deblurring Datasets

Image deblurring relies on paired sharp-blur image datasets. Traditionally, blurred images are generated by convolving sharp images with uniform or non-uniform blur kernels [Levin et al., 2009; Sun et al., 2013; K¨ohler et al., 2012; Lai et al., 2016]. Consequently, some researchers attempt to capture sequences of sharp frames while vibrating the camera, averaging or fusing such sequences of frames into corresponding motion-blurred images [Nah et al., 2017; Shen et al., 2019; Jiang et al., 2020; Noroozi et al., 2017]. HIDE dataset [Shen et al., 2019] is created by averaging 11 consecutive frames, with the central frame serving as the sharp image. The same strategy is employed in the collection of the Blur-DVS dataset [Jiang et al., 2020] and MSCNN (WILD) dataset [Noroozi et al., 2017]. However, models lack generalizability to real-world scenarios when they are trained on such synthetic datasets generated using the aforementioned methods. Recently, certain researchers gathered authentic pairs of sharp-blur images employing beam-splitting systems. They position two cameras at a fixed angle to ensure that both images share the same visual field, as described in works by Rim et al. and Li et al.. However, this approach can lead to color cast discrepancies in the paired images due to inherent issues with beam-splitting systems.

2.2 Blind Deblurring

The majority of conventional approaches employ priors of natural images to estimate latent images or blur kernels [Fergus et al., 2006; Shan et al., 2008; Cho and Lee, 2009; Ren et al., 2018; Whyte et al., 2012]. However, the aforementioned techniques have certain limitations by predicating upon the assumption of uniform image blur. To address this issue, some methods [Ren et al., 2017; Hyun Kim et al., 2013] estimate blur kernels at a pixel level, thereby accommodating more complex blurring situations.

With the advent of deep learning technologies, significant strides have been made in image deblurring, applying deep learning to predict blur kernels or latent images to procure clear images. In the work of [Sun et al., 2015], a method based on CNNs is proposed to predict the probability distribution of block-level motion blur. Gong et al. introduces a method to directly estimate the motion flow of blurred images, recovering non-blurred images from the estimated motion flow. MIMO-UNet [Cho et al., 2021] deploys a multiinput-multi-output single Unet network to simulate multilevel Unet for noise reduction across various image scales. MSSNet [Kim et al., 2022] enhances deblurring network performance by using a stage configuration reflecting blur scales, an inter-scale information propagation scheme, and a pixelshuffle-based multi-scale scheme. XYDeblur [Ji et al., 2022] further augments network efficiency and deblurring performance by employing rotated and shared kernels within the decoder.

2.3 GAN-Based Deblurring

In recent years, following the inception of GANs, their application in the domain of image deblurring has achieved remarkable success. DeblurGAN [Kupyn et al., 2018] first presents an end-to-end learning method for motion deblurring, and also introduces a new method for blur generation. DeblurGAN-v2 [Kupyn et al., 2019] introduces a dual-scale discriminator based on a relative conditional GAN framework and incorporates a feature pyramid into the deblurring process, which permits the flexible substitution of the backbone network. MSG-GAN [Karnewar and Wang, 2020] addresses the issue of insufficient overlap between the true and false support distributions during the transfer from discriminator to generator in GANs by allowing multi-scale gradient networks from the discriminator to the generator. FCL-GAN [Zhao et al., 2022] designs a lightweight domain conversion unit (LDCU) and a parameter-free frequency-domain contrastive unit (PFCU) for lightweight property and performance superiority. The aforementioned methods can handle standard blurred images, but they struggle to deliver satisfactory results on license plate blurring with very severe degradation. We propose an end-to-end generative model that accommodates multi-scale inputs and outputs, employing several novel modules to accomplish the license plate deblurring task better.

3 Proposed LPBlur Dataset

3.1 Data Collection

Causes of Motion Blur. Motion blur refers to the perceptible streaking effect observed when capturing the movement of objects. In the capturing process, the correlation between the amount of light entering the photosensitive component and the camera's basic parameters satisfies the following equation:

where La denotes the number of photons received by the camera, SL represents the light intensity of the scene, ISO represents the camera ISO value, Et is the exposure time, Ap denotes the camera aperture size. Cameras adjust these parameters automatically within limits depending on the lighting situation. For example, cameras increase their aperture and exposure time in low-lighting settings to capture enough light. Fast-moving objects leave trajectories within a single frame during this extended exposure time, resulting in a motion blur effect.

Paired Image Acquisition. To collect paired sharp-blur images, we use two identical scientific cameras that are set with different exposure times. As shown in Figure 2a, cameras S and L are fixed parallel on a tripod to maintain horizontally to the ground. Specifically, Camera S is set by an extremely short exposure time Ets, employed for the collection of sharp images, while Camera L is set by a relatively longer exposure time Etl for the acquisition of blurred images. Both of these cameras are interfaced with a computer via a synchronizer, which ensures the synchronization of the start of exposures, and both cameras capture the same scene.

Scenes are taken in a variety of locations, including above, on the right side, and on the left side of roadways, to guarantee the dataset's diversity. Also, depending on the road and the illumination conditions, we dynamically modulate camera exposure time according to the subsequent equation:

where v denotes vehicular velocity, Et represents exposure time, b represents pixels blurred, D is the distance between vehicle and camera, f denotes camera focal length, and p is pixel edge length on the sensor. Given that the actual speeds of individual vehicles are indeterminable, we standardize image captures on high-speed road sections with a regulatory speed limit of 70 km/h to ensure that D is remained within a range of 10-20 meters.

Moreover, to ensure equality in exposure between two cameras, we make their exposure times and ISO values to satisfy the following equation:

where ISOs is the ISO value for Camera S and ISOl Camera L. However, variations in ISO values can cause changes in image noise levels, in post-processing, we incorporate a denoising step for sharp images.

3.2 Data Post-processing

As shown in Figure 2b, the paired image post-processing includes denoising, geometrical alignment, and license plate cropping.

Denoising. Due to the disparate ISO settings of the two cameras, Camera S and Camera L capture images with a different noise level. Consequently, during the conversion of RAW images to RGB format, wavelet denoising [Liu et al., 2020] is employed after white-balancing, color mapping, and gamma correction.

Geometrical Alignment. Cameras S and L capture sharp and blurred image pairs with slight horizontal misalignment even though they are closely aligned left-to-right to minimize the difference. To align these image pairs perfectly, we first take a static image pair without any moving vehicles as the reference pair for each scene. Then, the Enhanced Correlation Coefficient Maximization [Evangelidis and Psarakis, 2008] is adopted to estimate the geometric transformation between the sharp and blur of the reference image pair. Finally, the estimated geometric transformation is applied to the image pairs in the same scene.

License Plate Cropping. A pre-trained YOLO v5 [Jocher, 2020] and a pre-trained CRNN [Shi et al., 2016] model are facilitated the detection and recognition of license plates under standard conditions, both models are pre-trained on the CCPD [Xu et al., 2018] license plate dataset. Following the geometrical alignment, the pre-trained YOLO v5 and CRNN detect and recognize the bounding box of each sharp plate in the paired images, both the sharp and blurred images are then cropped using the same detected coordinates.

In conclusion, we collect 10,288 image pairs, with an original image size of 1920×1220. Post-processing crops image size to 224 × 112 with blur size ranging from 20-50 pixels. Among them, 5672 pairs are captured under normal light conditions and 4616 pairs under low light conditions, including 1,000 pairs under rainy environmental conditions. For more information, please refer to the released dataset on the GitHub repository.

4 Method

Overview. The goal of our work is to improve the clarity of license plate images using a meticulously crafted image-toimage translation framework, called LPDGAN. As depicted in Figure 3, our approach first constructs a multi-scale feature extraction and fusion module designed to encode input blurred images effectively. Subsequently, an image decoder is employed to generate sharp and high-quality images. To further enhance the overall image quality, we integrate both a global discriminator and a partition discriminator for adversarial training. Additionally, we incorporate a text reconstruction module to enrich the semantic information embedded in the generated license plate images.

4.1 Multi-scale Feature Extraction and Latent Fusion Module

Feature Extraction. In real-world scenarios, license plate images affected by motion blur often exhibit intricate degradations, including noise, low resolution, and ghosting effects. Our feature encoder, denoted as E, is specifically used to address these degradations, extracting essential image features for subsequent processing. In particular, the Swin transformer block [Liu et al., 2021] is selected for its ability to capture global information through self-attention mechanisms. This is crucial to resolve the elongated ghosting artifacts that often appear in motion-blurred license plate images. To address variations in license plate image sizes due to differing capture distances, our approach employs a multi-scale feature extraction strategy, which is illustrated in Figure 3. This approach facilitates the encoding of features at each scale, which are represented as E(xi) for i = 1, 2, 3.

Latent Fusion. Based on the Spatial Feature Transform (SFT) [Wang et al., 2018], we further propose a Latent Fusion Module F (see Figure 3). This module is designed based on an affine transformation to effectively integrate the obtained multi-scale features. The corresponding formulas are provided below:

where E(xi+1)sp1 and E(xi+1)sp2 represent the two parts into which E(xi+1) is divided along the channel dimension. Specifically, for the fusion of E(x1) and E(x2), we first split E(x2) along the channel dimension. Each part is then processed through a series of convolutional layers to derive the fusion parameters α and β. These parameters are employed to modulate E(x1) through scaling and shifting operations.

Moreover, we reintegrate the original E(x2) using a skip connection, which is then combined with the modified E(x1) along the channel dimension. This fusion process is also applied between E(x2) and E(x3).

4.2 Decoder and Discriminator

Decoder. As depicted in Figure 3, our decoder D is composed of a sequence of Swin transformer blocks and patch expanding operations. Similar to the encoder, our decoder is designed to generate sharp images in a multi-scale fashion, and the output images are denoted as ˜ y1, ˜ y2 and ˜ y3 accordingly.

Discriminator. We design two discriminators: 1) the Global Discriminator (Dg) enhances overall spatial and color information in the restored images; 2) the Partition Discriminator (Dp) focuses specifically on refining character information by examining n randomly selected partitions of letters within the license plate image. The structure of Dp is shown in Figure 4. It identifies and locates letter positions in both the real image y and the fake image ˜ y. Following this, n partition images are randomly chosen for evaluation by Partition Discriminator. In the early stages of training, when our mode's capacity to produce sharp images is still developing, the generated image might not be recognized with high accuracy. To address this, an average partitioning approach is applied to both y and ˜ y, initially setting n to 7. As training progresses, a pre-trained YOLO v5 model is used for precise letter detection and the number of partitions n is set to 3. In our experiment, we employ WGAN-GP to train our model. In particular, the adversarial loss for the Global Discriminator can be formulated as follows:

Similarly, for the Partition Discriminator Dp, the formulation is as follows:

where P is the partition operation, λgp1 and λgp2 are the weighting parameters used to control the gradient penalty.

Note that we apply both discriminators to the outputs at three different scales.

In addition to the adversarial loss, we also incorporate reconstruction loss Lrec, defined as follows:

where the ψvgg represents a pre-trained VGG-19 network [Simonyan and Zisserman, 2014], from which we use feature maps from the 8th , 15th , and 22nd ReLU layers to compare shallow textures and deep features between real and generated images.

4.3 Text ReconstructionModule

We also incorporate a Text Reconstruction Module T specifically designed to enhance our model's ability to accurately interpret characters on license plate images. T merges the Decoder's intermediate feature FD with the fusion latent code F2,3 along the channel dimension. This combined feature then traverses a series of convolutional and linear layers, resulting in a vector that represents the recognized text. Concurrently, a pre-trained CRNN model extracts ground-truth text vectors from real images. We calculate the L1 loss between the output vector from our Text Reconstruction Module T and the ground-truth text vector. The loss is defined as:

where FD is the feature maps obtained from the middle layers of Decoder, ψcrnn represents the pre-trained CRNN model.

4.4 Fully Objective

Our full objective is

where λg, λp and λt control the relative importance of the different objectives. We aim to solve:

5 Experiment

5.1 Experimental Setups

Dataset setup. The proposed LPBlur dataset is partitioned into a training set with 9,288 image pairs, and a test set with 1,000 image pairs. The test set encompasses 500 pairs acquired under normal light conditions and another 500 pairs captured in low light conditions.

Evaluation metrics. To evaluate the image quality of deblurred images, we adopt three evaluation metrics: PSNR, SSIM, and Perceptual Loss (PerL). The Perceptual Loss is specifically defined by comparing the generated images with the ground truth images at the output feature maps of sequential layers 8, 15, and 22 of a pre-trained VGG-19 model. To assess the recognisability of the generated license plate images, we calculate the Text Levenshtein Distance (TLD) [Levenshtein and others, 1966] between the detected text of the generated images and the real images.

Implementation details. The shape of multi-scale input images for LPDGAN are (112, 224, 3), (56, 112, 3), and (28, 56, 3) respectively. Random rain adding and random cutout are utilized for data augmentation. The optimizer we use is Adam [Kingma and Ba, 2014]. The batch size is set to 7. The initial learning rate is 10-4 , and the linear weight decay is used after the 100th epoch. All experiments are conducted on a GeForce RTX 3090 GPU.

5.2 Deblur Results

To evaluate the deblur performance of our method, we compare LPDGAN with five SOTA methods: Pix2Pix [Isola et al., 2017], DeblurGAN v2 [Kupyn et al., 2019], MIMO-Unet [Cho et al., 2021], MSSNet [Kim et al., 2022], and LBAG [Li et al., 2023] on LPBlur.

From the results presented in Table 1, it can be observed that our LPDGAN outperforms all other models in both normal and low light conditions. In normal light conditions, our LPDGAN achieves a PerL of 3.31, PSNR of 29.95, and SSIM of 0.795, which are superior to the latest deblurring method LBAG. The performance gap becomes even more pronounced when dealing with low light images, with our model exhibiting improvements of 29.8%, 4.5%, and 3.7% in PerL, PSNR, and SSIM, respectively, compared to LBAG and MSSNet.

Figure 5 provides a visual comparison between sets of blurred and deblurred images under two light conditions. In the case of normal light, our LPDGAN effectively restores license plates afflicted with severe motion blur, accurately generating and reconstructing characters such as 'D', 'O' and 'B', as well as numbers like '7', '1', and 'L', which often pose challenges for other models, as shown in the 1st, 3rd and 7th column of Figure 5. In low light scenarios, where license plates are barely visible to the human eye, our model excels in generating details that significantly surpass the performance of other models. This highlights the inadequacy of models designed for minor blurs in large scenes when applied to the deblurring of license plates, which are subject to more severe blurring.

5.3 Text Recognition Results

We further evaluate the plate text recognition accuracy based on those deblurred images. The CRNN [Shi et al., 2016] is incorporated for the recognition of generated and sharp license plate characters. The 4th and 8th columns in Table 1 compare the TLD between the generated images and original sharp images. In the context of normal light conditions, LPDGAN exhibits comparable performance to LBAG in terms of TLD but surpasses all other models. Under low light conditions, LPDGAN is the only model with a TLD lower than 1. This implies that a pre-trained CRNN model, when employed to recognize deblurred license plate images produced by LPDGAN, will obtain an average error in less than one character per instance. Consequently, LPDGAN has the best performance overall, demonstrating robust capability in deblurring license plates across two scenarios.

5.4 Ablation Study

To evaluate the effectiveness of each proposed module, a series of ablation experiments are performed, which is shown in Table 3. The omission of the Latent Fusion Module leads to a decline in global metrics, underscoring its effectiveness in fusing multi-scale features and improving the model's performance in restoring sharp images. Removing the Text Reconstruction Module results in a significant downturn in global metrics, particularly noticeable under low light conditions. This highlights the pivotal role of the Text Reconstruction Module in enabling the model to have a deeper understanding and restoration capability for license plates affected by severe pixel disruption. Similarly, the exclusion of the Partition Discriminator Module deteriorates global metrics and notably affects the SSIMmetric to a greater extent. This confirms the module's contribution to enhancing the model focus on the details of each letter on the license plate.

Low Light

5.5 Necessity of LPBlur

We further demonstrate the importance of introducing a dataset consisting of real blurred images for the task of license plate deblurring. To assess this, we employ different blur kernels randomly to the sharp images in LPBlur dataset and finally create a synthetic dataset. The result samples, illustrated in Figure 6 and summarized in Table 2, clearly indicate that LPDGAN trained on the synthetic dataset fails to eliminate real-world license plate blur effectively. This is evident both visually and in the metric evaluations, showcasing poorer performance compared to when trained on the LPBlur dataset.

The aforementioned outcomes highlight a significant disparity between synthetic and real-world license plate blur, emphasizing that synthetic blurred image data cannot serve as a substitute for the LPBlur dataset. Thus, the LPBlur dataset proves to be more effective in training models for real-world license plate deblurring.

6 Conclusion

In this paper, we study the issue of motion license plates deblurring. We introduce the first large-scale license plate deblurring dataset for this research and address color bias and misalignment problems through appropriate data collection methods and post-processing. Furthermore, given that the degree of blur caused by vehicular motion substantially exceeds that induced by camera shake, we propose a model based on multi-scale input and output for license plate deblurring. This includes a latent fusion module, a supervision module for textual modality information, and a partition discriminator module. Experimental results indicate that our model performs favorably in comparison to current state-of-the-art deblurring algorithms. In the future, we intend to augment our dataset with license plates from a broader range of countries and regions to enhance its diversity. Regarding the model, we intend to incorporate modules that ensure the restoration capability for spatially complex characters, such as Chinese characters.

Ethical Statement

To prevent the disclosure of personal privacy, all private information, including human faces and surrounding scenery, is removed from the images in the LPBlur, and only the license plate number is retained. In addition, sensitive metadata in the image is removed, including GPS location, timestamp, etc.

Acknowledgments

This work was jointly supported by the National Natural Science Foundation of China (62201474 and 62206180), Suzhou Science and Technology Development Planning Programme (Grant No.ZXL2023171) and XJTLU Research Development Funds (RDF-21-02-084, RDF-22-01-129, RDF-22-01-134, and RDF-23-01-053).