Towards Robust Scene Text Image Super-resolution via Explicit Location Enhancement

1 Introduction

Scene text recognition (STR) is an important computer vision task and has a wide range of applications [Liem et al., 2018; Khare et al., 2019]. Despite impressive progress made, current STR methods are still struggling with low-resolution (LR) images [Wang et al., 2020]. Several approaches [Dong et al., 2015a; Tran and Ho-Phuoc, 2019] process LR input by treating text images as natural images and employing a generic super-resolution network to obtain high-resolution ones. However, as shown in previous studies [Chen et al., 2021; Chen et al., 2022], this scheme is not that satisfactory. Therefore, customizing super-resolution networks for scene text images has become a popular research topic.

∗Corresponding author: Tao Dai

This work is supported in part by the National Key Research and Development Program of China under Grant 2022YFF1202104, the National Natural Science Foundation of China under Grant 62171248, Shenzhen Science and Technology Program (Grant No. JCYJ20220818101014030, JCYJ20220818101012025), the PCNL KEY project (PCL2021A07), and Research Center for Computer Network (Shenzhen) Ministry of Education.

To improve the quality of LR images, many scene text image super-resolution (STISR) approaches have recently been proposed with promising results. For example, a locationaware loss function is proposed in [Chen et al., 2021] to consider character spatial distribution. By applying the character probability distribution, TPGSR [Ma et al., 2021] demonstrates the importance of using language knowledge as guidance in the STISR task. To handle spatially irregular text, TATT is proposed in [Ma et al., 2022]. Moreover, C3-STISR [Zhao et al., 2022] achieves favorable performance by using three perspectives of clues.

Despite many efforts, existing STISR methods typically treat the character regions and non-character background equally in their forward process while neglecting the adverse impact of the complex background. Intuitively, the noncharacter background is usually uninformative for the downstream recognition task, so it is unnecessary to reconstruct texture details of the background. Moreover, the complex background can bring disturbance to the reconstruction process. On one hand, the background may be mistakenly treated as characters, thus producing incorrect reconstruction (see Fig. 1 (a)). On the other hand, the background may prevent the model from accurately locating characters, thus leading to poor reconstruction results (see Fig. 1 (b)). Therefore, existing STISR methods usually suffer from performance degradation due to the complex background, thus limiting practical applications.

To address these issues, we propose LEMMA, a Location Enhanced Multi-ModAl network, to treat character regions and non-character background distinctly. Specifically, we introduce a novel Location Enhancement Module (LEM) to exploit the character location information in the attention map generated by the text recognizer. However, adopting the attention map for character region localization is nontrivial, because the low quality caused by the attention drift [Cheng et al., 2017] can produce wrong guidance. To this end, we propose the Compression and Expansion strategy to process the raw attention map and thus mitigate the attention drift. We then further select character region features using the feature selection technique, which mitigates background disturbances while reducing the computational complexity of the attention mechanism. In addition, we propose the Multi-modal Alignment Module (MAM) to perform visual-semantic bidirectional alignment, which facilitates better alignment between different modalities in a progressive manner. Finally, we introduce Adaptive Fusion Module (AdaFM) to adapt different guidance to different super-resolution blocks. By modeling character location to distinguish character regions from the background, our method can well handle scene text images with complex background.

Our contributions can be summarized as follows:

• We introduce a novel approach with explicit character location modeling to cope with the challenge from the complex background.
• We propose a visual-semantic bidirectional alignment and adaptive fusion strategy to generate and utilize highlevel text-specific guidance.
• Experiments on TextZoom and four STR benchmarks show that our method achieves consistently state-of-theart recognition accuracy.

2 RelatedWork

2.1 Scene Text Image Super-resolution

Different from single image super-resolution (SISR), whose goal is to improve image quality and obtain favorable visual effects. The main objective of STISR is to obtain easy-to-distinguish images to boost downstream recognition task. Early methods [Mou et al., 2020; Dong et al., 2015c; Wang et al., 2019] used CNN architectures to perform STISR tasks. In precisely, TextSR [Wang et al., 2019] uses adversarial training to enable the model to focus more on textual content. Based on the idea of multi-task learning, Plugnet [Mou et al., 2020] can obtain a unified feature for superresolution and recognition. Recently, the TextZoom dataset [Wang et al., 2020] was proposed to tackle real-world STISR

tasks. And they also proposed TSRN to exploit the sequential nature of scene text images. Benefiting from the global receptive field of attention mechanism, TBSRN [Chen et al., 2021] utilizes the content-aware loss and the positionaware loss to improve reconstruction results. PCAN [Zhao et al., 2021] improves performance by carefully designing SR blocks. TPGSR [Ma et al., 2021] takes a further step to exploit text prior. TG [Chen et al., 2022] shows that fine-grained clues can help yield more distinguishable images. TATT [Ma et al., 2022] uses the attention mechanism to work with irregular text images. C3-STISR [Zhao et al., 2022] uses clues from three perspectives to introduce better guidance. However, existing methods still treat character regions and background equally in the model design. Although some methods (e.g. TBSRN and TG) enable the model to focus on text by designing related loss, this implict manner is hard to observe whether the STISR model really focuses more on text, and incorrect supervision can be generated due to distractions such as attention drift. By contrast, we are the first to consider an explicit text focus to handle the challenges posed by complex background.

2.2 Scene Text Recognition

Scene text recognition is to recognize character sequences from scene text images. And it is closely related to scene text image super-resolution. Early text recognition methods used a bottom-up approach, but often suffered from lowresolution, small character, rotated, and illuminated scene text images. Recently, attention-based methods have gained interest because of their promising performance on irregular text. Specifically, SAR [Li et al., 2019] uses the 2D attention mechanism to recognize irregular text. RobustScanner [Yue et al., 2020] mitigates attention drift by enhancing positional cues. SRN [Yu et al., 2020] and ABINet [Fang et al., 2021] facilitate text recognition by using a language model. MATRN [Na et al., 2022] benefits from visualsemantic multi-modality. MGP-STR [Wang et al., 2022] boosts performance by multi-granularity prediction. Despite these advances, current scene text recognition methods still face challenges on low-resolution scene text images, and adapting models to low-resolution images through data augmentation has been shown to be limited [Chen et al., 2022; Chen et al., 2021]. Therefore, it is necessary to use the STISR method to obtain easily recognizable text images.

3 Methodology

3.1 Overview

Given low-resolution image XLR ∈ RH×W×3 , where H and W are the height and width respectively, the goal for STISR is to generate high-resolution text image XSR ∈ RfH×fW×3 , where f is the scale factor. In the proposed pipeline, XLR is first corrected by Spatial Transformer Network (STN) [Jaderberg et al., 2015] to tackle the misalignment problem. And then shallow feature XI is obtained using a shallow CNN. Then, XI will go through the guidance generation branch and the super-resolution branch. As for the guidance generation branch, the Location Enhancement Module (LEM) takes attention map sequence and XI as input to generate the selected feature XS. The Multi-modal Alignment Module (MAM) then performs visual-semantic bidirectional alignment using text distribution and XS. As for the super-resolution branch, XI will go through N stacked blocks, each of which contains an Adaptive Fusion Module (AdaFM) and a SequentialRecurrent Block (SRB) [Wang et al., 2020]. At last, PixelShuffle is performed to increase the spatial scale to generate XSR. Fig. 2 illustrates the architecture.

3.2 Location Enhancement Module

To improve the existing methods' equal treatment between character regions and background, we propose the Location Enhancement Module (LEM) to model the location of each character explicitly (Fig. 3 (a)).

Character Location Attention

We denote the attention map sequence generated by the pretrained attention-based text recognizer as hattn ∈ RT×H×W , where T is the max sequence length. Since the character length of different images varies and the raw attention map may introduce misguidance due to attention drift, we thus propose the Compression and Expansion strategy to tackle these problems.

Compression Strategy: Let L be the valid length of one character sequence which can be obtained from the pretrained recognizer, hjattn denotes the attention map corresponding to j-th character. Since different images vary in text lengths, we therefore remove additional paddings by choosing the first L valid attention maps to get {hjattn }Lj=1 , and then concatenate them followed by a max operator to reduce the channel dimension to 1. The result is denoted as hscore:

Expansion Strategy: We then use C convolution kernels to perform up-dimension on hscore followed by the Softmax function to get the result hpos:

Instance Normalization: According to previous studies [Huang and Belongie, 2017; Karras et al., 2019; Luo et al., 2022], the mean and variance contain the style of an image. To facilitate the subsequent alignment between image

(a) Location Enhancement Module

(b) Multi-modal Alignment Module

Figure 3: The detailed architecture of (a) Location Enhancement Module and (b) Multi-modal Alignment Module. Positional encoding is not represented for brevity.

and text, we perform Instance Normalization (IN) on XI to remove the varying styles so that focus more on the text content. The normalized features are then multiplied with hpos to obtain the location enhanced feature Xpos:

where ⊗ denotes the Hadamard Product.

Feature Selection

An intuition for visual-semantic alignment is that only character regions need to be aligned, while background does not necessarily perform expensive cross attention. Inspired by [Tang et al., 2022], we use the feature selection technique. Unlike [Tang et al., 2022] which uses manual annotation as well as a separate scoring network, in this work, we perform feature selection using the easily available attention map. Specifically, since hscore contains the pixel-level character confidence, we choose the top K large scores in hsocre to get the foreground coordinate set F:

We then use the coordinates set F as an index to gather foreground character features fromXpos. To avoid neighborhood information loss, we apply the pixel-to-region strategy before the gathering to make each indexed pixel representable to its neighbors by weighted summation in local regions:

where N denotes the neighborhood displacements and w(·, ·) denotes the weights at each displacement. In the practical implementation, we use the eight-neighborhood region as N . Note that it can be easily implemented by convolution.

Finally, we employ the coordinates in F to index X′pos and get the selected feature XS = {X′pos (i, j)}(i,j)∈F . It is worth noting that the proposed feature selection scheme not only enables more focus on the foreground but also reduces the computational complexity from the attention mechanism, see supplementary material for analysis.

3.3 Multi-modal Alignment Module

Existing approaches either use a single modal or use unidirectional multi-modal alignment to generate high-level guidance. We introduce the Multi-modal Alignment Module (MAM) to take a further step toward bidirectional visualsemantic alignment (Fig. 3 (b)).

Semantic Extraction

We first perform Semantic Extraction on the text distribution obtained by the text recognizer through linear projection and self-attention block. It can generate semantically rich feature htext which will be used to align with the visual modal.

Bidirectional Alignment

We propose a bidirectional strategy to facilitate cross-modal alignment. For the image-to-text alignment, we use the htext as query, XS as key and value to allow each character to find its corresponding image region:

where n denotes the n-th attention block. hn-1 is htext if n = 1 otherwise the output of previous block.

We denote the result of the first level alignment as hkey. Note that hkey has the same size as htext but contains information of XS. So, it can be easily aligned with XS in the second level alignment. For text-to-image alignment, we use hkey to bridge visual and semantic modals. Specifically, we use XS as query, hkey as key, and htext as value. Each element of XS can find which text feature it should attend by using hkey. Since the output of attention shares the same size as query, for the subsequent fusion, we use zero to pad XS before the second level alignment. The output high-level guidance XG is then used to guide super-resolution. Moreover, we refer to the positional encoding in [Na et al., 2022] to make full use of the attention map for better alignment.

It is worth noting that the previous unidirectional alignment only uses the second-level cross attention with htext as key and value [Ma et al., 2022]. However, without the progressive transition from the first level, XS (query) and htext (key) are difficult to align well. Related experiment can be seen in Section 4.4.

3.4 Adaptive FusionModule

Current guidance-based STISR approaches [Ma et al., 2021; Zhao et al., 2022; Ma et al., 2022] use the same guidance in different SR blocks. For this reason, we propose the Adaptive Fusion Module (AdaFM) to adaptively incorporate high-level guidance to different blocks. Given image feature Xn which is XI or the output of the previous block, and high-level guidance XG, we first concatenate them along channel dimension followed by three parallel 1×1 convolution to project Xn into three different feature spaces and denote them as X1n , X2n , and X3n respectively. We then perform the channel attention mechanism on X1n and multiply the resulting score with X2n to generate the channel attention feature, which will be added to X3n to get the final result. Notably, unlike the previous channel attention [Hu et al., 2018], we use global deep-wise convolution [Chollet, 2017] rather than global average pooling to better exploit the property of the spatial distribution of character regions in scene text images. The process can be formalized as follows:

where GDWConv denotes the Global Deep-Wise Convolution. The above procedure only uses 1 × 1 convolution and therefore results in only a minimal increase in the parameter complexity.

3.5 Training Objective

In this work, we use three loss functions, namely pixel loss, recognition loss, and fine-tuning loss, to train our model.

For the pixel loss, we use the L2 loss to perform pixel-level supervision:

For the recognition loss, we use the text-focus loss [Chen et al., 2021] to supervise the learning of language knowledge:

where A and p are the attention map and probability distribution predicted by a fixed transformer-based recognizer, respectively. WCE denotes weighted cross-entropy. λ1 and λ2 are hyperparameters.

Since we use a pre-trained text recognizer for guidance generation, as [Ma et al., 2022; Zhao et al., 2022] demonstrated, fine-tuning is better than fixed parameters. We therefore use the fin-tuning loss to adapt the text recognizer to lowresolution inputs:

where CE denotes cross-entropy loss, pPRE denotes probability distribution predicted by the pre-trained text recognizer.

The total loss is the weighted sum of these losses above.

where α1 and α2 are hyperparameters.

4 Experiment

4.1 Datasets

Scene Text Image Super-resolution Dataset

TextZoom [Wang et al., 2020] is widely used in STISR works. This dataset is derived from two single image superresolution datasets, RealSR [Cai et al., 2019] and SR-RAW [Zhang et al., 2019]. The images are captured by digital cameras in real-world scenes. In total, TextZoom contains 17367 LR-HR pairs for training and 4373 pairs for testing. Depending on the focal length of the digital camera, the test set is divided into three subsets, with 1619 pairs for the simple subset, 1411 pairs for the medium subset, and 1343 pairs for the hard subset. The size of LR images is adjusted to 16 × 64, and the size of the HR images is 32 × 128.

Scene Text Recognition Datasets

To verify the robustness of our method in the face of different styles of text images, we adopt four scene text recognition benchmarks, namely ICDAR2015 [Karatzas et al., 2015], CUTE80 [Shivakumara et al., 2014], SVT [Wang et al., 2011] and SVTP [Phan et al., 2013] to evaluate our model. Since these datasets do not contain LR-HR pairs and most of the images are of high quality, we therefore first perform preprocess to get LR images. Specifically, we select images with less than 16 × 64 pixels and then perform manual degradation on these selected images. One can find more information about these datasets' descriptions and degradation details in the supplementary material.

4.2 EvaluationMetrics

For the text recognition accuracy evaluation, consistent with previous work [Ma et al., 2021; Wang et al., 2020; Zhao et al., 2022; Ma et al., 2022], we use three text recognizers namely CRNN [Shi et al., 2016], MORAN [Luo et al., 2019], and ASTER [Shi et al., 2018] for evaluation in order to guarantee the generalization. For image fidelity evaluation, we used Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) to assess the quality of the generated SR images. However, as found in previous work [Chen et al., 2022; Chen et al., 2021; Zhao et al., 2022; Zhao et al., 2021], there is an inherent trade-off between text recognition accuracy and image quality, and we will discuss this observation in Section 4.6.

4.3 Implementation Details

We use ABINet [Fang et al., 2021] as the attention-based text recognizer because of the accessibility of code and pretrained model. We train our model with batch size 64 for 500 epochs using Adam [Kingma and Ba, 2014] for optimization. The learning rate is set to 1e-3 for the super-resolution and 1e4 for fine-tuning ABINet, both are decayed with a factor of 0.5 after 400 epochs. We refer to the hyperparameters on Ltxt given in [Chen et al., 2021], namely λ1 = 10, λ2 = 0.0005. For the other hyperparameters, we use α1 = 0.5, α2 = 0.01, see supplementary material for details.

4.4 Ablation Study

In this section, we conduct an ablation study to demonstrate the effectiveness of each component in LEMMA. CRNN [Shi et al., 2016] is chosen as the text recognizer for uniformity.

Different Choices on Location Enhancement

In the Location Enhancement Module (LEM), we use character location attention and feature selection to enable explicit character location modeling. Table 1 shows the effect of each part. The presence of character location attention resulted in a 3.9% improvement compared to no location enhancement. When using feature selection, recognition accuracy does not decrease due to the neglect of the background feature, instead it further increases by 2.2%. This suggests that there is redundant information in the feature map and removing this redundancy will improve performance.

Effectiveness of Bidirectional Alignment Strategy

To demonstrate the performance improvement does come from the bidirectional strategy rather than additional parameters, we increase the parameter number in the unidirectional alignment strategy by stacking more layers to make the complexity consistent. The results in Table 2 show that the bidirectional alignment improves the average accuracy by 1.8% compared to its unidirectional counterpart, demonstrating the effectiveness of the bidirectional alignment strategy.

Effectiveness of Different Modules

We study the effectiveness of different modules in LEMMA. Table 4 shows the result. It can be seen that the simultaneous presence of the three modules achieves the best result.

4.5 Comparison to State-of-the-Arts

We first compare the proposed method with others on the TextZoom dataset. Consistent with the previous methods, we evaluate the generalization using three text recognizers, namely ASTER [Shi et al., 2018], MORAN [Luo et al., 2019], and CRNN [Shi et al., 2016]. After that, we evaluate the model robustness on more challenging STR datasets.

Results on TextZoom

Table 3 shows the comparison results in terms of text recognition accuracy. It can be seen that our proposed LEMMA consistently outperforms the previous method on all three text recognizers. For example, using ASTER as the recognizer, compared to the previous SoTA method [Zhao et al., 2022], our method achieves 2.0%, 3.0%, and 0.6% improvement on easy, medium, and hard, respectively, which ultimately leads to an average improvement of 1.9%. The good results demonstrate the advantages of the proposed model. We also present the qualitative comparison results in Fig. 4.

Towards Robust STISR

In this section, we explore the model robustness in more challenging samples. All models are trained on the TextZoom dataset, after which we freeze parameters and evaluate them on four text recognition benchmarks namely ICDAR2015 [Karatzas et al., 2015], CUTE80 [Shivakumara et al., 2014], SVT [Wang et al., 2011], and SVTP [Phan et al., 2013].

Since most of the samples in these datasets contain realworld complex background, the result can thus give an indication of the model's ability to cope with complex background. As shown in Table 5, our proposed method allows the model to focus more on character regions and leads to better results.

Moreover, we also use all of the above four STR benchmarks to compare accuracy with different text lengths. It can be seen in Fig. 5 that our method outperforms others across all text lengths, and shows strong robustness in dealing with text of extreme length. The favorable performance of our approach on long text instances stems from the fact that explicit location enhancement allows the model to reinforce character region features and thus alleviate the long-range forgetting problem associated with long text.

4.6 Discussion

Trade-off between Fidelity and Accuracy

As presented in previous studies [Chen et al., 2021; Chen et al., 2022; Zhao et al., 2022], we also find the inherent tradeoff between fidelity and accuracy. Table 6 shows this observation. It can be seen that the highest PSNR is achieved with the settings of no LEM and recognition loss. The presence of LEM leads to a 0.4 dB reduction in PSNR, but it gives a 4.9% improvement in accuracy. The same phenomenon also happens with recognition loss. It can be explained by the fact that the proposed LEM mainly focuses on the foreground character regions and thus does not reconstruct enough of the background that occupies most of one image. Therefore, further recognition accuracy improvements (e.g. generating images with larger contrast) will come at the expense of fidelity.

Performance Brought from a Strong Recognizer

In this work, we use the ABINet to generate guidance instead of CRNN as in other works [Ma et al., 2022; Zhao et al., 2022]. Since ABINet itself can provide more reliable guidance, we conduct experiments to investigate how much the accuracy improvement is attributed to the use of a stronger recognizer. Specifically, we replace ABINet with CRNN to generate text distribution, and the average recognition accuracy is 55.2% (CRNN for downstream recognition). The adoption of a weak recognizer caused a 1.1% decrease compared with LEMMA. However, this setting still exceeds SoTA method (53.7%), demonstrating the validity of explicitly distinguishing between character regions and background.

Effectiveness of Strategies to Mitigate Attention Drift

We distinguish character regions from background using the attention map. As discovered in previous methods [Cheng et al., 2017; Yue et al., 2020], attention drift can affect the quality of the attention map and thus produce misguidance. Therefore, solutions are proposed in our framework to mitigate this adverse effect. First, in character location attention, we do not use the raw attention map directly, but process it with the convolution layer which can serve as a shift operator and thus correct the shifted attention map. Second, in feature selection, we use the pixel-to-region strategy so that each pixel can represent its neighbors and thus mitigate the impact of drift. The results in Table 7 indicate that all these strategies can alleviate the adverse effects from attention drift.

Failure Case and Limitation

Fig. 6 shows some failure cases. Our method, including previous STISR methods, has difficulty handling incomplete characters and artistic characters. Moreover, the quality of attention map which is used as prior guidance can affect the reconstruction results. Despite some strategies are proposed to mitigate this problem, it is far from being completely solved and we leave it as future work.

5 Conclusion

In this work, we propose the Location Enhanced MultiModAl network (LEMMA) to handle challenges in existing STISR methods by explicit location enhancement with more focus on character regions. The Location Enhancement Module extracts character region features from all pixels through character location attention and feature selection techniques. The Multi-modal Alignment Module employs a bidirectional progressive strategy to facilitate cross-modal alignment. The Adaptive Fusion Module adaptively incorporates the generated high-level guidance into different superresolution blocks. Results on TextZoom and four challenging STR benchmarks show that our approach consistently improves downstream recognition accuracy, taking a further step toward robust scene text image super-resolution.