TPS++: Attention-Enhanced Thin-Plate Spline for Scene Text Recognition

1 Introduction

Scene text recognition (STR) aims to understand and transcribe text images captured in the wild. With the prevalence of using cameras in applications such as autonomous driving, image retrieval, etc., it becomes one of the most active research themes nowadays [Baek et al., 2019; Long et al., 2021; Chen et al., 2021; Du et al., 2022; Wang et al., 2022]. However, STR still faces the problem that the recognition accuracy is not satisfactory for difficult text, partly because the text is not presented and captured in canonical, resulting in distortions such as perspective, orientation, and curved text. While humans can accommodate these distortions, for example by focusing on discriminative character patterns, modern recognizers still struggle to recognize such instances.

∗Corresponding Author

Text rectification emerges as a promising way to relieve the problem. Typically, it serves as a pre-processing that aims to correct the distortions and generate a cropped image concentrating on text foreground and in near canonical form. Therefore STR can be simplified by applying to the rectified image. Early works like STN [Jaderberg et al., 2015; Liu et al., 2016] used affine transformation to correct the distortions from image translation, scale and rotation perspectives. The irregularities, however, are diverse and complicated, so only limited success was achieved. With research efforts accumulated, Shi et al. [Shi et al., 2016] formulated the rectification using Thin-Plate Spline (TPS) transformation [Bookstein, 1989]. As depicted in the upper part of Fig.1, it first regresses a series of control points from image borders to text borders. Then, the movement of these points are used to calculate the TPS transformation parameters, which are further applied to cropping and rectifying the text. This branch of methods [Shi et al., 2016; Shi et al., 2018; Yang et al., 2019; Qian et al., 2021] offers a flexible rectification that is able to deal with anisotropic local distortions. It is one of the most influential solutions to date. Meanwhile, it shows advantages such as the rectification being trained in conjunction with the recognition in a weakly supervised manner, where only text labels are required. Moreover, it is a plug-in that can be inserted into any recognizer.

Despite great success, it is observed that this branch of rectifiers is less effective for difficult text. For example in Fig.1, ASTER [Shi et al., 2018] does not get a satisfactory rectification. The main reason lies in that it determines the TPS transformation parameters according to the regressed text borders, which are obtained via weakly supervised learning and are sometimes not well localized. More importantly, the borders carry very little text information. The transformation has not been taught how to accommodate characteristics of the text. Therefore, the rectified image, although basically maintaining the text's geometric structure, also gives rise to unnatural character deformation and out of bounds (e.g., the upper part of character "C" in Fig.1). On the other hand, there are a few studies that employ pipelines other than text border regression. For example, MORAN [Luo et al., 2019], ESIR [Zhan and Lu, 2019], STAN [Lin et al., 2021], etc. However, they either run slowly or do not perform better than the borderregression-based methods.

One essential cause of unnatural rectification is that the calculation of TPS parameters is content-free. The transformation is performed regardless of the text content. To solve it, more flexibility should be given to the computation of TPS parameters, allowing the correction to be aware of the text. It is therefore capable of suppressing undesired deformations by using text content to constrain the control point movement. Meanwhile, it is observed that the vast majority of methods treat rectification as a pre-processing. Therefore, image features are extracted independently for both rectification and recognition, wasting computational resources considerably. A lightweight and fast rectifier is supposed to obtain if it can share some features in common with the recognition.

With these observations, we propose TPS++, an attentionenhanced TPS to address the above issues. TPS++ aims to leverage the attention mechanism to establish correlations between control points and text content. By integrating them into the TPS formula, it assigns extra flexibility to the transformation, generating a more natural rectification that benefits the recognition. To enable this, three major upgrades are made. First, we re-endow the role of control points. Unlike previous studies that initiate them along the image borders, TPS++ adopts a grid-like initialization that uniformly distributes them spatially. Besides using their movements to estimate the TPS parameters, a majority of control points are located in text foreground rather than text borders. Therefore they catch the text content which is required in attention modeling. Second, we develop a gated-attention mechanism dedicated to modeling attention scores between control points and text content. With the score, additional flexibility is injected into the transformation. We devise a new formula that takes both the movement of control points and attention scores into account, building a more flexible content-aware TPS. Third, TPS++ gives a new paradigm where the rectifier is coupled with the recognizer. The rectifier shares the feature backbone with the recognizer in part. We perform the rectification at the image feature-level. The control point prediction and attention score estimation are carried out on feature maps that are also more flexible. It leads to a lightweight and efficient implementation. Meanwhile, it also has the merits of the two tasks easier to be jointly optimized. Extensive experiments are conducted on public benchmarks to verify the effectiveness of TPS++. It shows that adding TPS++ consistently improves the recognition accuracy. It correctly recognizes difficult text instances and achieves state-of-the-art accuracy, while the introduced overhead on parameters and inference time is just 0.5M and a few milliseconds. Moreover, TPS++ generalizes well on different backbones and recognizers.

TPS++ has four appealing properties. First, a flexible rectifier. It adds the attention score to TPS transformation, giving a more flexible text correction. To our knowledge, this is the first work exploiting the attention mechanism in text rectification. Second, an accurate rectifier. It gives large performance gains and achieves state-of-the-art accuracy when applied to popular recognizers. Third, a lightweight rectifier. It performs the rectification at feature-level instead of image-level, thus introducing very little overhead in terms of parameters and inference speed by allowing feature sharing. Fourth, a universal rectifier. It can be embedded into different backbones and recognizers without or with only trivial modifications, and consistently yielding accuracy improvements.

2 RelatedWork

2.1 Scene Text Recognition

STR is an intensively studied task in the deep learning era. Sequence-based models became popular due to their ability in integrating recognition clues from different aspects, e.g., visual, semantic, linguistic. Initially, CRNN [Shi et al., 2017] encoded the input image as a visual feature sequence, which was then modeled by BiLSTM for context reinforcement and CTC loss for text transcription. The paradigm was extended to GTC [Hu et al., 2020] by incorporating graph neural network and attention mechanism, and SVTR [Du et al., 2022] by leveraging visual transformers (ViTs). To use semantic clue, attention-based encoder-decoder models [Lee and Osindero, 2016; Cheng et al., 2017; Fang et al., 2018; Sheng et al., 2019; Bhunia et al., 2021] injected features extracted from text labels to the decoder, exploring the way of utilizing clues from both visual and semantic aspects. Robustscanner [Yue et al., 2020] and CDistNet [Zheng et al., 2021] devised dedicated position modeling modules, which were helpful in decoding characters by using the correct image features. Nevertheless, these methods used an iterative decoding scheme that identified the characters one-by-one, resulting in a slow speed. Consequently, parallel decoding schemes such as SRN [Yu et al., 2020], VisionLAN [Wang et al., 2021], ABINet [Fang et al., 2021] were introduced. Character placeholders were appropriately initialized such that the characters could be estimated in parallel with accelerated speed. Meanwhile, the linguistic clue was also modeled to maintain accuracy. Recently, self-supervised-based pretraining was taken into account in STR and led to improved accuracy [Yang et al., 2022; Yu et al., 2023].

While these models gained steady performance improvements, they did not specifically take into account text irregularities, thus less effective for irregular text. To overcome this problem, segmentation-based and rectificationbased methods were developed. The former regarded the recognition as a character-level segmentation task and exhibited impressive performance in dealing with these distortions [Liu et al., 2018; Liao et al., 2019; Xing et al., 2019; Li et al., 2017]. Nevertheless, they required character-level annotations which were not always readily available. Moreover, the segmentation was also sensitive to noise. On the other hand, rectification-based methods aim to rectify the text to generate a near canonical and easier readable counterpart. They were compatible with most off-the-shelf recognizers and thus received considerable research attention.

2.2 Text Rectification

Based on the transformation formulation, we roughly classify rectification-based methods into affine transformation-based, TPS-based methods and others. The first aims to eliminate the distortion based on simple transformations such as translation, scale and rotation. The first work was STN [Jaderberg et al., 2015] proposed for general object rectification. It was studied in STR by applying the transformation on characters [Liu et al., 2016] and image patches [Lin et al., 2021]. TPSbased approaches noted that the text captured in the wild exhibits anisotropic distortions, which were difficult to describe by affine deformations. Shi et al. proposed RARE [Shi et al., 2016] and ASTER [Shi et al., 2018] that enabled more flexible rectification by using TPS [Bookstein, 1989] transformation instead. However, their weakly supervised nature makes the calculation of TPS parameters less accurate for highly distorted text. To better handle such instances, an iterative rectified scheme was developed in [Zhan and Lu, 2019]. It reduced the dependence on point regression quality but increased the time consumption. In [Yang et al., 2019], a more accurate text border regression was obtained by predicting a series of text geometric attributes and utilizing them to assist the rectification. However, costly character-level supervision was required. Besides the two major branches, MORAN [Luo et al., 2019] regarded text image as a grid and directly regressed offsets at each cross point. However, unnatural distortion was observed due to these offsets are not constrained by any geometric transformation. Zhang et al. [Zhang et al., 2021] paid attention to color-related rectification and devised a structure-preserving rectifier.

Compared with these methods, TPS++ makes use of the attention mechanism to improve the quality of rectification. By adding attentional parameters, it greatly alleviates unnatural distortions observed in current rectifiers such as TPS. Moreover, it performs the rectification at feature-level rather than image-level, thus with only a limited recognition overhead in terms of parameters and inference speed.

3 Methodology

3.1 Overview

An illustrative framework of TPS++ is depicted in Fig.2, which consists of two parts: multi-scale feature aggregation (MSFA) and attention-incorporated parameter estimation (AIPE). We taking ResNet-45 as the backbone for illustration. Given a text instance, feature maps of the first three blocks in the backbone are injected into MSFA. These features are aggregated and two outputs are produced, i.e., the encoded feature Fe and decoded feature Fd. Then, AIPE uses the two features to predict the movement of control points and attention matrix A, which records the attention score between control points and text content. Based on these parameters, attention-enhanced TPS rectification is performed at the image feature-level. The rectified features are fed back to the backbone. The subsequent feature extraction and recognition are carried out the same as conventional.

TPS++ is featured by two distinct properties compared with existing rectifiers. First, it shares visual feature extractor in part between recognition and rectification, generating a tight coupling scheme that well controls the parameter and inference speed overhead, while also largely preserving its plug-in nature. Second, it introduces the attention mechanism to TPS, enabling a more flexible content-aware correction. Both improve the rectification quality and ease the recognition. Meanwhile, TPS++ also inherits merits such as end-toend trainable with STR, requiring no extra annotations beyond text labels. Details will be elaborated later.

3.2 Multi-Scale Feature Aggregation

Most previous text rectifiers [Shi et al., 2016; Shi et al., 2018] are designed as a pre-processing before STR. The scheme is computationally intensive as feature extraction is executed twice. To overcome the drawback, TPS++ shares the feature backbone in part with the recognizer. We design a thin module called MSFA that accepts feature maps generated from the first three backbone blocks. The features are scaled to the same size and concatenated from the channel dimension. Specifically, features from the 1st and 2th blocks are reduced by 4× spatially, while feature channels of the 1st, 2th and 3rd blocks are all aligned to 64. Then, a lightweight encoderdecoder-based feature extractor, whose structure is provided in Tab.1, is applied. It consists of a contracting path and a symmetric expansive path, both containing three convolution layers. In addition, CBAM [Woo et al., 2018], the channelspatial joint attention, is applied to highlight important features. With these operations, multi-scale visual features are aggregated and optimized towards the rectification purpose. Features obtained from CBAM (i.e., F e ∈ RWe×He×D ) and the outputted layer (i.e., Fd ∈ RWd×Hd×D ) are termed as the encoded and decoded features, respectively. Note that the two features have the same number of channels and Fd has the same spatial resolution as the scaled input feature.

Separating the feature extraction into two parts also is a key factor that makes TPS++ effective. By doing this, the first part emphasizes extracting generic visual features. While the second is MSFA which targets rectification optimization. It aggregates features from shallow blocks that contain more location-related clues, which is useful for control points regression and attention modeling. Note that the second part is slim and introduces only a few computational overhead.

3.3 Attention-Incorporated Parameter Estimation

AIPE is appended behind MSFA for control point regression and content-based attention score estimation. As depicted in the middle right of Fig.2, besides regressing control points, it predicts a content-aware attention score matrix A in parallel by using the gated-attention mechanism. Specifically, in control point regression, unlike TPS initializes the points along the image borders, we set control points C ∈ RWeHe×2 uniformly distributed on the feature map spatially, i.e., a gird-like distribution. Therefore, a large portion of points is located in text foreground rather than less informative borders. Note that the number of control points equals the spatial resolution of Fe. Meanwhile, Fe is reshaped as a feature sequence ˆ Fe ∈ RWeHe×D . It undergoes two linear layers to predict the offsets of each control point on x and y dimensions. Combining with the initialized coordinates, we get a set of regressed control points C′ ∈ RWeHe×2 .

In attention score estimation, we assess the correlation between Fe and Fd to get an attention score matrix A that is aware of text content. Specifically, the two features both feed into a dedicated designed module called dynamic gatedattention block (DGAB), where attention scores between control points and text are adaptively computed. Then, the obtained feature is reshaped and further combined with ˆ Fe via matrix multiplication, followed by a scaling of 1/√D and Tanh activation that restricts the attention scores to (-1, 1).

Fig.3 gives the detail of DGAB. First, the decoded feature F d is shrunk to two feature sequences of D×Wd and D×Hd by averaging along the H and W dimensions, respectively. Then, the two sequences are separately concatenated with the reshaped encoded feature ˆ Fe, followed by the linear layer to align the dimension back to D×Wd and D×Hd and dynamically calculate their weights along Wd and Hd dimensions, namely £h and £w. They represent the importance of each column and row, respectively. Meanwhile, the aligned features experience a Softmax operation and are multiplied by the derived weights. In the following, the obtained features are broadcasted back to dimension Wd × Hd × D and are merged by element-wise sum. An element-wise product with the raw Fd is followed up to generate the output.

With the estimated control points and attention scores, we develop a more flexible attention-enhanced TPS transformation. For each location p′i in the corrected feature map, its original location pi is determined by a geometric transformation as given by Equ.1. Then, pi is computed by T and F(·) contains the attention score as follows.

where K = WeHe is the number of control point. S ∈ RK×K is a square matrix with element sij = Eu(||ci - cj ||) defined as radial basis kernel applied to the Euclidean distance between ci ∈ C and cj ∈ C.

where Ai,k is the attention score between the i-th location and the k-th control point. λ and β are hyperparameters empirically set to 0.5 and 1, respectively. When λ=0, the equation goes back to conventional TPS exactly.

As seen in Equ.3, TPS++ gives rise to additional flexibility to TPS by incorporating the attention score. It allows a content-aware adaptive weighting on control points when performing the transformation. Since the rectification and recognition are jointly optimized, this flexibility is able to guide parameter updated towards a better STR, and meanwhile, generating a more natural rectification. Note that several studies [Lee and Osindero, 2016; Cheng et al., 2017; Shi et al., 2018; Sheng et al., 2019; Lin et al., 2021] also incorporated the attention mechanism in STR, but their attention is considered in the text recognition stage only. To the best of our knowledge, TPS++ is the first that introduces the attention mechanism to text rectification.

3.4 TPS++ on Different Recognizers

One appealing property of TPS is the plug-in nature. It can be seamlessly inserted into any text recognizer. TPS++ largely preserves this property. In case ResNet-45 is employed as the backbone, TPS++ can be directly appended no matter which recognizer is employed, otherwise a modification might be required. To be compatible with MSFA, we should give three feature maps whose sizes can be normalized to 16 × 64 × 64. This is not a strict restriction for popular CNNs even ViTs. Therefore to apply TPS++, only trivial accommodation is required to align the features in spatial and channel. We will verify this capability of TPS++ in experiments.

4 Experiments

4.1 Datasets and Implementation Details

Following the standard protocol in STR [Baek et al., 2019], models are trained on two synthetic datasets and evaluated on six public benchmarks, which are as follows:

MJSynth (MJ) [Jaderberg et al., 2014] and SynthText (ST) [Gupta et al., 2016] are the two synthetic datasets with 8.91M and 6.95M text instances, respectively.

ICDAR2013 (IC13) [Karatzas et al., 2013], Street View Text (SVT) [Wang et al., 2011], IIIT5k-Words (IIITK) [Mishra et al., 2012], ICDAR2015 (IC15) [Karatzas et al., 2015], SVT-Perspective (SVTP) [Phan et al., 2013] and CUTE80 (CT80) [Risnumawan et al., 2014] are six benchmarks widely used in assessing STR models. The first three mainly contain regular text while the rest three are irregular.

All models were trained with Adam optimizer for 12 epochs on the two synthetic datasets, only word-level annotations are utilized. The initial learning rate was set to 1e-3 , which was reduced to 1e-4 and 1e-5 at the 8th and 10th epoch, respectively. All input images were resized to 32 × 128. The batch size was set to 200. Warm-up strategy was used in the first epoch, and the initial warm-up ratio was set to 0.001. For different recognizers, for fairness we follow the parameter settings as their papers and performed data augmentation uniformly. TPS++ was trained by two steps. The recognizer was trained w/o TPS++ at first. Then, TPS++ was appended and jointly trained again. All models were trained by using a server with 6 NVIDIA 3080 GPUs.

4.2 Ablation Study

We employ ABINet-V [Fang et al., 2021] as the base network. It uses ResNet-45 and two Transformer units for STR. The language model is not taken into account.

Backbone feature utilization. Since TPS++ is coupled with recognition, how to share the backbone becomes an issue. We perform experiments to ablate this. The first feeds the raw image directly to the encoder-decoder-based feature extractor in MSFA (i.e., with insert position 0. Below defined similarity). It treats rectification and recognition separately as conventional methods. While the second and third trails acquire feature maps behind the 3rd and 6th blocks, respectively. As seen in Tab.2, the performance decreases for higher insert positions. It is explained as the feature spatial resolution is reduced along with the increase of insert position, thus less information is carried. For example, the first scheme operates on the full-resolution image while the second on a 16× smaller one. To tackle this dilemma, we propose to aggregate multi-scale features and scale them to the same size as features from the 3rd block. It not only gets the best performance but also the computational cost is well controlled.

Number of control points. We assess how the accuracy and inference time varies with the number of control points. As shown in Tab.3, increasing the number of control points along either width or height dimension both improves the recognition accuracy. It achieves the highest accuracy when 4 × 16 points are sampled. This is not surprising as sampling denser also enables a fine-grained perception of text content, therefore better alleviating the problem of unnatural character deformations. On the other hand, it is also seen that different sampling schemes have very close inference time consumption, indicating that the control point-related computation is efficient. Therefore the 4 × 16 scheme is chosen.

Attention formulation. We evaluate different attention formulations in TPS++ and compare them with existing rectification models. To maintain fairness, the backbone and recognition network are kept consistent. The results are presented in Tab.4, where Baseline does not take into account the rectification. Gird denotes that only Fe is utilized to regress control points. Grid+Atten means a simple attention mechanism is considered (w/o utilizing DGAB). TPS++ (w) means that DGAB is only considered in the width dimension. MORAN [Luo et al., 2019], SPIN [Zhang et al., 2021] and ASTER [Shi et al., 2018] are popular rectifiers and we reproduce them using ABINet-V, a more powerful recognizer.

As seen, MORAN employs an image-level grid rectification. It performs worse than Gird although improvements are observed compared with Baseline, indicating the effectiveness of performing the rectification at feature-level. ASTER uses TPS transformation to perform the rectification. The better result shows that anisotropic transformation is vital for text irregularities correction. Grid+Atten gets accuracy advantages on 5 out of the 6 benchmarks compared with ASTER, indicating the efficacy of incorporating the attention mechanism which brings more flexibility. While TPS++ further employs a dedicated gated-attention along both the width and height dimensions to reinforce the attention modeling. The accuracy increases step-by-step. It achieves the best accuracy in all 6 benchmarks while the parameter increment is 0.5M, much less than the 1.7M in ASTER [Shi et al., 2018] and 2.3M in SPIN [Zhang et al., 2021]. In Fig.4, three exemplars with their feature attention heatmaps before and after TPS++ are visualized, which is obtained by averaging feature activation along the channel dimension. It shows the feature has been rectified considerably after TPS++. Meanwhile, text foreground is mostly dyed deeper, implying that TPS++ learning has made text foreground playing a more important role in recognition. The results clearly demonstrate that the attention mechanism is appropriately formulated and the rectification can be well established at feature-level.

4.3 Results and Comparisons

We first assess how TPS++ is compatible with different backbones. Thus, three STR models adopting ResNet-31 [Yue et al., 2020], ResNet-45 [Fang et al., 2021] and ViT [Dosovitskiy et al., 2020] are selected. The first two can be directly used while for ViT a simple accommodation is required. The results are shown in Tab.6. Performance improvements are steadily observed no matter which backbone is employed. When looking into the irregular datasets, TPS++ receives performance gains of 1.7-5.5%, 3.6-5.0% and 2.4-4.7% on IC15, SVTP and CT80 across the three backbones, respectively. The encouraging results indicate that TPS++ has a strong adaptation to backbone changes. It also proves the effectiveness of attention-based feature-level rectification.

We then evaluate the performance of TPS++ with different recognizers. Concretely, CRNN [Shi et al., 2017], NRTR [Sheng et al., 2019] and ABINet [Fang et al., 2021] are selected, which stand for CTC-based, attention-based and parallel decoding-based methods, respectively. Similarly, other components are kept unchanged. Tab.7 presents the results, where ABINet-LV is ABINet-V plus with the language model. Generally, TPS++ brings performance gains compared to raw and TPS on all the recognizers especially in irregular datasets. The results imply that TPS++ generalizes well across recognizers. Together with the results on backbones, it reliably verifies the plug-in property of TPS++.

In Tab.5, we list the results of existing studies from the angle of both popular recognizers and rectifiers. When equipped with TPS++, both NRTR and ABINet-LV gain performance improvements and surpass almost all comparing methods. Specifically, 3 out of the 6 benchmarks obtained by ABINet- LV+TPS++ are new state of the art, while the cost is nearly 0.5M increase on parameters and 4.3ms on inference time compared to ABINet-LV. Both are marginal when compared to the differences among different models (Here both CRNN and NRTR are equipped with ResNet-45, a more powerful backbone compared to their raw implementation). When compared with ASTER, CRNN+TPS++ already gives nearly 3 percent accuracy improvements on irregular text, consuming just 74% parameters and 23% inference time. Meanwhile, ABINet-LV+TPS++ gives an even larger accuracy gap and still runs faster. The results clearly reveal that TPS++ is an accurate, efficient and universal tool towards better STR.

In Fig.5 we illustrate several successful and failed cases. The rectified image is also given by projecting the rectification back to the image. As seen, TPS++ gives a relative natural rectification even severe text distortions are presented, and thus gets the correct recognition. For the failure cases (the fourth line). They are mainly because the text is severely blurred. Despite being corrected to some extent, they are even unreadable for human beings, and recognizing such cases still remains a common difficulty for STR.

5 Conclusion

We have presented TPS++, an attention-enhanced rectifier for STR. It introduces the content-aware attention score to TPS formula, endowing the transformation with more flexibility. Moreover, it is a feature-level rectifier that partially shares the feature backbone with the recognizer, thus lightweight and fast. The experiments conducted on standard benchmarks basically validate our proposal, from which the effectiveness, efficiency and universality of TPS++ are well demonstrated. We hope that TPS++ will foster future research in STR, text spotting [Fang et al., 2022], etc.

Acknowledgments

The work was supported by National Key R&D Program of China (No. 2020AAA0104500) and the National Natural Science Foundation of China (No. 62172103).