Focus on the Whole Character: Discriminative Character Modeling for Scene Text Recognition

1 Introduction

Scene Text Recognition (STR) aims to recognize character sequences from cropped text images [Bautista and Atienza, 2022; Zhang et al., 2023; Cheng et al., 2023; Fan et al., 2023]. Existing STR methods adeptly read texts encompassing billboards, road signs, checks, etc. However, with societal advancements, the demands on STR models are no longer limited to performing well in simple texts, but also need to improve their performance in challenging texts.

∗Corresponding Author

The recent methods have obtained superior performance on six common benchmarks [Risnumawan et al., 2014; Mishra et al., 2012; Wang et al., 2011; Phan et al., 2013; Karatzas et al., 2013; Karatzas et al., 2015] which are easy to recognize as shown in Figure 1(a). However, with the introduction of a challenging Union14M-Benchmark [Jiang et al., 2023], existing STR models perform poorly on Curve, Artistic, and Contextless datasets. For example, MGP-Base [Wang et al., 2022] only achieves the accuracy rates of 55.2%, 52.8%, and 48.4% on them, respectively. We believe that the poor performance on challenging texts is primarily due to two ignored issues.

The first issue is summarized as Large Intra-class Variance (LISV). Figure 1(b) illustrates the different instances of the character 't'. There are variations in the visual appearances, shape, and size, which lead to errors in recognition results. We attribute the misrecognition of characters to the large variance of the same category characters. Due to the existence of LICV, the discriminative features of characters will be weakened, eventually leading to the misrecognition of characters. To solve LICV, it's essential to enhance the discriminability of features by encoding local patterns or structures within the character. This helps model the correlation between the components of a character, capturing discriminative information more comprehensively. However, existing STR encoders have difficulty learning the intra-character local patterns. CNN-based encoders [Shi et al., 2016; Baek et al., 2019] can learn local features, but the receptive field is too small to perceive the information of the whole character. Transformer-based encoders [Sheng et al., 2019; Bautista and Atienza, 2022] hardly focus on the local information at character level due to the global modeling during self-attention. Thus, improving the ability to encode local patterns (e.g., stroke, morphology, etc.) at character level is a crucial step in solving LICV.

The second problem is Small Inter-Class Variance (SICV). As shown in Figure 1(c), our baseline recognizes some characters as other different category characters due to their similar visual appearances. For example, the word 'happy' is misrecognized as 'floppy', which also conforms to linguistic rules. Therefore, using additional language models or linguistic information does not help solve the SICV problem. Furthermore, we argue that SICV can lead to a mixed distribution of different category character features in the decoding space (demonstrated formally in Sec. 4.6). To solve SICV and LICV together, CornerTransformer [Xie et al., 2022] designs a Character Contrastive loss (CC loss) to bring the same category characters closer and separate characters from different classes. However, since CC loss only considers the character feature distribution within a batch, it greatly limits the diversity and richness of the global character feature distribution. Hence, how to utilize global character feature distribution is a great challenge for solving LICV and SICV.

To address the above issues, we propose a Character Features Enriched model (CFE) to obtain the discriminative character features from two aspects. Firstly, the Character-Aware Constraint Encoder (CACE) is proposed to perceive the local patterns such as the morphological information. For CACE, the discriminative features are extracted by multiple stacked blocks. In each block, we design a decay matrix D to attenuate the self-attention mechanism according to the spatial distances between visual tokens. The greater the distances between tokens, the less attention is paid to them. Compared to the vanilla self-attention, the block can alleviate the noise interference from other characters and pay more attention to the region near each token. In this way, CACE encodes the local patterns and relationships between the components of each character. Additionally, to fully utilize the visual features output from the multiple blocks, a fusion strategy is employed to merge them. Secondly, an Intra-Inter Consistency Loss (I2CL) is introduced to solve LICV and SICV. Based on the contrastive learning [Qi and Su, 2017; Xie et al., 2022; Zhang et al., 2022], I2CL further predefines a long-term memory unit for each character category. For each character in a batch, its positive example refers to the unit with the same character category, while the other units serve as negative samples. In the training, I2CL updates these memory units based on each character. Different from [Xie et al., 2022;

Zhang et al., 2022] that only consider the local distribution, I2CL can efficiently represent all characters by learning a discrete distribution for long-term memory units. Finally, all characters will be tightly distributed around the memory units according to their categories. This ensures the intraclass compactness and inter-class separability, and improves the discriminability of characters. Compared with previous methods, we have fewer training parameters, while achieving better performance.

The main contributions of our work are as follows:

• We point out that the LICV and SICV issues in challenging texts lead to poor performance of the STR models, and propose a novel approach to effectively handle the two issues.
• We design a Character-Aware Constraint Encoder to focus on the local patterns of character level, which utilizes the morphological information to enrich features.
• We introduce an Intra-Inter Consistency Loss to reduce intra-class variance and increase inter-class variance by learning a set of long-term memory units.
• Experiments on common benchmarks and Union14Benchmark demonstrate that our CFE surpasses stateof-the-art performance, with accuracy rates of 94.1% and 61.6%, respectively.

2 RelatedWork

2.1 Scene Text Recognition

In scene text recognition models, the visual encoder is an essential component. It aims to provide discriminative visual feature representation for subsequent CTC [Graves et al., 2006] decoders, attention decoders, or Transformer decoders. Early methods employing CNN as encoder have been widely applied in various networks and applications [Shi et al., 2016; Shi et al., 2018; Baek et al., 2019]. However, CNN-based approaches typically compress the height dimension of images into 1 during feature extraction, causing each visual feature to correspond to a thin-slice region in the image. This limitation leads to poor performance on irregular datasets. Recently, due to the significant advancements of Transformer in the visual domain, many recent works [Atienza, 2021; Du et al., 2022; Bautista and Atienza, 2022; Wang et al., 2023] have opted to use Vision Transformer as the visual encoder. These methods demonstrate good performance on irregular datasets. ViTSTR [Atienza, 2021] utilizes the Vision Transformer as the encoder to model relationships between different visual tokens. SVTR [Du et al., 2022] employs a pyramid-style Transformer as the visual encoder to guide the model in establishing global relationships between characters and local relationships within character. Although SVTR and ViTSTR are both pure visual models, there exist some performance differences. We attribute this to SVTR enhancing its ability to model the relationships between the character components. For the problem of LISV, we also need to focus on the local features to obtain discriminative features for recognizing characters. Therefore, we propose the CharatcerAware Constraint Encoder. It can perceive the local patterns within character by utilizing the decay matrix in each block.

2.2 Contrastive Learning in STR

In scene text recognition, the optimized object is not only cross entropy or CTC loss [Graves et al., 2006], but also combines contrastive learning to handle different problems. ConCLR [Zhang et al., 2022] uses contrastive loss to bring identical characters closer and separate different characters in the embedding space. DiG [Yang et al., 2022] incorporates a contrastive learning branch to mimic human-like reading behavior and learn text discrimination. In the meantime, it employs a momentum branch to create a comprehensive and reliable dictionary on-the-fly. CornerTransformer [Xie et al., 2022] designs a Character Contrastive loss that implicitly learns common features for each character class for solving artistic text recognition. CLIP-OCR [Wang et al., 2023] introduces a linguistic consistency loss for aligning the intrarelationship and inter-relationship. We argue that contrastive learning is also helpful in solving the problems of LICV and SICV. Hence, we introduce the Intra-Inter Consistency Loss to take intra-class compactness and inter-class separability into account. This loss function can learn a long-term memory unit for each class, simultaneously ensuring that the memory units of different categories are far apart.

3 ProposedMethod

In this section, we first detail the pipeline of the proposed method CFE in Sec. 3.1, and then introduce Character-Aware Constraint Encoder (CACE) and Intra-Inter Consistency Loss (I2CL) in Sec. 3.2 and Sec. 3.3 respectively.

3.1 Pipeline

The pipeline of CFE is illustrated in Figure 2 and it can be viewed as an encoder-decoder architecture. Given a batch of images with size B×H×W ×3, after three stages in CACE, we acquire the three visual sequences: F1 ∈ RB× HW 16 ×C1 , F2 ∈ RB× HW 32 ×C2 , F3 ∈ RB× HW 64 ×C3 . Subsequently, we linearly map and concatenate them to get the final output visual sequences F ∈ RB× 7 64 HW×C . Next, F is fed into the Transformer Decoder to generate recognition features O ∈ RB×T×C . Finally, we calculate the CE loss and I2CL separately and sum them up to obtain the training objective.

3.2 Character-Aware Constraint Encoder

To address the issue of LICV, we present a novel CharacterAware Constraint Encoder (CACE). In this encoder, the visual features are extracted from three stages, and each stage consists of multiple stacked blocks. CACE introduces an explicit decay matrix D into the block to encode the local patterns (e.g., stroke, morphology, etc.) and relationships between the inter-character components. Additionally, to fully utilize the visual features output from the three stages, we employ a simple multi-scale fusion strategy to merge them.

The block is depicted in Figure 2, we summarize how the block works in Eq 1:

where X ∈ RB×L×Ci is the input of the block and LN stands for Layer Normalization. WQ, Wk, WV are the learnable projection matrices. Θ represents the position embedding and ˆΘ is its complex conjugate followed [Sun et al., 2023]. D ∈ RB×L×L represents the decay matrix which values in [0,

1]. We design three different ways to yield D:

where xi and yi denote the X and Y coordinates, respectively, of the i-th token in 2D space. γ values range between 0 and 1 [Sun et al., 2023]. [·] is an indicator function, if the condition is satisfied returning 1 else 0. w, h are used to control the region during self-attention which are set 5 and 3 (insensitive) empirically. Further, we visualize D in the form of heatmap. As shown in Figure 3, the first two options correspond to dynamic decay, which is inversely proportional to the distances. Option 3 employs fixed decay which utilizes a predefined binary window to control the attention region for each token like in SVTR [Du et al., 2022]. In this paper, we adopt option 2 as our choice. We argue that dynamic decay is compatible with human perception of distances. Ablation study in Sec. 4.5 demonstrates its effectiveness.

In Eq. 2, D introduces spatial distances prior knowledge into the self-attention mechanism. The longer distances between token i and j, Di,j become smaller. Its purpose is to avoid the influence of irrelevant noisy tokens over long distances, allowing each token to model the different components within the whole character. Finally, the decay matrix enables the visual encoder to learn more distinctive visual representations for characters. Consequently, compared to global modeling without D, it exhibits greater proficiency in capturing local patterns in each character. In practice, we employ D into the first x blocks for intra-character components modeling, while the remaining blocks do not use D for intercharacter modeling.

To better capture the different patterns of a certain character, we design a fusion strategy to obtain multi-scale character features. Specifically, it fuses the features F1, F2, F3 by projecting them into the same hidden dimension. Overall, on the one hand, by adding the decay matrix, CACE enables each token to perceive local patterns within character, allowing for better extraction of discriminative features. On the other hand, through multi-scale fusion, CACE can integrate the local patterns of a character at different scales, which can enhance the diversity of character features.

3.3 Intra-Inter Consistency Loss

Although CACE can help alleviate the problem of LICV, the existence of SICV still prevents the model from reaching its optimal performance. The SICV issue leads to the phenomenon of mixture distribution between character features. Therefore, it is necessary to increase the distance between characters of different categories and decrease the distance between characters of the same category. We believe that using the trait of contrastive learning to solve the SICV problem is a good choice. However, existing contrastive learning methods mostly cluster characters within a batch, lacking the perception of global character feature distribution. Hence, we introduce the Intra-Inter Consistency Loss (I2CL) to explore the distribution of each character category by learning a set of long-termmemory units.

Formally, the I2CL is defined as illustrated in Eq 3:

where Lcl represents the I2CL. Initially, we reshape the output feature O into RBT×C . BT denotes the number of characters in a batch and T is the maximum length of character sequences in a text. Oi ∈ RC signifies the i-th character feature, yi denotes the label of Oi. cyi represents the memory unit of the yi-th character category in decoding space which can be updated during training. V denotes the vocabulary size. δ is a constant used to prevent the denominator from equaling 0 and we set δ = 1 by default.

Compared with DiG [Yang et al., 2022] and CornerTransformer [Xie et al., 2022], although these approaches all use contrastive learning loss, there remain some differences. DiG and CornerTransformer only consider the intra-class compactness and inter-class separability of features in a batch. But I2CL considers the clustering between all the training samples. After training, the memory units of our I2CL will be a discrete distribution because there is a penalty for too small distance between different memory units. In addition, we can use these memory units to represent the global character feature distribution. From one perspective, this discrete distribution will increase the distance between characters of different categories and solve the SICV problem. From another perspective, all characters will be tightly distributed around the memory units corresponding to their categories, alleviating the LICV problem.

3.4 Training Objective

The final objective function of the proposed method is formulated in Eq. 4. Lce represents the cross entropy loss. λ denotes the scalar used to balance the two loss functions and is set to 0.2.

4 Experiment

4.1 Datasets

Following the setup of [Wang et al., 2022; Fang et al., 2021], we conduct experiments using MJSynth [Jaderberg et al., 2014; Jaderberg et al., 2016] and SynthText [Gupta et al., 2016] as training data. The training data consists of 16M synthetic text images. We evaluate the model on 6 common benchmarks containing IIIT [Mishra et al., 2012], IC13 [Karatzas et al., 2013], IC15 [Karatzas et al., 2015], SVT [Wang et al., 2011], SVTP [Phan et al., 2013] and CT [Risnumawan et al., 2014]. To further valid the effectiveness of our CFE, we extra test performance on Union14MBenchmark [Jiang et al., 2023], ArT [Chng et al., 2019], COCO-Text [Veit et al., 2016], Uber-Text [Zhang et al., 2017], and WordArt [Xie et al., 2022]. Beyond that, we also supply a few experiments trained on Union14M-L.

4.2 Implementation Details

To balance the accuracy and speed, we develop three variations with varying numbers of parameters similar to SVTR and LPV [Zhang et al., 2023]. The specific network configurations are detailed in Table 1. The images are resized to 32 × 128. For data augmentation, we follow the random data augmentation methods [Bautista and Atienza, 2022] including Invert, GaussianBlur, Sharpness, and PoissonNoise. The vocabulary size V is 96, which comprises mixed-case alphanumeric characters, punctuation marks, [BOS] for the beginning symbol, and [EOS] for the ending symbol. The maximum label length T is set to 25. We train 20 epochs with a warm-up of 1.5 epochs, utilizing the Adam optimizer [Kingma and Ba, 2014] with a learning rate of 5e-4. We add Lcl in the last 25% time of the training process. By this point, the model has already demonstrated convincing recognition capabilities, allowing more accurate learning for memory units and clustering. We use the Transformer Decoder with 1 layer as the recognition decoder and adopt autoregressive decoding for training. The experiments are conducted on 4 NVIDIA 4090 GPUs with a batch size of 384.

4.3 EvaluationMetric

For validation, we configure the vocabulary size to 36, encompassing 0-9 and a-z. We employ the word accuracy as the evaluation metric. Consistent with [Baek et al., 2021], we record the weighted average score (WAVG) on common benchmarks based on the number of samples. As for Union14M-Benchmark, we report the average score (AVG) like [Jiang et al., 2023].

4.4 Comparisons with State-of-the-Arts

In Table 2, we compare our CFE with multiple recent stateof-the-art methods on common benchmarks. All the methods are trained by synthetic image texts for fair comparison. CFE-Base shows significant performance, especially in datasets of regular IIIT and irregular SVTP and CT. It achieves SOTA accuracy of 94.1% while keeping low parameters (23.9M) compared to other STR models. Further, CFETiny outperforms LPV-Tiny with only 4.5M parameters, and CFE-Small acquires a good performance of 93.4% with 9.2M parameters. Besides, CFE-Base achieves SOTA performance on Union14M-Benchmark, with an average score of 61.6%. For challenging datasets in Table 3, although CFE-Base attains first or second accuracy, there remains a large gap with LISTER-Base on the Uber-Text dataset which contains many vertical texts. This is because LISTER-Base performs a rotation operation based on the original aspect ratio of the images, enhancing its ability to recognize vertical texts.

The performance trained on Union14M-L is shown in Table 4. Compared to the recent methods with additional linguistic information, our CFE-Base can continue to keep the best accuracy on most datasets of Union14M-Benchmark. These results all imply that our CFE is effective in enhancing the discriminative character features for solving challenging scene text recognition and can maintain superior performance in recognizing simple texts.

4.5 Ablation Study

The Effectiveness of CACE and I2CL

To investigate the effectiveness of CACE and I2CL, we conduct experiments in Table 5. The baseline refers to not using CACE and I2CL which achieves an accuracy of 59.8%. After adding CACE to guide the encoder to perceive the morphological information and character region, we can get an average accuracy of 60.8% (+1.0%). This improvement implies that the encoder can model the local patterns of each character by using D and multi-scale fusion. On the other hand, only adopting I2CL obtains the accuracy of 60.6% (+0.8%) which means learning long-term memory units is effective for solving LISV and SICV. When employing them together, we further improve the performance to 61.6% (+1.8%). These results demonstrate that our CFE can utilize the enriched features to enhance the discriminability of characters and that CACE and I2CL can cooperate well to optimize our model.

Moreover, Table 6 compares our I2CL with the Character Contrastive loss (CC loss) designed by CornerTransformer. The result indicates that our I2CL surpasses the conventional contrastive learning method CC loss. Compared to the performance without contrastive learning loss, the introduction of CC loss leads to a decrease of 0.9% (59.9%). We argue that achieving small intra-class differences and large inter-class differences is impractical by clustering characters only within a batch. Hence, CC loss fails to learn the global feature distribution of each category. In contrast, our I2CL employs the trainable long-termmemory units to represent the global distribution of each category, aiding the model in better achieving intra-class compactness and inter-class separability. This verifies that our I2CL can enhance the global character feature distribution for solving challenging STR.

The Components of CACE

To evaluate CACE, we compare the results with decay matrix D or multi-scale fusion in Table 7. The first row which only relies on Lcl for training, obtains a performance of 60.6%. When introducing D in the block, the model exhibits 0.5% (61.1%) improvement. The inclusion of D enables the encoder to perceive the local patterns within the character and generate discriminative features for recognizing characters.

However, when only adding multi-scale fusion, CFE-Base experiences a decrease of 0.4% (60.2%). We argue that relying solely on multi-scale fusion leads the model to ignore the local character-level features, decreasing performance on challenging datasets. A total improvement of 1.0% (61.6%) is achieved when employing them together. We attribute the improvement to two reasons: 1) Decay matrix allows the encoder to perceive the morphological information at character level, helping to distinguish the characters with different appearances. 2) Multi-scale fusion provides diverse character features by integrating the features from the three stages.

The Different Options of Decay Matrix D

In Table 8, we study different options of yielding D and find that option 2 achieves the best performance. When considering the tokens i and j are components of a character, option

2 generates a larger Di,j. So option 2 can focus more on the character region than option 1. The accuracy rates of option 2 is 61.6%, slightly higher than that of option 1 (61.4%). However, option 3 performs less competitively than options 1 and 2 with 60.0% accuracy. The reason is that dynamic decay will generate more precise attention region for each token.

4.6 Visualization and Analysis

The Visualization of CACE

From the perspective of understanding how to focus on the character region, we visualize the self-attention scores in Figure 4. Specifically, we use the token that corresponds to the red point as the query to calculate the attention scores with all visual tokens and then reshape them to 2D. For the first three rows, while focusing on the character region, the baseline also interacts with nearby characters, leading to recognition errors. This problem can be put down to insufficient exploration of the local patterns. In contrast, CACE recognizes the texts correctly by concentrating exclusively on the character region. For the last two rows, the baseline is prone to interference from distant noise, ultimately causing errors. Conversely, CACE avoids interference from distant noise, ensuring accurate recognition. Through this, we conclude that CACE can perceive the region of each character and then model the components of the character.

The Qualitative Analysis of I2CL

To evaluate the effectiveness of our I2CL and justify its design, we use t-SNE [Van der Maaten and Hinton, 2008] to reduce the feature dimension to 2D space for visualization. Figure 5 illustrates the feature distribution of 9 easily misrecognized characters. In Figure 5(a) without I2CL, we observe that the distribution of different category characters is severely mixed, as symbolized in the red rectangle. After introducing I2CL, the phenomenon of mixed distribution can be alleviated. In addition, the distance of characters between different categories is widened as shown in Figure 5(b). By training a discrete distribution for long-term memory units, all characters of different categories can be compactly distributed around the memory units, while ensuring the separability between classes. These phenomena prove the effectiveness of I2CL and are consistent with our design.

5 Conclusion

In this paper, we notice the two problems of Large Intra-Class Variance (LICV) and Small Inter-Class Variance (SICV) in challenging texts. To address these issues, a Character Features Enriched model (CFE) is proposed to obtain the discriminative features via Character-Aware Constraint Encoder (CACE) and Intra-Inter Consistency Loss (I2CL). Firstly, CACE enables visual tokens to perceive character morphological information by introducing the decay matrix, which enhances the discriminability of character features. Secondly, I2CL helps achieve intra-class compactness and inter-class separability by learning a discrete distribution for long-term memory units. The experimental results show that CFE can not only effectively improve the performance on challenging texts, but also maintain high accuracy on simple texts, taking a further step toward STR with strong robustness.

Acknowledgments

This work is supported by the National Key Research and Development Program of China (2022YFB3104700), the National Nature Science Foundation of China (62121002, U23B2028, 62102384). This research was support by the Supercomputiong Center of the USTC. We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.