Self-Supervised Pre-training with Symmetric Superimposition Modeling for Scene Text Recognition

1 Introduction

Reading text from images is a fundamental and valuable task in computer vision with practical applications such as multimodal analysis, visual search, self-driving cars, and more. Since labeled real text images are scarce and expensive, various self-supervised text recognition methods have been utilized to exploit the intrinsic knowledge of unlabeled real data to alleviate the data scarcity issues. These self-supervised text recognition methods can be categorized into two main types: 1) Sequence Contrastive Learning (SCL), and 2) Mask Image Modeling (MIM). Benefiting from representation learning on unlabeled data, these methods have effectively enhanced Scene Text Recognition (STR) performance. However, these methods face the challenge of achieving linguistic learning, which is proven essential for text recognition (e.g. ABINet [Fang et al., 2021] , LPV [Zhang et al., 2023]).

∗ Corresponding Author

For SCL methods, SeqCLR [Aberdam et al., 2021] and CCD [Guan et al., 2023] are representative works. As shown in Fig. 1(a), SeqCLR ensures the local representation consistency between the same instance window across the two augmented views. Fig. 1(b) shows that CCD [Guan et al., 2023] further ensures the character-level representation consistency based on the self-supervised segmentation. Hence, both of the two methods essentially focus on performing discriminative consistency learning on local character representations.

For MIM-like methods, MAERec [Jiang et al., 2023] has attracted considerable attention in self-supervised text recognition. As discussed in MAERec, MIM-like methods essentially forces the model to infer the whole character from a few smallest parts of a character, due to covering a large portion of the text image. However, masking 75% image patches drops nearly all text foreground areas, as shown in Fig. 1(c). Thus, MIM-like methods also focus on local character features but overlook linguistic information (spelling rules between characters) in the text image.

Based on the analysis above, we can derive an observation that the previous self-supervised STR methods mainly focus on learning robust visual features of characters, overlooking the linguistic relationship between characters. Hence, it is meaningful to simultaneously capture character features and the implicit linguistic information in visual space.

To this end, we propose a novel self-supervised learning paradigm, named Symmetric Superimposition Modeling (SSM). The pretext task of SSM is to reconstruct the direction-specific pixel and feature signals from the symmetrically superimposed input. We adopt a Siamese network with an online branch and target branch to implement SSM. Specifically, we first construct inverted images by randomly selecting from three inversion enhancement techniques: horizontal flip, vertical flip, and 180-degree rotation. Then we superimpose it onto the origin image to create the symmetric superimposed input. For pixel reconstruction, we directly recover the original and inverted images (with online branch), as shown in Fig. 1(d). The original and inverted images are the pixel targets to guide the decoupling of the superimposed input. For feature reconstruction, we utilized the target branch to decouple the same symmetric superimposed input with irregular views, creating the original and inverted target feature on the fly. Subsequently, the online branch reconstruction the original and inverted target feature of the irregular view at the semantic level. We jointly use discriminative consistency loss and dense reconstruction loss to supervise the feature reconstruction process. Compared to MIM-like methods, we do not mask any patches and skillfully use symmetric superposition to disrupt character shapes and linguistic rules. Consequently, the pretext task of pixel and feature reconstruction from the superposed input can facilitate the learning of character shape features and linguistic information from the perspective of visual texture and feature semantics. In summary, our contributions mainly include:

• We propose a novel pre-training framework based on Symmetric Superimposition Modeling, which is the first self-supervised STR method dedicated to linguistic learning in visual space.
• We present a dual architecture for the joint reconstruction at both the pixel and feature level. This design enables joint learning of character visual features and implicit linguistic information from the texture-level and semantic level, further improving representation quality.
• Experiments demonstrate that SSM achieves state-ofthe-art performance on various text recognition benchmarks. The 4.1% average performance gains on various STR methods also highlight the SSM's generality. Additionally, compared to other self-supervised methods, SSM has a 15.5% and 1.5% performance gain in multilingual text recognition with individual training and joint training settings, respectively.

2 RelatedWork

2.1 Text Recognition

Scene Text Recognition (STR) methods can be summarized into language-free and language-aware methods. Languagefree methods [Shi et al., 2017; Wang and Hu, 2017; Du et al., 2022] treat STR as a character classification task, focusing on how to extract robust visual features. Languageaware methods can leverage linguistic information to improve robustness. The attention-based methods [Yu et al., 2020; Litman et al., 2020; Lee et al., 2020; Sheng et al., 2019] use various attention mechanisms to implicitly model linguistic rules. LISTER [Cheng et al., 2023a] designs neighbor attention decoding for long text recognition. MGP-STR [Wang et al., 2022] and PARSeq [Bautista and Atienza, 2022] focus on learning an internal Language Model(LM) during visual predicting. Some alternative approaches like ABINet [Fang et al., 2021] and LevenOCR [Da et al., 2022] propose to refine the visual predictions via an external LM.

2.2 Self-Supervised Text Recognition

Self-supervised learning technologies have been widely used for various works such as ML-LMCL [Cheng et al., 2023b], TKDF [Cheng et al., 2023c] and MESM [Liu et al., 2024]. The most representative paradigms for computer vision are Contrastive Learning [Chen et al., 2021] and Masked Image Modeling (MIM) [He et al., 2022; Bao et al., 2022; Chen et al., 2023]. Recently, these self-supervised methods have been adopted for text recognition. SeqCLR [Aberdam et al., 2021] is the pioneering work that proposes to model Contrast Learning on high-level sequence features for the first time. PerSec [Liu et al., 2022] further performs Contrast Learning(CL) on both stroke-level features in shallow layers and Semantic-level features in deep layers. Later, DiG [Yang et al., 2022] proposes to learn discriminative and generative features by integrating the CL and MIM. To explicitly focus on character structures and ensure sequence consistency, Guan proposes a character-level self-distillation framework [Guan et al., 2023] based on unsupervised text segmentation maps. Recently, Union14M-U[Jiang et al., 2023], a 10M scale unlabeled real data has been presented together with MAERec-S, which uses MAE for self-supervised text recognition. In addition, MaskOCR [Lyu et al., 2022] and DARLING [Zhang et al., 2024] use synthetic data for text recognition pre-training.

3 Methodology

The architecture of our proposed SSM is shown in Fig. 2, which comprises a Symmetric Superimposed Input Constructor, an online branch (blue arrow), and a target branch (orange arrow). We use the superimposed image of the original image and the horizontally flipped image as the input case to illustrate the whole workflow of our SSM.

3.1 Symmetrically Superimposed Input

Symmetrically Superimposed Input Constructor To ensure a large character overlapping area of the superimposed inputs, we considered three symmetrical augmenting operations: Horizontal Flipping Hf(·), Vertical Flipping Vf(·), and

180-degree rotation Ro(·). For the input image X, we randomly select one of Hf(·), Vf(·), and Ro(·) to obtain the corresponding inverted image XR, where XR ∈ {Xh f , Xv f , Xro}. For the original image X, the original direction index Ip is fixedly set to 0. For the inverted image XR, the inverted direction index In is selected from {1, 2, 3} according to the type of inverted image, where (1, 2, 3) is assigned to (Xh f , Xv f , Xro), respectively. These indexes are subsequently encoded to guide direction-specific reconstruction. Finally, we get the symmetrically superimposed input XS by superimposing the original image X and its inverted images XR. In the case of Fig. 2, Xh f is selected and the In is set to 1 (red color).

Data Augmentation The superimposed image XS undergoes weak augmentations (e.g., gaussian blur, and grayscale conversion) to create a regular view X reg S as the final input of online branch. The X and XR are also transformed to X reg and X reg R to supervise the image reconstruction of the online branch. For the target branch, we utilize the combination of both weak augmentations and geometry-based augmentations (e.g., affine transformation and perspective warping) to generate a pair of irregular views: X irr S , X irr and X irr R .

3.2 Symmetric SuperimpositionModeling

1) Pixel-level Image Reconstruction We employ an Encoder-Regressor-Decoder architecture to perform the image reconstruction according to the specific direction index pair in the online branch. Specifically, we first leverage the ViT Encoder F (·) to map the X reg S as latent feature Fs ∈ R HW p2 ×d , where the p is the patch size and the d is the embedding dim. Meanwhile, we utilize the Prompt Generator G(·) to encode the direction index pair (Ip, In) into the direction prompt token pair (Tp, Tn) with the same dimension of the ViT Encoder.

where the Prompt Generator G(·) consists of an embedding layer, two-layer FFN with normalization.

After that, Tp and Tn are each concatenated with the latent feature of superimposed input Fs and then sent into the Transformer Regressor R(·) for feature decoupling in symmetric directions. The Transformer Regressor R(·) is a series of vision transformer blocks. Thanks to the global interaction capabilities of the attention mechanism, Regressor R(·) can extract the original direction feature Fp and the feature Fn corresponding to symmetrically reversed direction from the mixed features Fs. The process can be formulated as follows:

Next, we use a lightweight pixel decoder to predict the RGB pixels of the corresponding views X reg and X reg R from latent features Fp and Fn. To prevent the pixel decoder's parameters from dominating the learning process, we form the pixel decoder using only two linear layers with GELU. Finally, the L2 loss function is adopted to optimize the image reconstruction process, and the complete loss function of pixel-level image reconstruction can be defined as follows:

where y i p , ˆy i p , ∈ R 3 and y i n , ˆy i n ∈ R 3 are the prediction-target pair of x reg p and x reg n , respectively; N is the number of pixels. 2) Feature-level Representation Reconstruction To enhance the feature discriminability of character semantics and to model the spatial context between character visual semantics, we apply feature reconstruction in high-dimensional space. Specifically, we add the projector H(·) followed by the Regressor R(·) in the online branch. H(·) aggregates the decoupled features (Fp and Fn) into window-level features by adaptive mean-pooling and then maps them into high-level feature space, forming the corresponding prediction feature queries (Qp and Qn). To construct the target representation, we introduce a target branch with the same structure as the online branch, except for the pixel decoder. X irr S is fed into the target branch to obtain the window-level target feature keys (Kp and Kn) on the fly. Next, we jointly employ the discriminative consistency loss Ldis and dense reconstruction loss Lden to supervise the reconstruction from Qp to Kp and Qn to Kn across the regular and irregular views. The loss Ldis is to enhance the character classification representation at the semantic level, which can be formulated as:

where K + stands for the matched positive samples and K - indicates the negative samples collected from the same batch for both (Qp,Kp) and (Qn,Kn). τ denotes the temperature.

Since SiameseMIM [Tao et al., 2023] demonstrates that predicting dense representations helps improve sensitivity to global structures, we minimize the distance between the predicted and the target features by Lmse loss, aiming to achieve semantic-level context modeling. Assuming that predictions are Q = {q i ∈ R d |i = 1, ...,N} and targets is K = {k i ∈ R d |i = 1, ...,N}, where N is the total number of feature instances in one batch. The dense reconstruction loss Lden (Lmse) between the original Q-K pairs (Qp and Kp) as well as inverted Q-K pairs (Qn and Kn) can be formulated as:

Finally, the optimization objectives of SSM are as follows:

where α is a scaling factor, the training objective of semanticlevel feature reconstruction Lf eat is the sum of Ldis and Lden.

3.3 DownstreamTasks

For text recognition downstream tasks, we add a text decoder after the ViT Encoder inherited from SSM. The text decoder consists of 6 transformer blocks and a linear prediction layer with 96 channels to predict the final characters. Besides, the text super-resolution and text segmentation results are presented in Appendix B.1 and B.2.

4 Experiments

4.1 Datasets

Unlabeled Pre-training Data We utilize the latest unlabeled real-scene dataset Union14M-U for self-supervised learning, which contains 10 million instances collected from Book32, OCR-CC and OpenVINO. Besides, we also conduct pretraining on the complete OCR-CC dataset(15.77M unlabeled text images) to facilitate a fair comparison with works such as CCD [Guan et al., 2023] and DiG [Yang et al., 2022].

Text Recognition Fine-tuning Data We use three types of labeled data. 1) STD: The synthetic data, comprising 14M images fromMJSynth [Jaderberg et al., 2014] and SynthText [Gupta et al., 2016]. 2) ARD: 2.78M annotated real data used by DiG and CCD. 3) Union14M-L: 3.6M real labeled data.

Scene Text Recognition Benchmarks The Common benchmarks include IC13, SVT, IIIT5K, IC15, SVTP and CUTE80. Considering the saturation issue with existing benchmarks, we leverage Union14M benchmarks [2023] to evaluate. Some other challenging data such as TT, CTW, ArT, Uber,

and COCO are also utilized to test.

Multilingual Text Recognition Benchmarks. Following MRN [Zheng et al., 2023], we tested the multilingual generalization capability of SSM on MLT19 [Nayef et al., 2019].

4.2 Implementation Details

Self-supervised Pre-training The pre-training is conducted on ViT, with image resolution of 32 × 128, an AdamW optimizer, cosine learning rate scheduler with a learning rate of 5e-4, batch size with 1,024, a weight decay of 0.05, β1 = 0.9, β2 = 0.95, and warm-up for 1 epoch in a total of 20 epochs. Text Recognition Fine-Tuning Our text recognition network is fine-tuned with STD or ARD or Union14M-L dataset. Patch size is 4 × 4. The text decoder consists of a 6-layer transformer block with an embedding dimension of 384. The batch size is 384 and the warm-up time is 1 epoch. The AdamW optimizer and a OneCycle learning rate scheduler with a learning rate of 1e-4 are employed.

4.3 Comparasions With Self-SupervisedMethods

Evaluation on Common benchmarks. Tab. 1 shows that SSM-ViT-Tiny outperforms the SeqCLR by 13.6% and 9.4% on IIIT and IC13, as well as outperforms PerSec-ViT (using 100M private data for pre-training) by 9% on average accuracy. Besides we compare with previous state-of-art self-supervised methods DiG and CCD, using the same pretraining data, fine-tuning data (ARD), and network architectures (i.e., Tiny, Small). Tab. 1 illustrates that our methods achieve a new state-of-the-art average word accuracy on common benchmarks with 95.1% and 96.4% at the same model size. When fine-tuning with ARD, our methods achieve average performance gains of 0.9% and 0.8%. SSM-ViT-small even beats the CCD-ViT-Base which has a larger model size (96.4% vs 96.3%). The CCD attains high performance with explicit segmentation guidance, while our straightforward method surpasses it without needing extensive segmentation GT label preparation. These results demonstrate the effectiveness of guiding text image self-supervised pre-training from a linguistic learning perspective.

Evaluation on Union14M benchmarks. Considering the saturation of the common benchmarks, we further compare SSM with other self-supervised methods on more challenging benchmarks in Tab. 2. Although we established a strong baseline model Scratch-ViT-Small, and SSM-ViT-Small still achieved an average performance gain of 3.5% and 2.6% on Union14M and other challenging data, respectively. Additionally, SSM surpassed MAE and DiG by 1.8% and 1.3% of average word accuracy on the Union14M benchmarks. The performance gains on all of Union14M benchmarks' subsets demonstrate that SSM effectively learns intrinsic feature representations for different fonts, text orientations, and complex scenes. Surprisingly, SSM shows superior performance for contextless text images, which may be attributed to the fact that embedding linguistic learning in visual space can avoid the text distribution dependence of explicit text correction.

Table 3: Comparison with other self-supervised methods on MLT19. † stands for the reading of Arabic is naturally right-to-left.

Multilingual Text Recognition Performance. To further validate SSM's ability to learn linguistic information and its generality, we compare SSM with other self-supervised methods on MLT19 benchmarks. All methods are pre-trained on the Union14M-U for a fair comparison. In Tab. 3a, we finetune them on each language data one by one. Tab. 3a shows that our SSM outperforms the second-best method DiG by 15.5% in the average accuracy (57.0% v.s. 41.4%). Besides, for the right-to-left Arabic language, SSM even outperforms the second-best method by 27.4% (76.0% v.s. 48.6%), which is attributed to that SSM can capture linguistic information from right-to-left texts during the pre-training phase. These results demonstrate that SSM is an effective pretext task to capture linguistic information in multilingual texts and exhibits strong generalization and few-shot capabilities for datascarcity languages. Tab. 3b shows that all the methods benefit from jointly learning linguistic information of different languages, yet SSM still exhibits superior performance with 1.5% average accuracy gains compared to DiG.

4.4 Ablations and Analysis

Effectiveness of architecture. The third row of Tab. 4 indicates that there is a 3.0% average performance improvement on Union14M benchmarks with the pixel-level reconstruction, which need the combined effect of prompt generator G(·) transformer regressor R(·) and pixel decoder D(·). However, removing the regressor R(·) results in a slight performance downgrade. The result demonstrates that the transformer regressor plays an indispensable role in SSM by guiding and disentangling the features of the original and inverted images. Based on the pixel reconstruction, the Joint use of the EMA mechanism and project module H(·) further brings accuracy gains of 1.3% as it promotes learning discriminative character semantics and modeling spatial context at the semantic level. Here, H(·) is mainly used to prevent excessive background noise in negative sample sampling and to map the decoupling feature to high-dimensional space. Lastly, the addition of Irregular view augmentations improves the average test accuracy up to 84.3% due to diverse perspectives.

Ablations on the types of superimposed input. We compared our superposition strategies with add noise & blur and randomly add another text image. As shown in Tab. 5a, the performance gains of add another text image are close to that of add noise & blur, both of which are lower than existing superposition strategies. We attribute this to the fact that the model can easily reconstruct the two images with inconsistent text content and different background styles from low-level pixel differences, limiting the exploration of intrinsic charac-

ter semantics and linguistic information. We also found that individually superposing a 180-degree rotation view is more effective than individually superposing a specific flip view. The result suggests that the rotation influences character orientation and sequence reading order, helping capture learning richer character semantics and spatial context relationships during reconstruction. Finally, the best performance (84.3%) is achieved by combining HS, VS, and RS strategies, which improve the diversity of the superimposed input.

Ablations on the feature-level reconstruction loss. The first and second rows of Tab. 5b shows that only using distance measure functions such as MSE Lmse and Cosine Lcos results in lower word accuracy (82.5% vs 94.3%). We attribute this to the fact that only minimizing the feature distance between predictions and targets may lead to feature collapse. Besides, Tab. 5b shows that our discriminative consistency loss Ldis outperforms the dino loss Ldino by 0.3%. The combined use of Ldis and Lmse ultimately yielded the best results. Ldis can mitigate the feature collapse of Lmse by reducing the similarity between negative samples. Besides, Ldis focuses on enhancing discriminative features for character semantics while Lmse is more sensitive to spatial structures, which helps model linguistic information in spatial context at the semantic level.

Table 5: Ablation studies on the input type and consistency loss. HS, VS, and RS respectively means superimposing HFlip, VFlip, and Rotation View with the original image.

71.8

71.1

With Teacher Branch

Generative Pretext Task

70

75

Feature representation evaluation. We freeze the encoder and train the decoder with ARD, following the test setting of DiG. Fig. 4 shows that the pixel-level reconstruction of SSM (SSR-S) has better feature representation than MAE-S (61.2% vs59.3%). By introducing feature-level representation reconstruction, SSM outperforms DiG by 0.7%. We attribute this to strengthening the implicit linguistic representation through texture-level and semantic-level reconstruction. Fine-tuning with different data ratios. We fine-tune our method with 1%(27.8K), 10%(278K), and 100%(2.78M) of ARD. Fig. 5 shows that SSM-ViT-Small outperforms the previous SOTA methods by 2.3%, 2.6% and 1.3%, respectively. Improvement of Various STR methods. Benefiting from masking-free operations, our method is not troubled by information leakage during feature downsampling. Consequently, it can be flexibly applied to various types of STR Methods, including Conv-based methods (e.g., ABINet-V, ASTER, and CRNN), multi-scale attention methods (e.g., SVTR), and vanilla ViT methods (e.g., our baseline ViT-*, and PARSeq). Fig. 6 shows that SSM can bring a performance gain of 4.1% to the above STR method on Union14M benchmarks.

Qualitative recognition results. The top and bottom strings of Fig. 7 are predicted by DiG-S and SSM-S, with red indicating errors. Results show that SSM is robust to reversed text and complex-textured images. It also can infer complex characters based on contextual linguistic information.

4.5 Comparisons With State-of-the-Arts

In Tab. 6, we compare SSM and previous SOTA text recognition methods. Specifically, SSM-ViT-Tiny outperforms all supervised state-of-the-art (SOTA) methods (80.7% vs 76.9%) on more challenging Union14M benchmarks with only 20M parameters. Moreover, SSM-ViT-Small outperforms the other self-supervised pre-train methods with the same model size by 0.6% and 1.3% average performance. Surprisingly, by training 20 epochs, SSM-ViT-SmallTurbo pushes the new SOTA average performance on Common benchmarks and Union14M benchmarks to 96.5% and 86.6%, respectively. Note that SSM-ViT-Small-Turbo outperforms MAERec-B by 0.3% and 1.4% with only one-fourth of the parameters of the MAERec-B (36M vs 142M).

5 Conclusion

In this paper, we propose a novel self-supervised text recognition framework, termed SSM. In contrast to previous selfsupervised text recognition methods that only focus on local visual features, SSMmodels the global spatial context for implicit linguistic learning. SSM aims to reconstruct directionspecific pixel and feature signals from symmetrically superimposed text images. In this way, SSM captures both the local character feature and implicit linguistic information. Eventually, SSM improves the text recognition performance of various Scene Text Recognition methods and refreshes state-ofthe-art performance on various text recognition benchmarks.

Acknowledgments

This work is supported by the National Key Research and Development Program of China (2022YFB3104700), the National Nature Science Foundation of China (62121002, U23B2028, 62102384). This research is supported by the Supercomputing Center of the USTC. We also acknowledge the GPU resource support offered by the MCC Lab of Information Science and Technology Institution, USTC.