The Devil is in Fine-tuning and Long-tailed Problems: A New Benchmark for Scene Text Detection

1 Introduction

Scene Text Detection (STD) has received extensive attention both in academia and industry due to its wide applications, such as scene understanding [Zhang et al., 2025; Lyu et al., 2025], intelligent office [Zeng et al., 2023; Shen et al., 2024; Shen et al., 2023], etc. Researchers have proposed many prominent scene text detectors [Zhou et al., 2017; Long et al., 2018; Wang et al., 2019; Liao et al., 2020; Liu et al., 2021; Du et al., 2022; Qin et al., 2023; Bu et al., 2024]. Figure 1 (top) shows the year-over-year performance improvement of scene text detectors across three public benchmarks. Recent advances in scene text detection show

� Corresponding Author.

-10

Figure 1: (top) The performance tendency of scene text detectors in recent years. We choose several representative methods in chronological order. Circles indicate the performances of other methods. (bottom) A visualization of fine-tuning gap. Each bar shows a performance comparison on the same benchmark. "∗" denotes we finetuned the model with the open-source pretrained weight.

consistent improvements in F-measure scores, with recent models surpassing 90%, reflecting rapid progress in the field.

Despite the impressive benchmark results, scene text detectors still face challenges (e.g., texts within bad illumination and complex backgrounds) and fail to meet the expectations in real-world applications. The significant discrepancy between the high performance on academic benchmarks and the subpar performance in practical deployments prompts our investigation into the underlying causes. We attribute this to two main factors: fine-tuning gap and long-tailed problem.

Firstly, previous evaluations of scene text detectors mainly rely on the Dataset-Specific Optimization (DSO) paradigm, where detectors are pretrained on a variety of datasets and then fine-tuned on a single dataset's training subset, followed by assessment on its corresponding test subset. However, such a protocol might not reflect authentic generalization ability, as each dataset exhibits unique biases and domainspecific characteristics. Fine-tuning on a biased training subset could lead to overfitting to a specific domain, resulting in degraded performance on others. As shown in Figure 1 (bottom), a model consistently improves its performance on the test set after fine-tuning with the corresponding training data. Conversely, its performance drops significantly, with a decrease of up to 8.8% when fine-tuned on one dataset and evaluated on another. Evaluations on limited benchmarks reflect the detector's effectiveness for specific text types but fail to assess its overall performance in diverse scenarios. Previous works [Zhan et al., 2019; Tian et al., 2022] also refer to the domain-adaptation problem in STD. To address this issue, we propose a Joint-Dataset Learning (JDL) method, where a detector is trained on a unified training set from multiple datasets and directly evaluated on a combined test set. We posit that JDL offers a more accurate and comprehensive performance assessment.

Secondly, scene text categories commonly exhibit a longtailed distribution in real-world scenarios, with dominant head categories (e.g., printed text) and scarce tail categories (e.g., artistic text). We define long-tailed problems as challenges where detectors struggle with tail categories due to the lack of relevant training samples, resulting in substantial performance degradation on less frequent but critical text instances in practical applications. However, most existing studies focus on specific challenges and overlook long-tailed issues from a holistic perspective. In light of this, we analyze and define a comprehensive set of challenges, as shown in Figure 2. We evaluate four representative scene text detectors (CNN-based and Transformer-based) on three datasets with distinct styles (for diverse scenarios). Results reveal that the detectors fail to identify 7% of text instances. Among these undetected instances, only 65% are found to be genuinely challenging, while the remaining 35% are attributed to annotation errors or involved non-Latin scripts. We identify recurring patterns across these challenging cases and categorize them into three main groups: intra-instance, inter-instance, and background problems. These groups are further divided into 13 subcategories, each representing a long-tailed challenge in scene text detection. To support future research, we introduce a Long-Tailed Benchmark (LTB) as a standardized platform for evaluating the ability of scene text detectors to address these challenges.

Finally, inspired by MAERec [Jiang et al., 2023], we utilize self-supervised learning (SSL) for better text representation and propose a baseline based on MAE [He et al., 2022] from the data perspective, called MAEDet. In conclusion, our contributions are three-fold.

• We identify the devil of the discrepancy between the performance on academic benchmarks and that on real applications lies in fine-tuning and long-tailed problems.
• We introduce the Joint-Dataset Learning (JDL) protocol as a replacement for Dataset-Specific Optimization to avoids the fine-tuning gap and provides a more accurate evaluation of model generalization across different domains.
• We propose the Long-Tailed Benchmark (LTB), which poses 13 distinct challenges and serves as the first benchmark to comprehensively evaluate models' capability to address various long-tailed problems in scene text detection. A baseline based on self-supervised learning for the benchmark is also provided.

2 RelatedWorks

2.1 Approaches for Scene Text Detection

CNN-basedmethods. [Gupta et al., 2016; Jaderberg et al., 2016; Liao et al., 2017; Liao et al., 2018] are followed the object detection method to detect the scene text. [He et al., 2017] and its variants [Xiao et al., 2020; Du et al., 2022; Ye et al., 2020] represent the oriented and curve text contours. [Long et al., 2018; Liu et al., 2020; Wang et al., 2020; Wang et al., 2022; Liu et al., 2023; Su et al., 2024] also explore the efficient representation operators.

Transformer-based methods. Attributing to the broad application in visual tasks, Transformer, as the fundamental architecture, has been gradually embedded in the scene text detector [Raisi et al., 2021; Tang et al., 2022; Liu et al., 2023; Bu et al., 2024]. [Raisi et al., 2021] firstly introduces the Deformable DETR [Zhu et al., 2021] to detect oriented text instances. To alleviate the burden of the Transformer, [Tang et al., 2022] samples the features of text instances and reduces the computational cost. [Ye et al., 2023] notices the reading of inverse-like scene text. [Bu et al., 2024] combines the advantages of top-down and bottom-up and proposes a transformerpowered text detector.

2.2 Benchmarks for Scene Text Detection

Thanks to the Robust Reading Contest, massive and diversified datasets are proposed and evaluated fairly, such as ICDAR 2013 [Karatzas et al., 2013] for axis-align texts, ICDAR

2015 [Karatzas et al., 2015] for oriented and unfocused texts, ArT [Chng et al., 2019] for arbitrarily shaped texts, and MLT [Nayef et al., 2017; Nayef et al., 2019] for multi-language texts. These datasets include thousands of text images for training and testing. LSVT [Sun et al., 2019] and TextOCR [Singh et al., 2021] collect larger-scale annotated datasets. Additionally, some long-tailed problems in scene text detection have been noticed in recent years. For example, SCUTCTW1500 [Liu et al., 2017] is designed to handle curved text with line-level granularity. DAST1500 [Tang et al., 2019], HierText [Long et al., 2022], and InverseText [Ye et al., 2023] solve the problems in dense, multi-hierarchical and inversed texts, respectively. Our current research focuses on English word-level texts, with prospects for future expansion to additional languages and varied granularity.

3 Addressing the Fine-Tuning Gap

Traditional scene text detection methods often adopt the DSO paradigm, which can be formulated as follows. Given N datasets D = {Di | i = 1, 2, · · · ,N}, where each dataset Di consists of a training set Di train and a testing set Di test , a pretrained model Mpretrain is first obtained by training on largescale data (typically synthetic data or a combination of multiple real datasets). The pretrained model is then fine-tuned separately on each training set Di train to produce N fine-tuned models Mifinetune , which are finally evaluated on their corresponding test sets Di test , as illustrated in Figure 3 (left).

However, different benchmarks are tailored to solve various issues (e.g., oriented text detection, curved text detection) and exhibit significant variability due to diverse scenarios and data collection methods, leading to substantial dataset biases. A comprehensive analysis of these variations is provided in Appendix. This issue is further exacerbated by the DSO protocol, where fine-tuning a model on a single training set Di train might not adequately capture the overall distribution of realworld scenarios, leading to performance degradation on other test sets D\i test .

We define this limitation as the Fine-tuning Gap: the discrepancy between a model's performance on its fine-tuned dataset and its ability to generalize to unseen data distributions. To verify the fine-tuning gap, we conduct experiments using CNN-based ABCNet v2 and transformer-based DPText-DETR, with the results presented in Table 1. As shown in the first two columns, fine-tuning yields at least a 2.0% improvement on the target benchmark. However, the last two columns reveal degraded performance when models are fine-tuned on another dataset, which occurs irrespective of the detector architecture, whether CNN-based or Transformer-based. These findings underscore that the DSO approach is ill-suited for actual applications due to several critical issues: 1) Lack of generalization. Detectors fine-tuned on a single dataset struggle to adapt to real-world scenarios' diverse and variable conditions, often resulting in overfitting. 2) Resource inefficiency. Separate fine-tuning for each dataset requires substantial computational and time resources, significantly limiting scalability and practicality. 3) Unrepresentative benchmarking: Evaluating a model on a single, limited test set fails to provide an objective and comprehensive assessment, potentially leading to misleading conclusions.

To address the aforementioned issues, we introduce JointDataset Learning (JDL) protocol to STD, as illustrated in Figure 3 (right). JDL draws inspiration from previous advancements in scene text recognition and offers a feasible solution from data perspective. The protocol involves training a model Mjoint on the combined diversity of multiple datasets ⋃Ni=1 Di train , and directly evaluating it on ⋃Ni=1 Di test . This unified paradigm offers a more efficient training framework and a more equitable assessment mechanism for STD in practical applications.

We suggest that this new protocol provides a more comprehensive and accurate reflection of a model's adaptability to the complexities of real-world scenarios. To provide a more comprehensive view of the advantages and implications of Joint-Dataset Learning, we delve deeper into its importance and feasibility in Appendix.

4 A Long-tailed Benchmark

Except for the fine-tuning gap, scene text detection also suffers from the problem of long-tailed distribution. Text instances of tailed categories rarely exist in the training set, bringing obstacles to vanilla text detectors. In this section, we will define the common problems of current detection systems, and introduce our approach to design a benchmark to address them.

4.1 Challenge Definition

To uncover the challenges encountered in scene text detection, we first apply the four representative text detectors mentioned in Table 2 to the test sets of ICDAR2015 [Karatzas et al., 2015], Total-Text [Ch'ng and Chan, 2017], and ArT [Chng et al., 2019]. By analyzing the failure cases of these detectors, we identify recurring features among the challenging texts. Based on the comprehensive analysis, we categorize the undetected issues into three main groups: intrainstance problems, inter-instance problems, and background problems. Each category poses specific issues that impact text detection in distinct ways. Figure 4 visualizes these categories and their relationships.

https://github.com/Yuliang-Liu/bezier curve text spotting https://github.com/ymy-k/DPText-DETR

Intra-instance Problems

Intra-instance problems refer to challenges associated with the internal attributes of independent text instances. In other words, these problems derive from the low-quality appearance of texts rather than influence from adjacent texts or surrounding backgrounds.

Blurred Text. Blurred text may arise from out-of-focus conditions, like camera shake, limited camera performance, or image noise. Especially in scene images, texts are often not dominant and easy to be unfocused. As shown in Figure 4(a), blurred text is prevalent in scene text.

Artistic Text. Artistic texts frequently appear in graffiti, posters, and advertisements, characterized by elaborate but unconventional designs. These texts may feature highly unique fonts, diverse colors, and non-traditional character arrangements, such as staggered placements where characters are not aligned in a straight line. The non-standard instinct of artistic texts makes it difficult for detectors to generalize from training data, necessitating the development of advanced algorithms to handle these diverse forms effectively.

Glass Text. Glass texts refer to texts that are either directly on the surface of glass or reflected onto it. The optical properties of glass, including reflection, refraction, and distortion, present unique challenges for text detection.

Single-Character Text. In practical applications, individual characters may be large and clear yet still hard to detect because the lack of surrounding contextual clues from adjacent characters or words prevents the detection process.

Distorted Text. Text distortion may arise from several factors: capturing text from highly oblique angles, applying perspective transformations to the text image, or the existence of texts on irregular surfaces. Such misalignment and warping alter the original text proportions and geometric features, creating substantial challenges for detection systems.

Inverse Text. Inverse text occurs when a surface is visible from both sides, with one side showing the text normally and the other side displaying it in reverse. It can also result from image processing techniques like horizontal flipping. DPText-DETR[Ye et al., 2023] argues that inverse-like text instances harm the robustness of the scene text detector.

Delimited Text. A delimited text instance contains formats where specific delimiters segment the content, resulting in multiple discrete pieces rather than a unified whole. Examples include URLs (e.g., www.example.com), phone numbers (e.g., tel:123-4567), and numerical sequences (e.g., 1,0978). The segmentation can result in detection systems losing smaller delimiters and treating the segments as separate text instances, complicating the detection and analysis of overall content.

Inter-instance Problems

In addition to the challenges posed by individual text instances, the interaction among adjacent texts also exerts an influence. Specifically, dense and overlapped texts complicate text detection processes.

Dense Text. Dense text refers to instances where text is closely positioned, either in multi-line or side-by-side formats. This closeness may result in the incorrect merging of separate text instances into one, which is a common issue in document-like images.

Overlapped Text. When text instances overlap, they become partially obstructed but remain visible. This commonly occurs due to shadows cast on the text, reflections from glossy surfaces, or the presence of watermarks.

Background Problems

The challenges associated with noisy backgrounds stem from intricate or distracting patterns and colors. When backgrounds compete with text for visual attention, it complicates the detection process.

Occluded Text. Occluded text occurs when text instances are partially invisible due to being obscured by foreground objects, excessive illumination, or being located at the edge of an image where parts of the text extend beyond the image boundary. This phenomenon commonly arises in natural scenes, resulting in missing segments.

Low-Contrast Text. Low-contrast text refers to text that has a low contrast with its background. This typically occurs when the text color is similar to the background color or when the text is inadequately illuminated. Such text is challenging for humans to detect even in high-quality images, making it even more difficult for detection algorithms to identify.

Complex-Background Text.

It is difficult to discern the text boundaries when the background contains patterns that closely resemble the text or when the background features complex, multi-colored designs. For instance, text overlaid on highly textured or patterned backdrops or hollowed text that is outlined rather than filled tends to confuse text detectors.

Other Problems

Additionally, some challenges remain difficult to categorize because we lack a clear understanding of why certain texts, though evident to human observers, elude scene text detectors. These complexity and rarity cases are grouped into this category, underscoring the need for further investigation into these atypical failures.

4.2 Pipeline of Constructing Benchmark

While these unresolved challenges have been identified, existing benchmarks either do not focus on them or only concentrate on one specific problem, limiting their utility in evaluating model performance across diverse and complex scenarios. To address this gap, we introduce the long-tailed Benchmark (LTB), which consists of 924 images, including a wide range of challenging text instances in various realworld scenes. LTB provides a more rigorous and comprehensive standard for assessing the performance of text detection models.

Our strategy for data collection prioritizes two main criteria: ensuring diversity in scenes and incorporating text instances that present predefined challenges. Based on these criteria, we choose six datasets, which are ICDAR2015 [Karatzas et al., 2015], Total-Text [Ch'ng and Chan, 2017], ArT [Chng et al., 2019], InverseText [Ye et al., 2023], Occluded-RoadText, and NightTime-ArT [Yu et al., 2023]. Detailed information on these benchmarks can be found in Appendix. According to the characteristics of different benchmarks, we adopt a dual strategy for data collection: one assisted by detectors and one purely manual. For ICDAR2015, Total-Text, and ArT, we obtain detection results of 4 well-finetuned detectors, as outlined in Table 2. We then develop an algorithm to retain undetected text instances. For the remaining three datasets, we manually collect challenging samples due to the absence of fine-tuned models demonstrating superior performance. Appendix elaborates our data processing strategy of LTB.

4.3 Baseline

Inspired by CLIP [Radford et al., 2021] and MAE [He et al., 2022], we adopt self-supervised learning to address the long-tailed problems from the data perspective, termed as MAEDet. As shown in the Figure 5, during the stage of SSL, given an image I ∈ Rh×w×3 and its randomly masked version Imask, the model reconstructs the input as I rec . To focus on the text region and model the text feature better, a guidance mask Mg is generated by the frozen CLIP. Specifically, Mg is the attention map, and M is the binary map with T, which is the threshold for discriminating between the text region and background, and "Text" is the prompt of the text encoder of CLIP. Considering reconstruction bias between the text region and the background, we add an auxiliary branch to optimize MAEDet with the pixel-level balanced reconstruction loss Lbr, formulated as Equation (1), where α is the balance factor, 1 is the indicator function, and L2 is the L2 loss function. At the supervised learning stage, MAEDet is initialized from the pretrained backbone and optimized using detection loss Ldet.

https://rrc.cvc.uab.es/?ch=29

5 Experiments

In this section, we will first verify the performances of the representative detectors on mainstream benchmarks with the proposed JDL paradigm. Subsequently, the LTB benchmark is leveraged to evaluate further the long-tailed ability of these models, along with our baseline model MAEDet.

5.1 Experimental Setup

Implement Details. We use three different training strategies: 1) Pretrain: We utilize officially pretrained models for each detector, typically initialized with ImageNet pretraining weights; 2) Joint: Models are trained from scratch on the union of previous English word-level training sets, termed Joint98K, without loading any pretrained weights; 3) SSL: Our proposed MAEDet incorporate self-supervised learning using MARIO-LAION [Chen et al., 2024] before training on Joint98K.

Evaluation Protocols. We employ F-measure [Karatzas et al., 2015] as the standard evaluation metric, focusing solely on text instances labeled as "care" and disregarding "don't care" regions. For the LTB dataset, we report the F-measure for each of its 13 categories. Additionally, we use two overall metrics: Hard, which marks only challenging text instances as "care," and Norm, which retains the original ground truth annotations. These metrics provide complementary insights, with the Hard metric better differentiating model performance in challenging scenarios.

5.2 Main Results and Analysis

Table 3 presents a comparison between them. DBNet++ and DPText-DETR, when evaluated with the proposed JDL paradigm, achieve higher performance on more challenging datasets (COCO, ArT, LSVT, and TextOCR) than "Pretrain" settings. Additionally, the pretrained SRFormer exhibits excessively high performance on Total-Text, MLT2017, and MLT2019, likely due to data leakage, where its pertaining data includes some validation images from Joint98K.

Moreover, we also verify the effectiveness of JDL in reducing over-fitting in STD evaluation protocol between. As observed in Table 1, when DPText-DETR is fine-tuned on Total-Text, a notable performance gap of 15.1% emerges between its performance on ICDAR2015 and Total-Text. However, this discrepancy is significantly reduced to merely 6.0% when employing the JDL protocol. This substantial reduction in performance variance across different datasets substantiates the effectiveness of JDL in enhancing model generalization and alleviating the dataset-specific overfitting issue.

Table 4 presents a comparison of different training strategies on LTB. The results indicate that models trained on Joint98K achieve superior performance over their pretrained counterparts, suggesting that exposure to diverse, real-world data can positively influence model generalization under long-tailed conditions.

5.3 Ablation Studies

Exp. 1-3 in Table 5 demonstrate that appropriately increasing the input size for the LTB is beneficial for capturing challenging text instances. Mask threshold T in Equation (1) determines the proportion of text the backbone perceives during the self-supervised pretraining stage. Exp. 4-6 Table 5 show the best performance occurs when T = 0.1. We hypothesize that a lower T allows more text pixels to contribute to the loss calculation. All experiments in this ablation study are conducted with α = 1.0. Exp. 7-8 shows that after optimizing pretraining with Lbr, there is a significant improvement in performance from LTB to Joint98K, demonstrating that our auxiliary loss is effective in representing text features for scene text detection. The balance factor α controls the proportion of text and non-text regions perceived during the self-supervised learning stage. Exp. 9-11 demonstrate that MAEDet achieves the best performance when α = 0.9, suggesting that some background pixels should be included in the comprehensive optimization objective. Therefore, we use α = 0.9 as the default settings.

6 Conclusion

The paper identifies the fine-tuning and long-tailed problems as two significant factors contributing to the discrepancy between scene text detector performance on academic benchmarks and practical applications. Experiments demonstrate that models optimized for a single dataset often overfit specific domains but struggle to generalize to real-world scenarios. Given that Dataset-Specific Optimization (DSO) leads to such fine-tuning gap, we advocate Joint-Dataset Learning (JDL) for practical demands. Regarding the long-tailed problems, we define 13 categories of long-tailed challenges and introduce the Long-Tailed Benchmark (LTB), the first holistic benchmark to evaluate model's ability to detect different types of challenging texts, setting a new standard in scene text detection. Additionally, MAEDet is proposed as the baseline for our benchmark. We hope this work could provide valuable insights into improving scene text detectors and envision more efforts being devoted to practical approaches that effectively address long-tailed problems.

Acknowledgments

Supported by the National Natural Science Foundation of China (Grant NO 62376266 and 62406318), Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China, Beijing, China.