Structure-Aware Handwritten Text Recognition via Graph-Enhanced Cross-Modal Mutual Learning

1 Introduction

Handwritten text recognition (HTR) is considered a highlevel human-computer interaction, which aims to convert humans' handwriting into characters or texts. HTR still remains challenging since humans' handwriting can be very arbitrary and has distinct writing styles. With the recent advances in deep learning techniques, it has witnessed significant progress in HTR [Yousef et al., 2020; Kang et al., 2022; Li et al., 2023], in which the most widely used approaches are attention-based encoder-decoder models or net-

∗Corresponding author: Xinbo Gao

works equipped with the connectionist temporal classification (CTC) [Graves et al., 2008] auxiliary objectives.

However, most existing methods for HTR simply operate on handwriting data from a single modality, i.e., either visual images or geometric graphs alone. Specifically, visual-based methods simply focus on exploiting visual patterns of handwritten texts by modelling the low-level relationships of adjacent pixels, while ignoring the intrinsic geometric structures of characters. Instead, computational methods based on structured, relational information can better demonstrate humanlike learning. Particularly, recent advances [Gan et al., 2023; Chen et al., 2024] demonstrate that handwritten characters can also be considered as geometric graphs, which can explicitly learn the geometric structures of characters by utilizing graph-based networks [Yao et al., 2019]. Although such graph-based methods are primarily proposed for recognizing isolated handwritten characters, they can be feasibly extended to HTR by integrating the encoder-decoder framework.

Nevertheless, it remains under-explored whether the combination of visual and geometric information can benefit handwriting analysis. Existing uni-modal approaches typically overlook the fact that both visual and geometric information of characters are advantageous for handwriting recognition, in which the two distinct modalities can complement each other. Therefore, it is highly probable to achieve better recognition performance by adequately exploiting and fusing both visual and geometric properties of handwritten texts through multi-model learning. In other words, a structureaware handwriting recognition approach that explicitly incorporates both characters' geometric structures and visual appearances is promising to achieve better performance.

Furthermore, it also suffers from inadequate cross-modal fusion and conflicting optimization directions when directly adopting existing multi-modal fusion techniques for HTR. Recent advances [Huang et al., 2022b; Peng et al., 2022] demonstrate that multi-modal learning may be inferior to the uni-modal models under certain circumstances, since multimodal models sometimes cannot jointly utilize all modalities well due to the conflicting optimization directions and uncoordinated modality convergence. Moreover, existing encoderdecoders typically adopt the two-dimensional coverage attention to focus on the current visual region of characters for literal prediction, while ignoring the fact that geometric structures are more reasonable and intuitive for identifying characters, which therefore may cause the problem of attention drift. Nevertheless, it remains unclear how to appropriately and effectively integrate the geometric modality data to complement the visual modality for better performance.

To address those challenges, we introduce a graphenhanced cross-modal mutual learning network (GCM) to fully process handwriting images alongside their corresponding geometric graphs for HTR. Firstly, we introduce a graphenhanced cross-modal decoder to aggregate the geometric contexts from graphs and then employ such graph contexts to guide the visual coverage attention on visual regions in the G→V direction, thus achieving structure-aware handwriting recognition. Secondly, considering that geometric contexts can guide the decoding process on visual features (from G→V), visual context can also guide the decoding process on geometric features in the inverse direction (fromV→G). Intuitively, the cross-modal features from two inverse directions should be decoded into identical predictions, and thus, it is possible to enhance each other by minimizing their distances in the shared latent space. Therefore, we introduce two parallel decoders that sequentially aggregate cross-modal features in inverse orders (V→G and G→V) but are enhanced through mutual distillation at each time step. This eventually achieves one-to-one knowledge transfer and fully leverages complementary cross-modal information from both directions. Fig. 1 highlights the differences between our GCM and other prevalent approaches, and our contributions are listed as follows:

• We achieve structure-aware handwritten text recognition via graph-enhanced cross-modal mutual learning (GCM). Notably, cross-modal methods that fuse both geometric graphs and visual appearances remain under-explored, and we are also among the first to introduce mutual learning for graph-enhanced cross-modal fusion in HTR.
• We propose a novel bidirectional cross-modal mutual learning strategy that sequentially aggregates cross-modal features in inverse orders (V→G and G→V) and further
• enhances each other through mutual distillation. Notably, only one decoding branch is activated in inference, thus avoiding the increase of model parameters and computation costs for testing and deployment.
• Extensive experiments demonstrate that the proposed GCM surpasses previous state-of-the-art (SOTA) methods for HTR on benchmarks such as IAM, RIMES, and ICDAR-2013 when no extra training data is utilized.

2 RelatedWork

2.1 Handwritten Text Recognition

It has witnessed significant progress in HTR in recent years, which can be briefly divided into CTC-based and attentionbased methods. For CTC-based methods, [Graves et al., 2008; Graves and Schmidhuber, 2008] proposed to utilize recurrent neural networks (RNNs) [Lipton et al., 2015] for segmentation-free HTR. Furthermore, [Coquenet et al., 2020; Ingle et al., 2019] adopted convolutional neural networks (CNNs) to model the sequential handwritten texts in parallel. Recently, attention-based methods have emerged for HTR with promising performance [Yousef et al., 2020]. [Kang et al., 2022; Li et al., 2023] further introduced Transformers for HTR due to their better ability to model global context relationships. Moreover, [Gan et al., 2023] proposed to view handwriting characters as geometric graphs and further employed graph Transformers to recognize those graphs. Nevertheless, most existing methods for HTR simply operate a single modality alone, while leaving the multi-modal methods that fuse visual and geometric properties under-explored.

2.2 Mutual Learning for Handwriting Analysis

Mutual learning [Zhang et al., 2018; Zhu et al., 2018; Guo et al., 2020] refers to a learning process in which an ensemble of networks learn collaboratively and teach each other via knowledge transfer. However, very few methods have attempted to utilize mutual learning for handwriting analysis. Nevertheless, we are among the first to adopt mutual learning into multi-modal methods for HTR with promising results.

3 Methodology

As shown in Fig. 2, we propose a graph-enhanced crossmodal mutual learning network (GCM) to achieve structureaware handwritten text recognition, which explicitly exploits both visual and geometric characteristics of handwritten texts. Given a handwritten text image v, we first convert it into a skeleton-based graph g and then adopt the recognition model to produce the target character sequence y = (y1, · · · , yl, · · · , yL) with the maximal probability, i.e.,

where each character yl ∈ A is in the alphabet set A.

3.1 Graph Construction from Images

Following [Gan et al., 2023], we first extract skeletons from handwriting images and then resample the key points in identical intervals while keeping their original connections as shown in Fig. 3. Eventually, we obtain geometric graphs from visual images without introducing any extra annotation costs.

3.2 Independent Multi-Modal Encoding

To exploit characters' visual and geometric properties, it is important to first encode the multi-modal data into a shared latent representation space for effectively capturing complementary information from each other. Since geometric graphs and visual images of characters have distinct data organization, a more appropriate way is to utilize two independent multi-modal encoders (i.e., the graph encoder and visual encoder) for extracting the cross-modal features separately.

Visual Encoder We adopt the most prevalent convolutional recurrent network (CRN) [Shi et al., 2016] to directly extract the visual information from handwriting images, which is a hybrid network that fully integrates the advantages of both CNNs and RNNs. Given a handwriting image v, a visual encoder V is utilized to extract visual features as ˜v = V(v).

Graph Encoder To fully exploit the geometric structures of handwritten texts, we utilize the pyramid graph Transformer (PyGT) [Gan et al., 2023] to process character graphs, which is stacked by multiple graph-enhanced attention blocks. Given a character graph g, the graph encoder G is adopted to exploit the local topology structures as ˜ g = G(g).

3.3 Graph-Enhanced Cross-Modal Decoding

Considering that geometric and visual features are essentially complementary, the graph context can guide the decoding process on the visual regions of characters (from G→V), and similarly, visual context can also guide the decoding process on geometric features in the inverse direction (from V→G). To fully integrate the cross-modal features of characters, we introduce novel graph-enhanced cross-modal decoders in two inverse directions through hybrid multi-modal fusion and cascaded attention aggregation.

Graph → Visual Graph-Enhanced Decoder

G→V decoding branch aggregates geometric context from character graphs and further employs it to guide the coverage attention over visual images.

Graph Guidance The geometric context is captured as the graph guidance for the later decoding on visual images. Specifically, a graph recurrent cell RG→V g first calculates the decoding context cGl of l-th character based on its previous graph hidden state hGl-1 and the previous character yl-1 as

The graph hidden state hGl is then updated based on the geometric feature ˜ g and decode context cGl via attention-based aggregation as

where ⊤ is the transpose operation. The obtained graph hidden state hGl can be considered as the graph guidance.

Graph-Guided Cross-Modal Decoding The graph guidance hGl is integrated to guide the decoding process on visual regions of characters based on the hybrid multi-modal fusion and cascaded attention aggregation. Specifically, the visual recurrent cell RG→V v computes the current decoding state cG→V l depending on its previous visual hidden state hG→V l-1 as well as the previous character yl-1, and also fuses the graph guidance hGl at the early stage, i.e.,

where αg is a learnable fusion parameter. Then, the visual hidden state hG→V l is updated through attention-based aggregation as

where ˜v denotes visual features. Finally, we fuse the crossmodal contexts at the late stage as

where OG→V l is the discriminative pattern in the G→V branch, and F is a fully connected layer. Finally, the probability of producing the l-th character in the G→V branch is calculated as

where y<l denotes the previous l - 1 characters.

Visual → Graph Visual-Enhanced Decoder

Similarly, we can also aggregate the visual context to guide the coverage attention over geometric structures of characters with the V→G decoding branch.

Visual Guidance Similarly, the visual context is computed through a visual recurrent cell RV →G based on the visual features ˜v, previous visual hidden state hVl-1 , and the previous character yl-1 as

where the visual context hVl is treated as the visual guidance. Visual-Guided Cross-Modal Decoding We similarly compute the discriminative pattern in the V→G branch as

Lastly, the probability of producing the l-th character in the V→G branch is calculated as

Eventually, we obtain the probabilities of the l-th character yl in inverse directions at each time step.

3.4 Bidirectional Cross-Modal Mutual Learning

Since the cross-modal representations are complementary to each other, the cross-modal discriminative patterns from two inverse directions (i.e. OV→G l and OG→V l ) should produce the identical character yl at each decoding step. Therefore, the cross-modal patterns in inverse fusion directions can enhance each other via one-to-one knowledge transfer, which is achieved by minimizing their distances in the shared latent space with better multi-modal alignment. Specifically, we introduce Kullback-Leibler (KL) divergency [Zhang et al., 2018] to quantify the difference in prediction distributions between OV→G l and OG→V l . The objective is to minimize the distance between the probability distributions of two branches at each decoding step through mutual distillation as

where ϵ(O∗ l , T) = Softmax(O∗ l /T) is the soft probabilities with the temperature T. This eventually achieves one-to-one knowledge transfer between inverse branches and fully leverages the complementary cross-modal information from both directions.

3.5 End-to-End Optimization Objectives

Given the target sequence ˜y = ( ˜ y1, · · · , ˜ yl, · · · , ˜ yL), the model first minimizes the cross-entropy losses over multimodal features in two inverse branches as

where OV →G l,yl and OG→V l,yl are the logits of yl-th class. Moreover, we further introduce the uni-modal learning objectives besides multi-modal joint learning to alleviate the uncoordinated modality convergence issue [Huang et al., 2022b] as

where LCE is the cross-entropy loss, and O∗|G→V l, ˜ yl denotes the uni-modal logits of the G→V branch, i.e., either the visual modal logit OV |G→V l, ˜ yl or the graph modal logit OG|G→V l, ˜ yl as

Similarly, the uni-modal learning objective is calculated as

with the uni-modal logits of the V→G branch as

Finally, the overall training objective is summarized as

where λ is the hyperparameter that controls the significance of the cross-modal mutual learning.

4 Experiments

4.1 Experiment Settings

Datasets All used datasets are publicly available and have the official dataset splits or proportions, which includes:

• IAM [Marti and Bunke, 2002] is the most widely used public handwritten English text dataset, which contains 1539 handwritten pages comprising 115,320 words.
• RIMES [Grosicki et al., 2009] is a public French handwriting dataset, which is contributed by over 1300 people with 12,723 pages corresponding to 5605 mails.
• IAHEW-UCAS2016 [Gan and Wang, 2019] is a public in-air handwriting English word dataset, which contains 150,480 samples covering 2280 English words.
• ICDAR2013 [Yin et al., 2013] is the most widely used Chinese handwriting dataset, which contains 3755 classes of Chinese characters with 224,419 handwriting samples.

EvaluationMetrics We use the most widely used Character Error Rate (CER) and Word Error Rate (WER) to evaluate the performance of handwriting recognition models.

Implementation Details The whole architecture is implemented with the PyTorch [Paszke et al., 2017] deep learning framework. The model is optimized via the Adam [Kingma and Ba, 2015] algorithm with a batch size of 64. We set the initial learning rate to 0.001 and the hyperparameter of mutual learning λ to 0.75 by default, and the training process is terminated when the model reaches convergence. We set the beam width to 64 during the decoding stage. All experiments are conducted on a workstation with an Intel(R) Core(TM) i911900K CPU, 64GB RAM, and an RTX-4090 24GB GPU.

4.2 Comparison with PriorWorks

To demonstrate the effectiveness of our method, we compare the proposed GCM with previous SOTA methods for HTR on public benchmarks. It is worth noting that our GCM can recognize both online and offline handwritten texts.

Results for Offline Handwritten Text Recognition As listed in Table 1, we compare our GCM with previous SOTA methods for HTR on two public offline handwritten text datasets IAM and RIMES. For a fair comparison, we only compare the methods without utilizing the extra synthetic or training data in Table 1. It can be observed that most previous methods are only based on the image modality, while graphenhanced multi-modal methods are rarely explored. Moreover, our method also outperforms previous SOTA methods for offline HTR when no extra training data is utilized. Experiments demonstrate that it can achieve better performance for HTR by exploiting both visual and geometric information of characters, especially for scenarios with limited data.

Results for Online Handwritten Text Recognition It is worth noting that the proposed GCM can be feasibly extended to online HTR. As listed in Table 2, we extended GCM to online HTR on IAHEW-UCAS2016, in which each sample is written in mid-air with a single stroke. Additionally, we mainly compared GCM with prior prevalent approaches, including RNN-CTC, RNN-ATTN, etc. It can be observed that our multi-modal GCM outperforms the uni-modal models (such as RNN-Decoder & GCN-Decoder) and has achieved the new SOTA performance on IAHEW-UCAS2016, demonstrating its effectiveness for online HTR.

Results for Handwritten Chinese Character Recognition We demonstrate that GCM is also effective for radical-based handwritten Chinese character recognition (HCCR), in which handwritten Chinese characters are recognized as radical sequences. As shown in Table 3, we compare the GCM with previous SOTA methods for HCCR on ICDAR-2013. It is worth noting that radical-based methods generally perform inferior to conventional class-based methods, since it is more challenging to simultaneously recognize both individual structures and radicals of Chinese characters. Experimental results demonstrate that our GCM has achieved the SOTA performance among radical-based approaches for HCCR.

4.3 Ablation Study of ProposedMethod

To investigate the contributions of individual parts of GCM, we have conducted extensive ablation studies of different modules or learning strategies. For a fair comparison, we applied greedy decoding for testing in all ablation studies.

Effectiveness of Cross-Modal Fusion As shown in Table 4, we compare the performance of different network configurations by integrating different modalities of data. The experimental results reveal several key insights: (i) directly processing images achieves high accuracy, as spatial information plays a crucial role in handwriting recognition; similarly, decoding geometric graphs also yields comparable accuracy, suggesting that geometric structures can also benefit handwriting recognition; (ii) the cross-modal encoderdecoder frameworks can achieve better performance than unimodal ones, since the former explicitly leverages both the geometric structures and spatial information of handwritten characters; (iii) it can achieve better performance by enhancing the cross-modal fusion and multi-modal learning.

Effectiveness of Bidirectional Mutual Learning As shown in Table 5, we compare the performance of crossmodal models with different fusion directions. It can be observed that: (i) the single-direction fusion (either G→V or V→G) can achieve satisfactory performance, since both geometric and visual features can benefit handwriting recognition; (ii) moreover, fusing cross-modal data with two identical directions can also improve the recognition performance via one-to-one knowledge transfer, but its improvement may become marginal since the branches with identical directions attain mirror differences; (iii) instead, the branches with the inverse directions can learn more distinguishing patterns and thus can be more complementary via mutual learning.

Necessity of Uni-Modal Learning Objectives As shown in Table 6, we compare the performance of cross-modal decoders with or without utilizing LU . It is worth noting that existing multi-modal fusion generally suffers from imbalanced multi-modal learning problems due to the conflicts of multimodal optimization directions. However, such a problem can be effectively alleviated by adding uni-modal learning objectives besides multi-modal joint learning. As shown in Table 6, the models jointly optimized with both uni-modal and multimodal objectives can achieve better performance.

Hyperparameter Sensitivity Analysis of LKL We also perform the sensitivity analysis on the hyperparameters λ and T in LKL, which control the impact of the mutual learning loss between two inverse branches. As shown in Fig. 4, LKL with λ = 0.75 and T = 2 achieved the best performance.

4.4 Qualitative Analysis

Visualization of Cross-Modal Attention To demonstrate the effectiveness of cross-modal fusion, we compare the attention matrices and visualization results between crossmodal and uni-modal methods as shown in Fig. 5 & 6. It can be observed that uni-modal methods suffer from more served attention drift problems than cross-modal ones. Moreover, the cross-modal guidance can help the attention module more precisely locate the corresponding character regions, thus demonstrating the effectiveness of cross-modal fusion.

Visualization of Feature Distribution To demonstrate that GCM can learn more discriminative features, we further visualize feature distributions of lowercase characters between the cross-modal and uni-modal methods as shown in Fig. 7.

We can observe that cross-modal methods can learn more discriminative feature distributions.

5 Conclusion

Most existing approaches for HTR typically operate on the visual modality alone, while ignoring the importance of the intrinsic geometric structures of characters. Instead, this paper aims to achieve a structure-aware handwriting recognition method that explicitly incorporates both geometric structures and visual appearances of characters. Particularly, we propose a novel graph-enhanced cross-modal mutual learning network GCM for handwritten text recognition, which can fully process handwriting images alongside their corresponding geometric graphs. Specifically, GCM first extracts both visual and geometric information from the cross-modal inputs, and then it utilizes two parallel decoders to sequentially aggregate cross-modal features in inverse orders (V→G and G→V). Furthermore, the two inverse branches can further enhance each other via bidirectional mutual learning, which involves one-to-one knowledge transfer and fully leverages complementary cross-modal information. Notably, only one decoding branch of GCMwill be activated in inference, thus avoiding the increase of the model parameters and computation costs for testing and deployment. Extensive experiments demonstrate the effectiveness of the proposed GCM for HTR.

6 Limitation

The proposed GCM is unsuitable for recognizing complex scene texts, since it is challenging to obtain geometric graphs from scene text images when compared to handwriting images. Nevertheless, we leave this problem as our future work.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (NSFC) under Grant No. 62206035, 62472060, and 62276258, the Science and Technology Innovation Key R&D Program of Chongqing under Grant No. CSTB2024TIAD-STX0023 and CSTB2023TIADSTX0016, the Surface Project of Natural Science Foundation of Chongqing under Grant No. CSTB2022NSCQMSX0547, the Science and Technology Research Program of Chongqing Municipal Education Commission under Grant No. KJQN202200612 and KJQN202400648, China Postdoctoral Science Foundation under Grant No. GZC20233362 and 2024MD754043, and the Open Projects Program of State Key Laboratory of Multimodal Artificial Intelligence Systems under Grant No. MAZS-2023-11.