AI-Powered Oracle Bone Inscriptions Recognition and Fragments Rejoining

1 Introduction

Oracle Bone Inscriptions (OBI) is the writing system of the Shang Dynasty and the main carrier of human history at that time. Therefore, OBI research is very important to the investigation of the ancient Shang history. It can also reveal the origin/evolution of the Chinese characters. The main presentation form of OBI is rubbing. Figure 1 shows a sample OBI image from [Guo, 1982]. Since many Oracle Bones (OB) are fragments, the restoration work is often a prerequisite for OBI research. OBI recognition can greatly save human efforts in arranging and processing the OB materials.

The rejoining work of Oracle Bones began in the early 20th century [Zhang, 2019]; since then, many OBI scholars have been making tremendous endeavor in this direction. However, existing rejoining work is mainly based on human intelligence which comprehensively utilizes semantics, context, writing style, marginal sealability (fitness) and material characteristics in the rejoining process. Such manual restoration manner requires OBI scholars to have very strong memory and association ability, and consumes them a huge amount of time and energy. Artificial Intelligence (AI) techniques have a strong potential in Oracle Bone rejoining, yet little effort has been devoted towards this direction.

The ancient OBI characters have large shape, scale and orientation variations. OBI characters are handwritten and their sizes are uneven and the arrangements of the OBI texts are non-uniform and arbitrary. Three types of interfering factors are present in the OBI images: moires (natural), crackles (arising from burning during divination), and noises. These challenges make OBI recognition very hard. Existing annotations (notes) made by OBI researchers are image-level, i.e., they only provide a whole paragraph of text annotations for each rubbing image, instead of the corresponding coordinates and class label for each character in the rubbings. Such image-level annotations cannot be directly fed into off-theshelf deep learning/machine learning approaches.

Cooperating closely with OBI domain experts, we design a red-green-line framework for OB rejoining which yields a top-10 accuracy of 98.39%, and a novel OBI recognition algorithm which achieves an overall accuracy of 80.9%. These algorithms enable the rejoining of OB fragments, as well as the localization and recognition of the OBI characters on the rubbings. Moreover, we develop software tools to facilitate the annotation of the OBI characters over the rubbing images and the selection of rejoinable OB candidates. In this paper, we will introduce these algorithms and software tools.

2 RelatedWork

2.1 Oracle Bone Rejoining

In the past two decades, more than 4,000 groups of OB fragments have been successfully rejoined, well-known achievements are [Guo, 1982], [Tsai, 1999], [Huang, 2019], and [Lin, 2008]. Besides, Dr. Yi Men focuses on rejoining the Huang type OB and she has successful put together 200 groups of Oracle Bones [Men, 2008; Men, 2012]. Dr. Bofeng Mo and other researchers have also achieved good results in this field [Mo, 2012; Mo, 2016]. They are also collaborators/domain experts of this work.

In the literature, there are few methods that address AIaided OB rejoining. The authors in [Zhang and Wang, 2012] proposed to use contour matching in OB rejoining, but only used 4 groups of rejoined bones in their experiments.

2.2 Oracle Bone Inscription Recognition

The research in [Guo et al., 2016] is a representative work in OBI recognition. However, they only used a synthetic OBI dataset under clean background, which is impractical for realworld problems. In [Meng, 2017], the author proposes a traditional template-matching framework for OBI recognition, which might be severely interfered by shell moires.

[Franken and van Gemert, 2013] presents an approach for recognizing ancient Egyptian hieroglyph with low-level feature descriptors. [Hu et al., 2015] proposes a Maya glyph retrieval system which combines shape and context.

In recent years, significant progress has been achieved in robust scene text reading [Long et al., 2018]. CTPN [Tian et al., 2016], EAST [Zhou et al., 2017], and TextBoxes++ [Liao et al., 2018] are representing methods for scene text detection, while CRNN [Shi et al., 2017] and ASTER [Shi et al., 2019] are well-recognized methods for scene text recognition.

3 Main Contributions

We have made the following effort towards AI-powered OBI recognition and fragments rejoining:

(1) We build OB-Rejoin which is a real-world dataset of 1,000 OB fragments. OBI scholars stroked the fragmented borderlines of each OB in red color, they also added a greenline which is the tangent of the left or right original border, to ensure the smoothness of the left or right borders of two OB fragments after being put together.

(2) We design an effective "red-green-line" algorithm for OB rejoining. For two input OBs, we first align their green lines (be on the same line), then horizontally shift and rotate the bottom image at a certain range. We propose a time serialization method to transform the stroked red borderline curves into numerical "time series" data so that time series analysis algorithms can be adopted, in which we devise the tolerance difference (T-Diff ) similarity measure for two time series. In T-Diff, for two time series, only the corresponding segments with a bitwise subtraction value below the tolerance threshold thr will be counted when measuring their pair similarity. It is robust when there are holes along the fragmented borders. On the OB-Rejoin dataset, our red-green-line method with T-Diff yields a top-10 accuracy of 98.39%. Besides T-Diff, existing distance measures such as DTW [Ding et al., 2008] can also be applied, but their performance is inferior to T-Diff.

1. We develop a software tool to help OBI scholars conveniently filter out the unpromising candidates for OB rejoining. It consists of three steps, "immediate rejection", "further selection", and "final selection". For each rubbing, redgreen-line method outputs its top-10 most rejoinable candidates. With "immediate rejection" step, we let OBI scholars to quickly exclude the candidates that are clearly not rejoinable from their professional perspectives. The remaining candidates will be carefully investigated by domain experts.

(4) We design a software tool to facilitate the annotation of the OBI characters on the rubbing images. As mentioned before, existing OBI annotations are image-level, which cannot be directly fed into character recognition algorithms. To tackle this problem, we develop an annotation software to make fine-grained character-level annotations, shown in Figure 2. For each rubbing, we first use deep learning based scene text detection algorithms to predict the bounding box of each single character in the image, which can be observed from Figure 2. Next, we read each sentence in the image-level annotations and let the annotator sequentially click (specify) the corresponding bounding box for each character in the sentence. This way, we can conveniently obtain the exact position of each single character in the rubbing. Finally, the character crops will be used in OBI recognition.

(5) Using the above software, two OBI professors and ten OBI graduate students build OracleBone-8000, which is the first real-world OBI dataset with character-level annotations. It contains 7,824 OBI rubbing images from the book [Guo, 1982] and 128,770 annotated character crops/instances. This dataset is highly imbalanced and sparse, it provides a unique benchmarking platform to OBI recognition research.

(6) We propose a novel deep learning based template matching algorithm for OBI recognition, which automatically learns the similarity/distance between an OBI crop from the rubbing image and the template typeset images of characters from the OBI font library, and significantly outperforms the competitors. The idea behind our algorithm is that, although OBIs are handwritten, such template typeset images from the OBI font library can generally embody the common shape characteristics of the OBIs. In our algorithm, we adopt deep learning based Siamese network [Zagoruyko and Komodakis, 2015] to automatically learn the similarity between an OBI crop and the corresponding template typeset images. It obtains an overall accuracy of 80.9% in OBI recognition.

For OBI character localization in the rubbing images, we adopt the EAST [Zhou et al., 2017] scene text detection algorithm, which achieved state-of-the-art performance in scene text detection. Using EAST, we obtain an overall recall of 88.8% and a precision of 90.7% on OracleBone-8000.

4 Conclusion

In this paper, we introduce our research in AI-powered OBI recognition and fragments rejoining, and the software tools developed for OBI annotation and filtering of rejoinable candidates. These algorithms and tools should be very helpful to domain experts in OBI research.