WenyanGPT: A Large Language Model for Classical Chinese Tasks

1 Introduction

Classical Chinese is an important component of Chinese culture, with a long history and profound cultural heritage. It is essential for understanding traditional Chinese culture. With the rapid development of artificial intelligence technology, intelligent processing of ancient Chinese texts offers a new solution for the preservation and inheritance of Classical Chinese. Modern technologies, such as digitalization and natural language processing (NLP), can efficiently preserve and spread traditional culture while fostering deeper and more innovative academic research. Maximizing the potential of AI in processing Classical Chinese texts has become an urgent need for cultural inheritance and academic development.

Early research in Classical Chinese language processing focused on tasks such as punctuation, word segmentation, part-of-speech tagging (POS) , named entity recognition (NER) and translation. These tasks initially relied on traditional machine learning methods, such as Hidden Markov Models (HMMs) for POS [Huang et al., 2002], Conditional Random Fields (CRFs) for punctuation [Huang et al., 2010] and NER [Yuan et al., 2019; Li, 2018]. In deep learning, RNNs, LSTMs, GRUs, and Attention mechanisms have been applied to various tasks, including couplet generation and classical poetry generation [Yan et al., 2016; Yi et al., 2017], punctuation and POS using BiLSTM-CRF models [Wang et al., 2019; Cheng et al., 2020; Zhang et al., 2023b; Chang et al., 2024]. As Transformer architectures emerged [Vaswani et al., 2017], studies began using large-scale parallel corpora to train models for translating Classical Chinese into Modern Chinese [Liu et al., 2018] and for generating classical poetry [Huang et al., 2020]. The introduction of pre-trained models including BERT [Kenton and Toutanova, 2019] and GPT [Radford and Narasimhan, 2018] provided new opportunities for intelligent Classical Chinese processing. Some research integrated ancient Chinese texts into the training data of general pre-trained models, improving the processing performance of Classical Chinese compared to typical pre-trained models [Tian et al., 2020; Wang et al., 2022; Wang et al., 2023a; Liu et al., 2023a]. Other studies used Classical Chinese corpus to continue pre-training and fine-tuning large language models, aiming to build conversational models for Classical Chinese [Zhang et al., 2024; Yang et al., 2024b; Cao et al., 2023; Cao et al., 2024].

∗Corresponding author.

However, challenges remain in Classical Chinese processing. Different tasks require training specialized models, and no effective universal model exists. Additionally, there is a lack of standardized evaluation benchmarks in this field, with existing evaluation tasks, datasets, and metrics being inconsistent, making it difficult to perform cross-task comparisons and systematic assessments of model performance.

To address these issues, we propose WenyanGPT, a large language model for Classical Chinese. Some examples of WenyanGPT are shown in Figure 1. We also construct the largest pre-training corpus for continued pre-training, enhancing the model's domain adaptability. Additionally, we propose a framework for generating domain-specific instruction data for supervised fine-tuning in the development of WenyanGPT. To promote research in the intelligent processing of Classical Chinese, we build the WenyanBENCH evaluation dataset and conduct extensive experiments for detailed analysis. The main contributions are as follows:

• We propose WenyanGPT, a large language model focused on Classical Chinese. It demonstrates superior performance and wide applicability in tasks such as punctuation, POS, translation, etc.
• We release pre-training and instruction fine-tuning datasets, along with a novel method for constructing domain-specific fine-tuning data, providing valuable resources for future research.
• We introduce WenyanBENCH, an evaluation benchmark for Classical Chinese tasks, with extensive experiments verifying WenyanGPT's leading performance across multiple tasks.

2 RelatedWork

2.1 PLMs

In 2017, Google introduced a new neural network architecture, Transformer. It utilizes self-attention mechanisms to better handle long-distance dependencies and significantly improves training efficiency through parallel computation. Based on Transformer, various LLMs have been proposed. BERT employs an encoder-only Transformer architecture and is pre-trained using masked language modeling and next sentence prediction tasks. The GPT series, uses a decoderonly Transformer architecture and an autoregressive language model (ALM). Over the course of the GPT series, the model size has steadily increased, from the original GPT to subsequent iterations, including GPT-2 [Radford et al., 2019], GPT-3 [Brown et al., 2020], and GPT-4 [Achiam et al., 2023], with continuous improvements in performance. PaLM [Chowdhery et al., 2023] uses the standard Transformer architecture in a decoder-only model with a modified SwiGLU activation function. This model, with 540 billion parameters, achieved human-level performance in 1-shot learning on the BIG-bench [Srivastava et al., 2023] dataset. In 2023, Meta AI released the LLaMA model [Touvron et al., 2023]. This model also follows a decoder-only Transformer architecture and excels in various NLP tasks after large-scale training. In 2024, LLaMA 3 [Dubey et al., 2024] was released, including a pre-trained version with 405 billion parameters and a posttraining version, alongside the LLaMA Guard 3 model for input-output safety. Pre-trained language models have seen rapid development, with Transformer-based models becoming the mainstream technology in NLP.

2.2 Classical Chinese PLMs

Pre-trained language models have achieved widespread success in the field of NLP. However, studies show that generaldomain models often lack specialized knowledge for tasks in specific domains. Models pre-trained with domain-specific data tend to perform better for specialized tasks [Ke et al., 2023; Gupta et al., 2023; Ibrahim et al., 2024; Taylor et al., 2022; Lehman et al., 2023; Liu et al., 2020]. In the field of Classical Chinese, several studies have extended models such as BERT, RoBERTa, and GPT by incorporating Classical Chinese corpora for pre-training, resulting in specialized models like AnchiBERT [Tian et al., 2020], SikuBERT and SikuRoBERTa [Wang et al., 2022], GujiBERT and GujiGPT series [Wang et al., 2023a], and SikuGPT [Liu et al., 2023a]. These models show improved performance over general pretrained models in Classical Chinese tasks. Instruction finetuning is another effective strategy. Using Supervised FineTuning (SFT) can activate LLMs' ability to understand and answer questions in a specific domain [Liu et al., 2023b; Xiong et al., 2023; Wang et al., 2023b; Yue et al., 2023; Huang et al., 2023; Cui et al., 2023; Yang et al., 2023b; Zhang et al., 2023a; Dan et al., 2023]. Classical Chinese LLMs [Zhang et al., 2024; Yang et al., 2024b; Cao et al., 2023; Cao et al., 2024] are in the early stages of development. For example, the "Xunzi 1 " Classical Chinese large language model was trained on Classical Chinese-related corpora based on general models such as Qwen2.5 [Yang et al., 2024a], Baichuan2 [Yang et al., 2023a], and GLM-4 [Zeng et al., 2024]. It has shown excellent performance in intelligent tagging, information extraction, and other tasks. TongGu [Cao et al., 2024], through two-stage instruction fine-tuning, is capable of Classical Chinese punctuation, translation, and appreciation tasks. In this paper, WenyanGPT is fine-tuned on higher-quality pre-training data and a larger, more diverse instruction dataset, showing superior and more comprehensive task handling capabilities.

3 WenyanGPT

In order to obtain the WenyanGPT Classical Chinese model, we first construct a Classical Chinese pre-training corpus and continue pre-training based on LLaMA3-8B-Chinese (Section 3.1). Then, we propose a method to construct domain instruction data (Section 3.2). In our framework, instruction generation is manually constructed, guided by LLMs, and tested to ensure the high quality of the fine-tuning data. The complete training process is shown in Figure 2.

1 https://github.com/Xunzi-LLM-of-Chinese-classics/ XunziALLM.

3.1 Pre-training

The corpus used in the pre-training phase is sourced from authoritative websites such as Daizhige, Wenyanguji, and various Classical Chinese-related data collected and organized from GitHub. The detailed data sources and scale are shown in Table 1. We uniformly format and store data from these different sources, removing redundant information, errors, special symbols, and invalid characters. As a result, we obtain a clean, large-scale, high-quality Classical Chinese corpus of approximately 16GB. This corpus covers the Four Books and Five Classics, including Confucian classics, historical records, works of various philosophers, poetry, essays, drama, novels, miscellanies, and other literary genres. It also encompasses diverse fields such as local gazetteers, genealogies, religious texts, agricultural, law, medicine, astronomy, geography, craft books, and military texts. The corpus integrates both simplified and traditional Chinese characters, spanning from the Pre-Qin period to the Republic of China, providing rich materials for deep learning and research on ancient Chinese texts. We select LLaMA3-8B-Chinese as the base model and use the bfloat16 data format during training to improve efficiency. The hyper-parameter settings in pretraining are shown in Table 2.

3.2 Supervised Fine-Tuning

Based on continued pre-training, we perform supervised finetuning to better adapt the model to specific tasks and instructions. We use a high-quality set of instruction fine-tuning data, which we have previously collected and organized, to trigger the knowledge the model acquires during pre-training. The detailed process for constructing the instruction finetuning data is shown in Figure 3.

Data Selection and Initial Organization. We select relevant data from the Classical Chinese corpus, including three major categories: question-answer pairs, parallel corpora, and tagged corpora. Parallel corpora are used for translation tasks, while tagged corpora support fine-grained tasks such as punctuation and POS. In cases where the corpus lacks clear question-answer pairs, we generate supplementary data. During the selection phase, we prioritize high-quality data with clear content, standardized semantics, and task relevance to build the initial high-quality input-output pairs.

Manual Design of Task Instructions and Model Expansion. We design initial task instruction templates manually based on high-quality input and output, covering tasks such as Classical Chinese punctuation, and translating Classical Chinese into Modern Chinese. Then, we use LLMs, such as GPT and Qwen series to expand the task instructions. On the one hand, we generate diverse instructions from existing ones; on the other hand, we allow LLMs to perform reverse reasoning from the existing high-quality input-output pairs to generate new instructions, ensuring diversity in the instructions. After the expansion, we conduct an initial screening of the generated instructions, removing those with unclear or unreasonable semantics, resulting in a seed instruction set.

Testing the Instruction Set and Optimizing Fine-Tuning Data. We randomly select high-quality input-output pairs and combine them with the seed instruction set to evaluate the model's adherence to instructions across different task scenarios. We analyze the test results to refine the instructional design and identify the task instructions that produce higher-quality outputs. Then, we combine the optimized instruction set with high-quality input-output pairs to construct fine-tuning data for specific instructions, providing reliable data support for subsequent model training.

Generation and Supplementation of Specific Task Data. Due to the lack of specific task instruction data in the corpus, we manually design diverse initial instructions and inputs.

Then we use LLMs including Qwen2.5-14B and Qwen2.572B to generate high-quality answers. We remove lowquality or irrelevant content through manual selection and automatic quality checks, forming another portion of the instruction data.

Data Integration and Output Validation. We integrate the data generated from the corpus and the supplementary data from the LLMs to form a complete instruction dataset. During integration, we ensure consistency between instructions and outputs, covering multiple Classical Chinese task scenarios. Through comprehensive data validation and optimization, we create a high-quality instruction dataset. In the end, we obtain about 1.85 million instruction fine-tuning data. The detailed data sources and counts are shown in Table 3. We use these data to fine-tune the pre-trained model. The hyper-parameter settings in fine-tuning are shown in Table 4.

4 Benchmarking Classical Chinese Tasks

WenyanBench. In order to evaluate the model's performance on Classical Chinese tasks, we devise a benchmark known as WenyanBench. WenyanBench shares the same data sources as the instruction fine-tuning data and has undergone duplicate data removal, as well as validation by both manual and LLMs. For quality control, we sample a subset of the data. The distribution and detailed statistics of WenyanBench are shown in Table 5.

Tasks. Our benchmark includes six tasks related to Classical Chinese. Among them, we subdivide 14 types of punctuation marks in the punctuation task, divide Classical Chinese word classes into 17 categories in the POS task, such as nr for personal nouns, ns for place nouns, and define 4 categories for the NER task.

Metrics. For the WenyanBench benchmark, different evaluation metrics are used for different types of tasks (understanding tasks and generation tasks). For understanding tasks, evaluation primarily relies on Precision, Recall, and F1-Score. For generation tasks, BLEU and BERT-Score are used as evaluation metrics. BLEU measures the N-gram overlap between generated content and reference answers, while BERT-Score better captures the semantic similarity between the generated content and reference answers.

Evaluation Method. To efficiently assess model performance, we design a set of scripted tools to automatically compute BLEU, BERT-Score, and other metrics. These tools quickly and accurately quantify model outputs, providing clear feedback for model optimization. This automated evaluation approach improves evaluation efficiency and ensures the consistency and comparability of the results.

5 Experiments

The experiments evaluate WenyanGPT's performance on understanding and generation tasks in Classical Chinese.

5.1 Experimental Setup

Baselines. The baselines include general-domain and Classical Chinese domain LLMs. The general-domainLLMs are Qwen2.5-7B-Instruct, Baichuan2-7B-Chat, GLM-4-9BChat, Meta-Llama-3-8B-Instruct, Llama3-8B-Chinese-Chat, GPT-4o [Hurst et al., 2024], and Deepseek-V3 [Liu et al., 2024]. The Classical Chinese domain LLM is XunziQwen1.5-7B-Chat.

Data and Evaluation. We use the WenyanBench benchmark for testing. The understanding tasks, including punctuation, POS, and NER, are evaluated by Precision, Recall, and F1-Score. The generation tasks include word explanation, translation, and reverse dictionary, where BLEU is used for word explanation and translation, and BERT-Score is used for reverse dictionary.

5.2 Experimental Analysis

WenyanGPT demonstrates a significant lead in language understanding tasks. The experimental results for understanding tasks are presented in Table 6. In the NER task, WenyanGPT's precision, recall, and F1 score all exceed 90%, while the second-best model, GPT-4o, fails to surpass 77% in any of these metrics. In the punctuation task, WenyanGPT's F1 score is 16.65% higher than that of the second-best model, Deepseek-V3, reaching 75.66%. Additionally, in the POS task, WenyanGPT's F1 score is 7.35% higher than that of the second-best model, GPT-4o. These results highlight WenyanGPT's overwhelming superiority in understanding tasks, particularly in NER and punctuation. This performance is attributed to the model's extensive pretraining on Classical Chinese data, enabling it to handle complex linguistic phenomena and ensuring higher accuracy and stability in fundamental language understanding tasks, such as POS and NER.

WenyanGPT demonstrates excellent capabilities and high efficiency in subcategories of Classical Chinese compre- Table 7: Results for generation tasks (Translation, Word explanation, Reverse dictionary) on WenyanBench. The results underlined represent the second-best model's BLEU1 score and BERT-Score-F1 score.

hension tasks. As shown in Figure 4, the F1 scores of our model in the three task's subcategories are generally stable and high. Specifically, in the NER task, WenyanGPT's F1 score remains above 80%, demonstrating its strong ability to correctly identify entities such as historical figures, place, and proper nouns in ancient Chinese texts. This performance shows that it has strong accuracy and robustness when processing ancient Chinese texts, and can effectively capture complex contextual relationships and word meaning changes. Overall, WenyanGPT's high F1 score in ancient Chinese comprehension tasks not only reflects its high efficiency in basic tasks, but also demonstrates its advantages and potential in processing fine-grained tasks in ancient Chinese.

WenyanGPT can significantly enhance the quality of generated content in generation tasks. The experimental results for generation tasks are displayed in Table 7. WenyanGPT exhibits strong semantic preservation and contextual consistency, demonstrates superior and more stable performance in translation and word explanation tasks, with

consistently high BLEU scores (BLEU1-BLEU4). In classical Chinese translation, the model's BLEU1 score of 0.47 exceeds that of the second-best model by 0.06. Furthermore, WenyanGPT's F1 score in the reverse dictionary task is 3.47% higher than the second best model. Through meticulous pre-training and multitask training, WenyanGPT develops robust contextual consistency, enabling it to generate content that accurately reflects the intended meaning and context. This capability ensures high-quality semantic transmission and content coherence in both long-text generation and complex tasks, such as word explanation.

WenyanGPT can achieve better performance than the previous open source LLMs in Classical Chinese tasks. WenyanGPT, through large-scale pre-training and multitask joint optimization, demonstrates substantial capabilities in Classical Chinese processing tasks, establishing a clear lead over existing mainstream LLMs. This validates the effectiveness of the proposed approach. The performance of various LLMs on WenyanBench is shown in Figure 5. It is evident that WenyanGPT has achieved the highest score in six Classical Chinese tasks. WenyanGPT's multitask training strategy, which integrates various Classical Chinese processing tasks, enhances the model's ability to learn across tasks. The mutual reinforcement of shared linguistic features and semantic information between tasks notably boosts the model's generalization capabilities. This inter-task synergy not only improves performance on individual tasks but also enables WenyanGPT to effectively handle multiple tasks simultaneously, maintaining stable performance, especially in complex tasks, and demonstrating superior adaptability across tasks.

5.3 Case Study

We provide response examples for understanding tasks (POS) and generation tasks (word explanation) from five LLMs: WenyanGPT, Deepseek-V3, GPT-4o, Qwen2.5-7B-Instruct, and Xunzi-Qwen1.5-7B-Chat, as shown in Table 8 and Table 9. By analyzing these examples, we can see the performance differences of different LLMs in ccessing tasks, especially in the understanding and generation of classical Chinese.

Table 8 illustrates some typical errors in the POS task. Specifically, GPT-4o makes mistakes in tagging time words and proper nouns. For instance, it incorrectly tags the time Table 8: Response examples for the POS task from different LLMs. Errors in POS are marked in red, while text errors are highlighted in blue.

expression "四年" (the fourth year) as a common noun, and it also struggles with recognizing the proper noun "衞州 吁" (Wei Zhou Xu). Qwen2.5-7B-Instruct errors in tagging the part of speech for "春" (spring) and sometimes replaces characters in the original text with inappropriate alternatives. Xunzi-Qwen1.5-7B-Chat primarily fails in distinguishing between time words and nouns, mistakenly using the simplified character "杀" instead of the correct "弑". These errors highlight the model's difficulty in making precise distinctions between similar words.

For the word explanation task, WenyanGPT's response is the closest to the Ground Truth. Although WenyanGPT adds some modifiers, it maintains simplicity and refinement in describing the color, which matches the tone of the poem. Deepseek-V3, GPT-4o, Qwen2.5-7B-Instruct, and XunziQwen1.5-7B-Chat do not specify the subject as "海棠花" (begonia flowers) and instead over-explained the moistness of the petals and the process of spreading color, deviating from the standard, concise description of "晕红" (blushing red). Specifically, the responses from Qwen2.5-7B-Instruct and Xunzi-Qwen1.5-7B-Chat extend the impact of the external environment, which do not fully align with the original text's expression. This over-expansion shows that these LLMs fail to capture the intended simplicity and depth of the original description.

WenyanGPT demonstrates strong semantic understanding and retention capabilities in handling Classical Chinese tasks. Its ability to accurately tag parts of speech and generate concise, poetic word explanations demonstrates a deep grasp of the nuances of Classical Chinese. In contrast, other LLMs struggle with both understanding the subtle distinctions of Classical Chinese and producing responses that remain faithful to the content and meaning of the original text. This positions WenyanGPT as a powerful tool for handling complex tasks in the domain of Classical Chinese, far surpassing the other LLMs in terms of both accuracy and literariness.

Table 9: Response examples for the word explanation task from different LLMs.

6 Conclusion

We propose a comprehensive solution for Classical Chinese language processing challenges, including the development of WenyanGPT, a large language model focused on the Classical Chinese domain, and WenyanBENCH, an evaluation benchmark dataset for Classical Chinese tasks. We release pre-training and instruction fine-tuning datasets and describe the method for constructing the instruction fine-tuning dataset. Through systematic experiments and analysis, we demonstrate the significant impact of domain-specific pretraining and multi-task instruction fine-tuning on improving Classical Chinese processing capabilities. Our model outpermforms existing mainstream LLMs in various downstream tasks. In future, we attend to explore the potential of multimodal models by combining Classical Chinese texts with image data (such as inscriptions and manuscripts) to enhance processing capabilities.

Acknowledgements

We would like to sincerely thank all reviewers for their valuable feedback and constructive comments. This work is supported by the National Social Science Fund of China (Grant No. 22&ZD035).