A Multi-Modal Neural Geometric Solver with Textual Clauses Parsed from Diagram

1 Introduction

Automatic geometry problem solving (GPS) is a longstanding and challenging AI task, and has attracted much attention in the CV and NLP community recently [Seo et al., 2015; Sachan et al., 2017; Lu et al., 2021; Chen et al., 2021; Zhang et al., 2022]. A geometry problem is formed with a textual problem and a geometry diagram, where the textual

the reasoning objective in natural language, and the geometry diagram carries rich structural and semantic information beyond the textual problem to aid problem solving. GPS requires mathematical and multi-modal reasoning capabilities combining the textual problem and geometry diagram. The multi-modal integration and geometric knowledge application are the keys of GPS.

Existing works of GPS can be classified into two categories: symbolic geometric solvers [Seo et al., 2015; Sachan et al., 2017; Lu et al., 2021] and neural geometric solvers [Chen et al., 2021; Cao and Xiao, 2022; Chen et al., 2022]. The symbolic solvers parse the diagram and textual problem into a unified formal language. Based on the geometric theorem knowledge, they usually perform symbolic reasoning by path search and condition matching to produce new conditional states until they find the search target. While the symbolic solvers have better interpretability compared with neural solvers, they are carefully designed with complex rules and hard to extend. Besides, some symbolic solvers may solve problems slowly with many redundant steps, and the search process also does not match humans' solutions. The neural solvers proposed recently embed the diagram and textual problem jointly with the hybrid encoder and selfsupervised auxiliary tasks and generate the solution program in sequential form. Although neural solvers have achieved impressive results with simple pipelines, recent works [Lu et al., 2021; Lu et al., 2022] show that there remains a large performance gap compared with symbolic solvers. One of major reasons is that neural solvers, adopting similar frameworks of general multi-modal reasoning tasks applied for natural images, cannot exploit the diagram efficiently. Because primitives in geometry diagram are slender and overlapped, and have complex spatial relationship, the feature map based [Anderson et al., 2018], region proposal based [Yu et al., 2019] or image patch based [Kim et al., 2021] frameworks cannot extract fine-grained features and even damage structural and semantic information.

Considering the under-representation of diagram and difficulty of cross-modal fusion in neural solver, we represent the geometry diagram by textual clauses including structural clauses and semantic clauses, as demonstrated in bottom of Figure 1. Compared with visual image, clauses have highly syntactic structures with less redundant information naturally. Textual clauses are better suited to describe fine-grained and multi-level information in geometry diagram. Different from the formal language consisting of complex multi-order logic forms in symbolic solver, our clauses only describe basic relationships, where structural clauses depict the connection relations among geometric primitives and the semantic clauses describe the semantic relations between non-geometric primitives and geometric primitives. We do not pursue higher level relations constructed by geometric rules since they do not conform to the goal of neural solver. The model is hoped to have the ability of matching geometric patterns and constructing high-level relations from basic relations.

To promote the research on GPS, we also build a new large-scale and fine-annotated GPS dataset named PGPS9K. PGPS9K contains 9,022 geometry problems paired with geometry diagrams. In contrast to existing datasets, PGPS9K is labeled with both diagram annotation and solution program. The diagram annotation employs the same primitive level annotation method as [Zhang et al., 2022; Hao et al., 2022], which could be translated to textual clauses simply and uniquely. Given the solution complexity of GPS, we design a new annotation form based on geometric theorems for solution program. The solution program provides the problem solving procedure wherein each step is an application of a theorem (axiom), as shown in top of Figure 1, rather than the specific solution steps involving fundamental arithmetic operations [Wang et al., 2017; Amini et al., 2019]. Our solution program carries rich geometric knowledge and has better interpretability, and as well reduces the burden of model learning.

Taking advantage of multi-modal information, we thus propose a new diagram-text fusion neural solver named PGPSNet as shown in Figure 1. In addition to the geometry diagram and textual problem, PGPSNet combines structural clauses and semantic clauses and generates the solution program to solve problem. To fuse different parts of text modality, we propose a structural and semantic pre-training strategy based on Masked Language Modeling (MLM) task [Devlin et al., 2019], for improving the model's understanding of structural and semantic content by explicit modeling. To overcome the limitation of data size and pre-trained corpus, we design five data augmentation strategies based on diversity and equivalence of geometric representation. Besides, we construct a self-limited GRU decoder to shrink the representation space and search space of operands and speed up training and inference. Experiments on an existing dataset Geometry3K [Lu et al., 2021] and PGPS9K dataset demonstrate that our PGPSNet boosts the performance of GPS prominently, largely exceeds the performance of existing neural solvers, and also achieves comparable results as well-designed symbolic solvers.

The contributions of this work are summarized in four folds: (1) We use textual clauses to express the fine-grained structural and semantic information in geometry diagram efficiently. (2) We propose a new neural solver PGPSNet, fusing multi-modal information through structural and semantic pre-training, data augmentation, and self-limited decoding. (3) We construct a large-scale dataset PGPS9K labeled with both fine-grained diagram annotation and interpretable solution program. (4) Our PGPSNet outperforms existing neural solvers significantly.

2 RelatedWork

2.1 Multi-modal Reasoning

Multi-modal reasoning uses multi-modal datasets to perform reasoning tasks, e.g., visual question answering [Anderson et al., 2018; Kim et al., 2021], document visual question answering [Xu et al., 2020; Tito et al., 2021], table question answering [Zhu et al., 2021; Lu et al., 2023], where GPS is also a special multi-modal reasoning task. Because of difference of data modality and reasoning ability, it results in significant semantic gaps between different tasks. The core to multi-modal reasoning lies on how to unite modalities and incorporate domain knowledge.

2.2 Geometry Problem Solving

Generalized GPS contains geometry calculation [Seo et al., 2015; Tsai et al., 2021] and geometry proving [Chou et al., 1996; Gan et al., 2019], and can be divided into symbolic solvers [Seo et al., 2015; Lu et al., 2021] and neural solvers [Chen et al., 2021; Chen et al., 2022] in method. The symbolic solvers that have been studied for years possess their advantages and limitations as introduced in Introduction. With development of neural network, neural solvers have shown their potential in GPS, whereas they still cannot handle modal representation and fusion well. In context of the geometry calculation, our work unifies the geometry diagram and textual problem into the text modality to better construct structural and semantic relationships in geometry.

2.3 Pre-trained LanguageModel forMathematical Reasoning

Language models [Lewis et al., 2020; Brown et al., 2020], pre-trained on large text corpus with self-supervised learning tasks such as MLM [Devlin et al., 2019] or CLM [Lewis et al., 2020], have demonstrated remarkable performance gains on wide range of NLP tasks such as text classification [Minaee et al., 2021] and question answering [Khashabi et al., 2020]. Inspired by them, pre-trained language model is also applied to mathematical reasoning task gradually, e.g., math word problem solving [Liang et al., 2022; Zhang and Moshfeghi, 2022] and GPS [Lu et al., 2021; Chen et al., 2022]. However, lacking of large-scale mathematical corpus and targeted pre-training tasks, existing language models pre-trained on natural corpus seem to have limited effects on downstream reasoning tasks with only fine-tune tactic. In this work, we pre-train language model with MLM task combining textual clauses and textual problem, equipping model with capacity of structural and semantic understanding in geometry and therefore improving the performance of GPS substantially.

3 PGPS9KDataset

Although several datasets [Seo et al., 2015; Sachan et al., 2017; Lu et al., 2021; Chen et al., 2021] for GPS have been proposed, as presented in Table 1, they are either in small scale only for rule-based symbolic solvers, or coarse-grained annotated neglecting rich information in diagram. To facilitate the application of neural solver, we build a new largescale GPS dataset called PGPS9K 1 labeled both fine-grained diagram annotation and interpretable solution program. To the best of our knowledge, PGPS9K is the largest and the most complete annotation dataset for GPS up to now.

3.1 Collection and Description

PGPS9K is composed of 9,022 geometry problems paired with non-duplicate 4,000 geometry diagrams, where 2,891 problems paired with 1,738 diagrams are selected from Geometry3K dataset [Lu et al., 2021], the rest of problems are collected from five popular textbooks across grades 612 on mathematics curriculum websites 2 . Our PGPS9K is divided into 30 problem types as exhibited in appendix A.1, covering almost all problem types of plane geometry problem in corresponding grades. As shown in Figure 2, PGPS9K dataset has five properties: (1) Theorem-based: Solving problems in PGPS9K need to apply geometric theorem knowledge to carry out algebraic calculation and get numerical results finally; (2) Diagram-dependent: Above 90% of problems must be solved using the diagrams because necessary conditions such as variable content and geometric structure are displayed via visual form instead of text; (3) Abstract: The diagram is integrated with basic geometric primitives (point, line, circle) and non-geometric primitives (text, symbol). No complex semantic scenarios are involved in textual problem except abstract geometric conditions; (4) Fine-grained: Problems with the same diagram vary in conditions or targets. Slight distinctions in textual problems usually lead to completely different solutions to problems; (5) Condition-redundancy: Lots of conditions in semantic clauses or textual problem are not needed in problem solving at hand. These five properties make PGPS9K focus on the challenges at geometric reasoning and alleviate the bias introduced by the text [Manjunatha et al., 2019; Patel et al., 2021]. Moreover, for convenience of experimental comparison, we split PGPS9K in two ways: The first is leaving out the test set of Geometry3K [Lu et al., 2021] as test set (589) and other disjoint samples as training set (8,433); The second is dividing samples of each problem type according to ratio of 8:1 (training set 8,022 and test set 1,000).

1 http://www.nlpr.ia.ac.cn/databases/CASIA-PGPS9K

2 https://www.mheducation.com/

3.2 Annotation Form

The annotations of PGPS9K include diagram annotation and solution program, where the diagram annotation is to extract structural and semantic information in diagram and the solution program defines the solution steps of problem.

Diagram annotation adopts the same primitive level labels as [Zhang et al., 2022] which includes primitive contents and primitive relations in tuple form. Then we translate them into two kinds of textual clauses: structural clauses and semantic clauses. The structural clauses are confined to the connection relationship among geometric primitives and described by clauses with points on lines or points on circles, wherein points are arranged in order. The connection relation reveals the most fundamental structural relation displayed in diagram but omitted in textual problem. The semantic clauses depict basic relations between geometric primitives and nongeometric primitives with natural language. These relations are necessary parts for problem solving and complement each other in diagram and textual problem. Noting that the definition and descriptive approach of textual clauses remain open and the overall design principle is to characterize complete features of diagram to help with GPS. More details about textual clauses please refer to appendix A.2.

Our solution program gives the geometric solution procedure consisting of several deduction steps. It is composed of 34 operators OP and 55 operands PN, where a operator and a few of related operands form one step. Each operator implies one geometric theorem or axiom wherein operands involved are sorted according to the corresponding theorem formula. Operands can be divided into four types: problem variables N presented in textual problem or semantic clauses, process variables V generated during the process, arguments ARG are alphabetic unknowns [a - z], and constants C. For example, the Pythagorean theorem reveals the relationship of right sides and hypotenuse in right triangle with theorem formula a2 + b2 = c2 , so we express it as "Gougu(a, b, c)". Compared with other annotation methods proposed [Amini et al., 2019; Tsai et al., 2021; Chen et al., 2021], our annotation eliminates elementary arithmetic operations such as +, -, *, /, and thus has advantages of structuralization, knowledge-guiding and interpretability (as illustrated in Figure 3), making the solution program more concise and reducing the difficulty of model learning. Besides, we firstly introduce process variables V as unknown variables in intra-step and as transfer variables in inter-step, unifying the forward and reverse operations within one theorem. For instance, in the Pythagorean theorem, "Gougu(V , *, *)" and "Gougu(*, *, V )" can be set to solve the right side and hypotenuse, respectively. Paired with annotation form of solution program, we also create a powerful program executor to compute numerical results. It implements symbolic algebraic operations with multiple unknowns according to theorem formulas based on SymPy library of Python. More details of solution program are demonstrated in Appendix A.3.

4 PGPSNet Model

4.1 Overall Framework

To fully fuse multi-modality of geometry problem, we propose a new neural solver PGPSNet as depicted in Figure 1. The inputs of PGPSNet include the geometry diagram D and problem text T , where the problem text consists of structural clauses Tstru, semantic clauses Tsem and textual problem Tprob that T = {Tstru, Tsem, Tprob} = {ti}ni=1 . The diagram image is encoded with convolutional neural network (CNN) and the problem text is passed through the structural and semantic pre-trained language model. Then these two modal tokens are concatenated together, fed into the bidirectional GRU encoder to perform fusing encoding. Next they are decoded by the self-limited GRU decoder to get the solution program Y = {yi}mi=1 correspondingly. Finally the solution program is calculated by the program executor and obtain the numerical result of geometry problem.

4.2 Structural and Semantic Pre-training

While textual clauses describe the fine-grained structural and semantic information parsed from diagram, these clauses are low-level and lack of overall structure and context. Additionally, long, trivial and disordered texts still bring great difficulty to modal fusion and semantic comprehension. Inspired by the pre-training language model, we design a structural and semantic pre-training method based on masked LM task [Devlin et al., 2019] for the text modality learning as shown in Figure 4.

Firstly, we assign the class tag and section tag for each token in text. The class tag indicates the semantic class of tokens in five classes: general, variable, argument, point and angle ID. The section tag refers to section that tokens belong to, where a problem is divided into structure, condition and target three sections. The input embedding ei of language model fuses not only positional encoding but also embedding of class tag and section tag as:

Fine-grained class tags and section tags promote the modeling of geometry problems and alleviate the imbalance of textual token. Then we mask 30% of text tokens with mask tokens following [Cho et al., 2021] but keep tags unchanged. The model is trained to recover the masked text in a unified text generation manner.

This pre-training method is extremely applicable to structural and semantic modeling of geometry under our modal representation. For instance, it can be reasoned that the mask token in the semantic clause "BD ⊥ EA on [M]" is "C" according to structural clauses "line B C D" and "line E C A", facilitating model learning the geometric knowledge of line intersection. However, in some cases, model may cannot infer the mask content exactly but geometric knowledge is filled in its token candidates. Taking instance of the textual problem "Find [M]D.", there is high probability that the mask token is "C" or "B" according to the structural clause "line B C D". In summary, pre-training makes model acquire advanced geometric cognition that is an elementary skill for GPS.

4.3 Encoder and Decoder

In our model, the CNN encoder only extracts coarse-grained global visual features of diagram such as geometric style, to determine possible operations quickly and accelerate model convergence. The GRU encoder integrates the diagram encoded as one visual token and textual tokens enhanced by structural and semantic language model, and obtains the mixed encoding context HE = {hEi }n+1 i=1 as output.

Because of complexity and flexibility of solution process of geometry problem, solution program cannot convert into binary or general expression tree. Tree decoders widely used in math word problem [Xie and Sun, 2019; Tsai et al., 2021] are not applicable to GPS. We propose a self-limited GRU decoder to generate the sequential solution program in an autoregressive manner. The differences between self-limited decoder and the general attention-based decoder [Bahdanau et al., 2015] are two folds: (1) Selflimited decoder reduces token embedding space. The input embeddings of problem variables N and augments ARG presented in problem text are copied from encoder output, which also enriches decoder inputs with contextual semantic. Specifically, token embeddings are defined as:

where VV , VC, VN and VARG are target vocabularies of process variables, constants, problem variables and augments, loc(y, T ) is the location of token y in problem text T . (2) Selflimited decoder narrows the search space of output tokens. It limits the output candidates of problem variables N and augments ARG into that appear in the problem text. Concretely, the probability of predicted token y is:

where y ∈ {VV , VC, VN ∩ T, VARG ∩ T }, Score is the score function, hD is the hidden output of decoder, c is the context vector generated from HE using the same attention mechanism as [Bahdanau et al., 2015], e(y) is the token embedding of candidates. In experiments, We find that the self-limited GRU achieves even better performance than complex tree decoders and with much faster training and inference speed.

5 Data Augmentation

Despite that PGPS9K is the largest dataset so far and of highquality, it still cannot satisfy the model learning of PGPSNet well, especially for the structural and semantic pre-training task. Therefore, we adopt five data augmentation strategies based on diversity and equivalence of geometric representation, taking the problem in Figure 4 as an illustrative example:

• Token Replacement: The replaceable tokens include points, angle IDs and augments three types. Once a token is changed, all same tokens in textual clauses and textual problem should be replaced uniformly. Point B is replaced as point V, then get new texts: "line V C D", "⊙A lieson E V D", "VD ⊥ EA on C", "Find VD".
• Connection Rotation: The connection relationship in structural clauses could be re-represented by changing the location order of points. "line B C D" is equivalent to "line D C

B" with the opposite order, "⊙A lieson E B D" is the same as "⊙A lieson E D B" in clockwise order.

• Representation Transposition: There are several equivalent representations of geometric primitives of line, angle and arc, e.g., "EA = AE", "̸ STR = ̸ RTS", "̂ EF= ̂ FE". We randomly transpose the geometric primitives representation in semantic clauses.
• Clauses Shuffle: We shuffle semantic clauses to produce new ID of problem variable while modify the corresponding solution program. When semantic clauses are adjusted as "CA = 3(N0), BD ⊥ EA on C, EC = 2(N1)", the solution program is changed as "· · · Gougu N0 V1 V0 · · · ".
• Diagram Flip: Since the textual content is already parsed in semantic clauses, the visual text in diagram could be ignored. So the flipped or rotated diagram is identical to the original diagram for problem.

These five augmentation strategies are independent and can be incorporated with each other. Abundant samples generated from data augmentation equip model with basic geometric knowledge, thus promoting high-level reasoning.

6 Experiment

6.1 Setup

Implementation details. Our model is implemented using Pytorch on one GTX-RTX GPU. The CNN model adopts the ResNet10 [He et al., 2016], feeding with diagram images resized as 128 × 128. The language model select a transformer encoder [Vaswani et al., 2017], having 6 layers, 8 attention heads, and a hidden embedding size of 1024. The GRU encoder is a two-layer bidirectional GRU [Cho et al., 2014] with input embedding size 256 and hidden state size 512. The selflimited decoder is a two-layer GRU setting same input embedding size and hidden state size of 512. The random probability of data augmentation is set as 0.7 in pre-training and 0.5 in training. We choose the AdamW optimizer [Loshchilov and Hutter, 2017] with weight decay 1e-2 and step decline schedule with decaying rate 0.5. During pre-training, the learning rate of language model is initialized as 5e-4 decaying at 1K, 2K and 3K epochs with a total 4k epochs. During training, all modules of PGPSNet train together with initial learning rate as 5e-5 for language model and 1e-3 for other modules, decaying at 160, 280, 360, 440 and 500 uniformly with a total 540 epochs. In addition, the batch size and dropout rate are set as 128 and 0.2 in all processes.

Evaluation metrics. We evaluate performance of geometric solvers at two levels: numerical answer and solution program. At each level, there are three evaluation patterns: Completion, Choice and Top-3. In Completion, the symbolic solver gives the result searched for and the neural solver selects the first executable solution program. The Choice is defined as choosing the correct option from four candidates but selecting one randomly if answer is not in. The Top-3 computes the accuracy ratio of correct answer existing among top three solution candidates, less strict than Completion pattern. We set beam size to 10 being the same as [Chen et al., 2021] for all evaluation manners. Noting that the performance of solution program is often lower than the actual result due to the equivalence of operation orders and diversity of solution strategies. Datasets. We conduct experiments on PGPS9K dataset split in two ways as introduced in Section 3.1, signified as Geometry3K and PGPS9K in the following experiments. To simplify experiments, we adopt the textual clauses generated from ground truth of diagram annotation for both symbolic solvers and neural solvers. The language model is pretrained from scratch on our PGPS9K on account of huge gap from natural corpus and short of geometric corpus. Datasets GeoQA [Chen et al., 2021] and GeoQA+ [Cao and Xiao, 2022] are not considered in experiments because they have no fine-grained diagram annotation and their diagrams are lowquality, hard to be parsed by existing diagram parsers.

6.2 Comparison with State-of-the-art Methods

To evaluate our PGPSNet solver, we compare its performance with state-of-the-art GPS solvers InterGPS [Lu et al., 2021], NGS [Chen et al., 2021] and Geoformer [Chen et al., 2022]. Symbolic solvers. The InterGPS is a rule-based symbolic solver which parses the problem text and diagram into formal language. Table 2 displays the performances of InterGPS in three input modes with the best search strategy, where Inter-GPS(Predict) indicates that diagram and text formal language are predicted by its diagram and text parser, InterGPS(DiagramGT) indicates that text formal language is generated by its text parser but diagram formal language uses the ground truth, Inter-GPS (All GT) indicates that all utilize the ground truth. On Geometry3K, the results show that our PGPSNet outperforms Inter-GPS(Predict), achieves comparable performance as Inter-GPS(DiagramGT), and is inferior to InterGPS(All GT) in Completion. But in Choice, PGPSNet has surpassed all input modes and even gains a 2.0% improvement over InterGPS(All GT). On PGPS9K, we implement the InterGPS(Diagram GT) by converting diagram annotation into diagram formal language and using its text parser. The PGPSNet shows more performance improvements in Completion and Choice, and result in Top-3 implies the improvement potential of our PGPSNet.

Neural solvers. NGS and Geoformer are two neural solvers and we re-implement them on our dataset. For full and fair comparison, we keep their pre-trained diagram encoder fixed, take both semantic clauses and textual problem as their text input, and employ the same data augmentation. Thanks to our good modal representation and effective modal fusion, our PGPSNet demonstrate superior performance improvements compared to baseline neural solver, NGS and Geoformer as displayed in Table 2. Besides, we note that there remains a huge performance gap between our solver and human expert, having much room for improvement.

6.3 Ablation Study

To demonstrate effects of different modules of PGPSNet, we conduct ablation studies on Geometry3K, taking the selflimited decoder, data augmentation, structural clauses and pre-training as ablation objects as displayed in Table 3. The comparison between row 1 and row 4 shows that the data augmentation promotes GPS through adding geometric representation knowledge into the augmented data. By comparing row 2 to row 4, we find that the self-limited decoder improves performance of geometric reasoning, who limits feature and search space to reduce difficulty of model learning. The language model, with structural and semantic pretraining, brings an amazing performance gain, especially in Completion with a 26.6% answer accuracy improvement, as shown in row 4 and row 6. We also discover that structural clauses show less affect without pre-training compared row 3 with row 4, but obtain a remarkable improvement with pretraining as displayed in row 5 and row 6, revealing that basic connection relations can promote model's structure cognition by a befitting modal fusion approach. The performance trends of solution program are consistent with the numerical answer in all experiments.

6.4 Case Study

We also conduct case studies for discussing the ability and limitation of our PGPSNet as shown in Figure 5. The case (a) examines the application of Angle Bisector Theorem. The methods of NGS and PGPSNet w/o LM fail to determine the relation of corresponding edges correctly while our PGPSNet obtains the right solution program. The case (b) is challenging and need to combine two conditions of Segment Length Theorem. Solutions of all solvers are incorrect for case (b) but the PGPSNet's is most similar to the ground truth with only wrong in the second step, illustrating that PGPSNet is not yet qualified for complex geometric reasoning nowadays but has great potential.

7 Conclusion

In this work, we propose a new diagram-text fusion solver PGPSNet combining textual clauses parsed from diagram, and construct a large-scale and fine-annotated GPS dataset PGPS9K. Benefiting from effective modal representation and efficient modal fusion, PGPSNet makes full use of basic structural and semantic information to implement geometric reasoning. Besides, the interpretable solution program and well-designed data augmentations provide model with critical geometric knowledge for GPS such as geometry theorem and geometric representation. The experimental results demonstrate the potential of neural solvers and our work still has large room for improvement with more training samples or more elaborate modal fusion.

Acknowledgments

This work has been supported by the National Key Research and Development Program under Grant No. 2020AAA0109700, the National Natural Science Foundation of China (NSFC) grants U20A20223 and 61721004.