Solving Math Word Problems with Teacher Supervision

1 Introduction

Making computers to understand and solve math word problems (MWPs) is a very important and fundamental task of Nature Language Understanding (NLU) and has been studied a half century ago [Bobrow, 1964]. As it shows in Table 1, a math word problem usually contains text description with some given quantities, then has a question regarding an unknown quantity. The answer of the question is a mathematical formula showing how to infer the unknown quantity correctly. Building automatic MWP solvers remains a challenging task due to the wide semantic gap to parse the human-readable words into machine-understandable logics so as to facilitate quantitative reasoning [Zhang et al., 2019].

Since deep neural networks have shown their effectiveness on addressing diverse nature language processing (NLP) tasks, MWPs solvers are recently designed to 'translate' math problems into a mathematical expression via Seq2Seq models [Wang et al., 2017; Wang et al., 2018; Wang et al., 2019; Li et al., 2019; Xie and Sun, 2019; Zhang et al., 2020b], which have a typical encoder-decoder structure: the encoder learns the representation of the problems, while the decoder transforms the latent representation to a math formula expression (e.g., x = 2/10 in Table 1). Compared with earlier semantic-parsing methods [Shi et al., 2015; KoncelKedziorski et al., 2016; Huang et al., 2017] or templatebased machine learning approaches [Kushman et al., 2014; Hosseini et al., 2014; Roy and Roth, 2018], Seq2Seq models are able to generate new formula expressions that have not been seen in the training data. Moreover, tree-structured decoder is recently introduced to model the relationship between quantities [Liu et al., 2019; Xie and Sun, 2019]. To catch the relationships and order information among the quantities in MWPs, a Graph2Tree model is proposed in [Zhang et al., 2020b], which has a graph-based encoder to learn the latent quantity representations and a tree-based decoder to generate solution expression trees.

∗Corresponding Author

Although the Seq2Seq based models have achieved promising results, they fail when a delicate variation is applied to the word expression of a math problem, and leads to a remarkably different answer. For example in Table 1, Problem1 and Problem2 have similar expression in text and math, but different answers. Graph2Tree [Zhang et al., 2020b] trained by samples including Problem1 gives a wrong solution to Problem2. The key reason is that Seq2Seq based models do not intentionally segregate MWPs with different answers but similar math formula expression. Figure 1 visualizes the representation of two types of MWPs from the encoder in Graph2Tree model by T-SNE. We can see that some MWPs with solution (a ∗ b)/c have similar representation of to those with solution (a - b)/c. They differ only by one operator but have obviously different meanings. Therefore, we can conclude that the existing Seq2Seq based models, including Graph2Tree, concentrate only on the generation of expression, e.g., (a ∗ b)/c or (a - b)/c, as correct as possible from the given MWPs, and are weak on managing of distribution of MWPs representation obtained from the encoder. The overlap of two different types of MWPs can cause wrong solution generation, and lead to wrong answers.

We propose to intentionally separate the representations of MWPs with similar expression by a teacher supervision. The idea is from the process how we are taught to solve math problems. We are supervised by a human teacher to give out the correct solutions, and are also warned to avoid the wrong solutions, such that we master math problems by knowing what is correct and how the correct answer is different from the wrong ones. Therefore, we add a teacher module to make the encoder generate the representation matching the correct solution but disaccording from the wrong solutions, which are manipulated from the correct solution. By doing so, the representation of MWPs with different solutions but similar expression can be more separated, as shown in Figure 1(b).

The overall framework of our proposed solution is illustrated in Figure 2. In the training process, the encoder and decoder are jointly updated to generate correct solutions, while the encoder is additionally trained to generate high-quality representations that are examined by the teacher with both correct and manipulated wrong answers. Meanwhile, the teacher is trained to make accurate judgement about the encoded representation. Once the training is completed, the teacher module is discarded. The trained encoder-decoder takes input a given MWP and solves it by outputting the inferred solution and answer.

To evaluate the effectiveness of our proposed model, we conducted extensive experiments on two benchmark MWPs datasets and verified the following advantages of our model:

• The designed teacher module guides the encoder to
• learn separable representation between MWPs with similar math formula expression but markedly different answers. The resulted MWP solver outperforms state-ofthe-art baselines.
• The teacher module can flexibly work with any Seq2Seq based MWP solvers, with various encoder and decoder architectures, and always improve the solvers with better encoded representation and more accurate solutions.
• The idea of presenting wrong solutions manipulated from correct solution strengthens the capability of MWP solvers. In particular, manipulating both numbers and operators in the math expression is more effective than manipulating only numbers or operators. This resonates our idea of introducing the teacher supervision to know not only what are correct and also what are wrong.

2 RelatedWorks

MWPs were initially solved by rule-based methods [Fletcher, 1985; Bakman, 2007] and traditional statistical methods [Mitra and Baral, 2016; Kushman et al., 2014]. However, those prior methods usually use a template to generate a solution, and thus require all templates to be covered in the training set. Otherwise the problems out of the training templates will be solved with wrong solutions. Recently, MWPs are addressed as a translation task, translating nature human language into mathematical language. Therefore, Seq2Seq models with encoder-decoder structure become the dominant solver. The vanilla Seq2Seq was initially employed in [Wang et al., 2017], and achieved impressive performance. Then, equation normalization method [Wang et al., 2018] is proposed to deal with the problem that A + B and B + A are both effective solutions and have the same result. Inspired by the success of Transformer [Vaswani et al., 2017], multi-head attention mechanism is also applied in MWP solver [Li et al., 2019]. On the one hand, the encoder in Seq2Seq is designed to learn with graphs to enrich the feature representation of MWPs in [Zhang et al., 2020b]. Teacher-student network [Zhang et al., 2020a] is also applied in math problem solver, which leverages knowledge distillation and differs from our approach. On the other hand, the decoder in Seq2Seq is popularly constructed as tree-based math expression generators [Wang et al., 2018; Liu et al., 2019]. Moreover, inspired by the human-like goaldriven study, a tree-structure recursive decoder achieves great performance and becomes a standard choice of decoder for MWPs solvers [Xie and Sun, 2019]. All above-discussed models have no control on reducing the overlap of two different types of MWPs in the problem embedding space. Our designed teacher module can be plugged into those state-ofthe-art solvers and improve their performance.

3 ProposedMethods

3.1 Problem Statement

Solving MWPs is to take the input of a text sequence W = {w1,w2, ...,wn}, and give the output answer sequence A = {A1,A2, ...,Am}. Here n is the length of W and m is the length of A. Each wi in the text sequence W is a word, and Ai in the math answer sequence A is one numeric value from Vnum, or one operator from Vop, or one constant from Vcon. Note that, Vnum contains all the numeric values that appeared in the input sequence W, defined by their orders. The set Vop contains all the operators in the output, e.g., '+', '-'. Vcon contains all the constant values used in the answer, e.g., π.

3.2 Model Architecture

The designed model architecture is shown in Figure 2. It contains an encoder module, a decoder module and a teacher module. We next introduce these modules in details.

Encoder Module

Following the state-of-the-art MWP solver GTS model [Xie and Sun, 2019], we employ a bidirectional Gated Recurrent Unit (GRU) [Cho et al., 2014] as our encoder for learning the MWP representation. Firstly, the text sequence W is presented as a sequence of the embedding vectors corresponding to the words in W. The resulted sequence Wembed is given to the bidirectional GRU and transformed to the representation Z ∈ Rn∗d ,Z = ---→ GRU (Wembed) + ←--- GRU(Wembed) (1)

where d is the embedding dimension, ---→ GRU denotes the gated unit for left-to-right direction, and ←--- GRU denotes the unit for right-to-left direction.

In Graph2Tree model [Zhang et al., 2020b], a graph transformer is employed as the MWP encoder for learning the representation Z. Our model is in fact flexible for the usage of any form MWP encoder, and can be deployed with the future powerful encoders, which is not the focus of this study.

Decoder Module

The decoder is expected to output answer sequence A = {A1,A2, ...,Am}, from which a tree-based math expression can be built. One example of the tree is shown in Figure 3, where numeric values are at leaf nodes and operators sit at the non-leaf nodes. The tree has no parenthesis and makes the answer expressed in a simple way. The decoder in [Xie and Sun, 2019; Zhang et al., 2020b] predicts the pre-order traversal sequence of the expression tree, and has been shown as an effective generator for the mathematical expression for the solution.

To generate the tree, the root node is featured by adding two last hidden states of the GRU encoder:

where qroot ∈ Rd is the latent feature of the root node, ---→ GRU(Wembed)n is the last hidden state of forward sequence at position n, ←--- GRU(Wembed)0 is the last state of backward sequence at position 0.

The root node (featured as qroot) is put in a stack. The generator takes the decoding steps given next to construct the tree, i.e., predicting the token of Ai in the answer sequence A. In each step, the prediction is in fact to classify the node feature q to one of token in {Vnum ∪ Vop ∪ Vcon}: Ai = ft(q). Here, ft is implemented as a two-layer neural network following [Xie and Sun, 2019]. The numeric values in Vnum are extracted from the W, the Vop is a set of known operators, and the Vcon is given by prior knowledge.

Step 1 Pop one node from the stack. Predict the token of current node, if the prediction Ai is a number (in Vnum or Vcon), repeat step 1 until the stack is empty. If the prediction is an operator (in Vop), go to step 2.

Step 2 Generate two children nodes for the current node. For the left child, calculate its representation ql = fL(qp, Z), where qp is the parent node representation and Z is the encoder output. For the right child, calculate its representation qr = fR(qp, Z, ql). Here, fL and fR are implemented as a two-layer neural network with gating mechanism following [Xie and Sun, 2019]. Then, push the right child into stack. Predict the token of the left child node. If the prediction is a number, go to step 1. If the prediction is an operator, repeat step 2.

Teacher Module

With the designed encoder and decoder module, the typical Seq2Seq-based MWP solvers (e.g., [Xie and Sun, 2019; Zhang et al., 2020b] optimize the neural network parameters by maximizing the probability of generating the sequences A with input W in training data. Our proposed teacher module additionally examines the conformity between the representation Z of input problemW and the ground-truth sequence A.

As shown in Figure 2, an answer sequence A is sent to a bidirectional GRU, for getting its representation hA ∈ Rm∗d :

The representation hA can then be compared with the representation Z for measuring their conformity, since they represent the same math problem but are obtained from two different views. Thus, the mean vector of hA and Z are concatenated and sent to a fully-connected (FC) layer. Their conformity is predicted as a binary classification problem:

where ¯ Z ∈ Rd and ¯hA ∈ Rd are the mean vector of hA and Z, respectively.

Considering to extract higher level features about the problem W, we stack another layer of bidirectional GRU on Z, and have the second option for measuring the conformity,

where hZ ∈ Rn∗d is the representation of problem W from two-layer bidirectional GRU, and ¯hZ ∈ Rd is the mean vector over the n hidden states (corresponding to n words). In our evaluation results, we compared these two options and show that the higher-level representation hZ is more comparable to hA than Z in the conformity measurement.

To train the FC classifier, both positive examples and negative examples should be provided. Positive examples are those Z and A pairs from training data. Negative samples should be specially designed for meeting our teacher supervision target. In pairwise ranking recommendation systems, negative samples are shown to play important roles for promote the recommendation results [Rendle et al., 2009]. In

Algorithm 1 Negative Answer Generation

Input: Positive answer A = {A1, A2, ..., Am} Parameter: Disturbance probability λ Output: Negative answer Aneg = {Aneg 1 , ...,Aneg m } 1: Let Aneg = A 2: while Aneg == A do 3: for i = 1 to m do 4: p ← a random value between 0 and 1 5: if p<λ then 6: if Ai ∈ Vnum then 7: Aneg i ←a random element from Vnum \ Ai 8: else if Ai ∈ Vop then 9: Aneg i ←a random element from Vop \ Ai 10: end if 11: end if 12: end for 13: end while 14: return Aneg

our case, one simple idea is to randomly select an answer from a different problem to be a negative sample of the given problemW. However, the selected answer can be markedly different from the positive answer A, making it as a too easy negative sample to be distinguished from the positive. Then, the classifier works for a too simple task as a supervisor. To this end, we design a negative answer generation algorithm for manipulating the positive answer A to get a valuable negative answer Aneg for the given problem, as shown in Algorithm 1. The parameter λ is a disturbance probability, deciding if changing the current token in A or not. Empirically, we set λ as 0.1. For each positive answer, we can generate different number of negative answers for training the teacher module. The evaluation results reported later show that one negative answer is enough.

3.3 Learning Objectives

Loss Function for Teacher Module. The teacher module conducts binary classification for the positive and negative samples. The objective is to minimize the binary crossentropy loss:Lteacher(θ) = - log P (y = 1|A, Z, θ)

where θ denotes the network parameters to optimize in the teacher module, including those in the bidirectional GRU and the FC classifier network.

Loss Function for Encoder-Decoder Module. Given a training pair (W,A), the Encoder-Decoder module mainly targets on maximizing the probability of generating A for W . Moreover, with the participation of the teacher module, the encoder should maximize conformity of the representation Z and the answer A. Thus, the loss function to minimize is:

where φ denotes the network parameters in the EncoderDecoder module, and parameter α tunes the weight of teacher module loss in the training process.

4 Experimental Settings

4.1 Dataset

We take two widely used benchmark datasets for the experimental evaluation, Math23K and MAWPS.

Math23k. Math23k [Wang et al., 2017] is one of the most commonly used dataset for MWP solver evaluation. It has 23161 math word problems. Each of these problems has one ground truth mathematical expression and answer value. A test set including 1000 problems is provided in Math23k. Most of the prior work evaluates their MWP solvers on this test set. There are also others using a 5-fold cross validation to measure the performance of their solvers. In our experiments, we report the accuracy for both settings.

MAWPS. MAWPS [Koncel-Kedziorski et al., 2016] is a relatively small dataset which only contains 2373 problems. We also perform 5-fold cross validation on this dataset. To make a thorough evaluation, the performance of our solvers and baseline solvers are also evaluated on this dataset. However, the ablation study and case study are conducted on the larger dataset Math23k.

4.2 Implementation Details

The embedding dimension of Wembed is set to 128. The latent feature dimension d is set to 512. Our models are trained for 120 epochs. The training was conducted in two stages. Stage 1-minimizing Eq.(6) and the first term of Eq.(7): we train the teacher and the encoder-decoder module separately at first. Stage 2-minimizing Eq.(6) and Eq.(7): after 30 epochs when the teacher was able to give a reasonable guidance to encoderdecoder, we apply teacher loss (i.e., the second term in Eq.(7)). We use Adam optimizer [Kingma and Ba, 2014] with initial learning rate 0.001, which is halved every 30 epochs. Dropout [Hinton et al., 2012] on embedding matrix of probability 0.5 is employed to prevent overfitting. During testing, we use beam search of size 8 to generate the math expression sequence. The weight α in Eq. (7) is set to 0.1, by its sensitivity analysis. Our code in Python with Pytorch framework can be found at https://github.com/derderking/MWP-teacher.

4.3 Baselines

We conduct a comprehensive comparison with the state-ofthe-art baselines, including:

• Deep Neural Solver (DNS) [Wang et al., 2017], a vanilla Seq2Seq model;
• Math-EN [Wang et al., 2018], employing math equation normalization to reduce the space of solution;
• T-RNN [Wang et al., 2019], applying tree-structured templates;
• Group-ATT [Li et al., 2019], employing multi-head attention;
• GTS [Xie and Sun, 2019], a goal-driven tree-structure solver with a powerful recursive decoder;
• Graph2Tree [Zhang et al., 2020b], enriching the representation from encoder by using graph neural networks.

To avoid the implementation error that may cause unreproducible results of baseline models, we reported the results of these baselines from the papers where they are published. Since the testing set is fixed, we report our results on the same testing data when varying the encoder-decoder architectures. The GTS and Graph2Tree model have achieved better performance than all other baselines. We thus evaluate our approach with the encoder-decoder in GTS and Graph2Tree, to show the effectiveness of the proposed teacher supervision strategy.

The evaluation metric is solution accuracy, which is used as a common setting in MWP solver evaluation. We firstly translate the generated solution tree into a numerical value, then compare it with the ground truth value. The percentage of correctly addressed problems in the testing set is reported as accuracy.

5 Experimental Results

5.1 Accuracy Analysis

Table 2 shows the performance of baseline models, and the performance of GTS and Graph2Tree with our proposed teacher module. It is obvious that our teacher supervision can improve the accuracy of both GTS and Graph2Tree by a notable margin among different evaluation settings and datasets. For example, the accuracy on Math23k dataset of GTS model is improved by 0.9% with teacher loss, i.e., addressing 9 problems more correctly in the testing set. The accuracy of Graph2Tree model increases 1.7%, i.e., addressing 17 problems more correctly in the testing set. Although the solved problem set increased only by less than 20, this is already a big step, as the left un-solved problems are those specially challenging one. Note that the improvement of Graph2Tree over GTS was also just 1.8%. In the end, among 1000 problems in Math23K testing set, the model Graph2Tree with Teacher can correctly address 791 problems. This new stateof-the-art result is contributed by both the advanced encoder architecture in Graph2Tree, and the proposed teaching module on better learning of the that encoder. Later in case study, we will report the problems that Graph2Tree failed but Graph2Tree+Teacher addressed correctly, to further analyze the effectiveness of the proposed teacher supervision.

5.2 Ablation Study

To further evaluate the contribution of the teacher module, we conduct extensive ablation experiments. All the experiments here are conducted on the fixed testing Math23k dataset, while accuracy is still used as the evaluation metric.

Analysis of the Generated Negative Samples

The negative samples are generated in Algorithm 1 by manipulating the positive sample A. We compare the manipulation on different elements and report the results in Table 3. The manipulation on both operators and numbers is more effective than the manipulation of either one. This resonates our idea of introducing the teacher supervision to know not only what are correct and also what are wrong.

Regarding the number of negative samples to generate, the results in Table 3 show that using only one negative sample is sufficiently good. Too many negative samples discourage the representation learning of the encoder.

The Sensitivity Analysis of α

The weight α in Eq. (7) tunes the importance of the teacher loss. We report its sensitivity in Table 4. In general, keeping α as a relatively small value (e.g., α =0.1) is appropriate. A too large value of α will emphasize too much on the teacher loss and make the model lose control on the answer generation. However, a too small weight on teacher loss will have a limited impact on improving the accuracy of the model.

The Usage of Z and hZ in Teacher Classifier

We also compare the two options of Eq.(4) (using Z) and Eq.(5) (using hZ) in teacher module. The results in Table 4

show that using hZ is the key to make the teacher module work. Because the higher level features hZ is more appropriate for measuring the conformity with the generated A.

5.3 Case Study

Table 1 has already shown a case study, comparing the performance of Graph2Tree and Graph2Tree+Teacher on solving a math problem that has a delicate variation from the known problem. Table 5 shows another case study, where the testing sample is varied slightly from a training sample on word expression. However, the Graph2Tree model gives a wrong solution, while the Graph2Tree+Teacher solves it correctly.

6 Conclusion

In this paper, we target on solving MWPs, especially those with similar text description, but markedly different solutions. We propose to intentionally separate their representations by a teacher supervision, which measures the conformity between the problem representation and the ground truth solution in math expression. The resulted representations of MWPs with different solutions but similar expression are more separated than those from the baseline models. Experimental results show that the teacher module can make Graph2Tree achieve new state-of-the-art performance. Moreover, our teacher loss is only computed in training phase, without any computational burden during testing. In our future work, we will explore how to use dynamic negative sampling which has been widely used for training recommendation systems to get better negative samples.

Acknowledgements

This work is supported by King Abdullah University of Science and Technology (KAUST), Saudi Arabia.