Earth Mover's Distance Pooling over Siamese LSTMs for Automatic Short Answer Grading

Earth Mover's Distance Pooling over Siamese LSTMs for Automatic Short Answer Grading

Sachin Kumar, Soumen Chakrabarti, Shourya Roy

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence
Main track. Pages 2046-2052. https://doi.org/10.24963/ijcai.2017/284

Automatic short answer grading (ASAG) can reduce tedium for instructors, but is complicated by free-form student inputs. An important ASAG task is to assign ordinal scores to student answers, given some “model” or ideal answers. Here we introduce a novel framework for ASAG by cascading three neural building blocks: Siamese bidirectional LSTMs applied to a model and a student answer, a novel pooling layer based on earth-mover distance (EMD) across all hidden states from both LSTMs, and a flexible final regression layer to output scores. On standard ASAG data sets, our system shows substantial reduction in grade estimation error compared to competitive baselines. We demonstrate that EMD pooling results in substantial accuracy gains, and that a support vector ordinal regression (SVOR) output layer helps outperform softmax. Our system also outperforms recent attention mechanisms on LSTM states.
Keywords:
Machine Learning: Neural Networks
Natural Language Processing: Natural Language Processing