Segmenting Chinese Microtext: Joint Informal-Word Detection and Segmentation with Neural Networks

Segmenting Chinese Microtext: Joint Informal-Word Detection and Segmentation with Neural Networks

Meishan Zhang, Guohong Fu, Nan Yu

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence
Main track. Pages 4228-4234. https://doi.org/10.24963/ijcai.2017/591

State-of-the-art Chinese word segmentation systems typically exploit supervised modelstrained on a standard manually-annotated corpus,achieving performances over 95% on a similar standard testing corpus.However, the performances may drop significantly when the same models are applied onto Chinese microtext.One major challenge is the issue of informal words in the microtext.Previous studies show that informal word detection can be helpful for microtext processing.In this work, we investigate it under the neural setting, by proposing a joint segmentation model that integrates the detection of informal words simultaneously.In addition, we generate training corpus for the joint model by using existing corpus automatically.Experimental results show that the proposed model is highly effective for segmentation of Chinese microtext.
Keywords:
Natural Language Processing: Phonology, Morphology, and word segmentation
Machine Learning: Neural Networks
Natural Language Processing: Natural Language Processing