Extracting Visual Knowledge from the Web with Multimodal Learning

Extracting Visual Knowledge from the Web with Multimodal Learning

Dihong Gong, Daisy Zhe Wang

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence
Main track. Pages 1718-1724. https://doi.org/10.24963/ijcai.2017/238

We consider the problem of automatically extracting visual objects from web images. Despite the extraordinary advancement in deep learning, visual object detection remains a challenging task. To overcome the deficiency of pure visual techniques, we propose to make use of meta text surrounding images on the Web for enhanced detection accuracy. In this paper we present a multimodal learning algorithm to integrate text information into visual knowledge extraction. To demonstrate the effectiveness of our approach, we developed a system that takes raw webpages as input, and automatically extracts visual knowledge (e.g. object bounding boxes) from tens of millions of images crawled from the Web. Experimental results based on 46 object categories show that the extraction precision is improved significantly from 73% (with state-of-the-art deep learning programs) to 81%, which is equivalent to a 31% reduction in error rates.
Keywords:
Machine Learning: Classification
Natural Language Processing: Tagging, chunking, syntax, and parsing
Robotics and Vision: Vision and Perception