Multi-Scale Spatial-Temporal Integration Convolutional Tube for Human Action Recognition

Multi-Scale Spatial-Temporal Integration Convolutional Tube for Human Action Recognition

Haoze Wu, Jiawei Liu, Xierong Zhu, Meng Wang, Zheng-Jun Zha

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence
Main track. Pages 753-759. https://doi.org/10.24963/ijcai.2020/105

Applying multi-scale representations leads to consistent performance improvements on a wide range of image recognition tasks. However, with the addition of the temporal dimension in video domain, directly obtaining layer-wise multi-scale spatial-temporal features will add a lot extra computational cost. In this work, we propose a novel and efficient Multi-Scale Spatial-Temporal Integration Convolutional Tube (MSTI) aiming at achieving accurate recognition of actions with lower computational cost. It firstly extracts multi-scale spatial and temporal features through the multi-scale convolution block. Considering the interaction of different-scales representations and the interaction of spatial appearance and temporal motion, we employ the cross-scale attention weighted blocks to perform feature recalibration by integrating multi-scale spatial and temporal features. An end-to-end deep network, MSTI-Net, is also presented based on the proposed MSTI tube for human action recognition. Extensive experimental results show that our MSTI-Net significantly boosts the performance of existing convolution networks and achieves state-of-the-art accuracy on three challenging benchmarks, i.e., UCF-101, HMDB-51 and Kinetics-400, with much fewer parameters and FLOPs.
Keywords:
Computer Vision: 2D and 3D Computer Vision
Computer Vision: Action Recognition