Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error

Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error

Taiji Suzuki, Hiroshi Abe, Tomoya Murata, Shingo Horiuchi, Kotaro Ito, Tokuma Wachi, So Hirai, Masatoshi Yukishima, Tomoaki Nishimura

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence
Main track. Pages 2839-2846. https://doi.org/10.24963/ijcai.2020/393

Compression techniques for deep neural network models are becoming very important for the efficient execution of high-performance deep learning systems on edge-computing devices. The concept of model compression is also important for analyzing the generalization error of deep learning, known as the compression-based error bound. However, there is still huge gap between a practically effective compression method and its rigorous background of statistical learning theory. To resolve this issue, we develop a new theoretical framework for model compression and propose a new pruning method called {\it spectral pruning} based on this framework. We define the ``degrees of freedom'' to quantify the intrinsic dimensionality of a model by using the eigenvalue distribution of the covariance matrix across the internal nodes and show that the compression ability is essentially controlled by this quantity. Moreover, we present a sharp generalization error bound of the compressed model and characterize the bias--variance tradeoff induced by the compression procedure. We apply our method to several datasets to justify our theoretical analyses and show the superiority of the the proposed method.
Keywords:
Machine Learning: Deep Learning
Machine Learning: Deep-learning Theory
Machine Learning: Learning Theory
Machine Learning: Feature Selection; Learning Sparse Models