Generalization-Aware Structured Regression towards Balancing Bias and Variance

Generalization-Aware Structured Regression towards Balancing Bias and Variance

Martin Pavlovski, Fang Zhou, Nino Arsov, Ljupco Kocarev, Zoran Obradovic

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence
Main track. Pages 2616-2622. https://doi.org/10.24963/ijcai.2018/363

Attaining the proper balance between underfitting and overfitting is one of the central challenges in machine learning. It has been approached mostly by deriving bounds on generalization risks of learning algorithms. Such bounds are, however, rarely controllable. In this study, a novel bias-variance balancing objective function is introduced in order to improve generalization performance. By utilizing distance correlation, this objective function is able to indirectly control a stability-based upper bound on a model's expected true risk. In addition, the Generalization-Aware Collaborative Ensemble Regressor (GLACER) is developed, a model that bags a crowd of structured regression models, while allowing them to collaborate in a fashion that minimizes the proposed objective function. The experimental results on both synthetic and real-world data indicate that such an objective enhances the overall model's predictive performance. When compared against a broad range of both traditional and structured regression models GLACER was ~10-56% and ~49-99% more accurate for the task of predicting housing prices and hospital readmissions, respectively.
Keywords:
Machine Learning: Ensemble Methods
Machine Learning: Machine Learning
Machine Learning: Structured Prediction