On the Intrinsic Differential Privacy of Bagging

On the Intrinsic Differential Privacy of Bagging

Hongbin Liu, Jinyuan Jia, Neil Zhenqiang Gong

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence
Main Track. Pages 2730-2736. https://doi.org/10.24963/ijcai.2021/376

Differentially private machine learning trains models while protecting privacy of the sensitive training data. The key to obtain differentially private models is to introduce noise/randomness to the training process. In particular, existing differentially private machine learning methods add noise to the training data, the gradients, the loss function, and/or the model itself. Bagging, a popular ensemble learning framework, randomly creates some subsamples of the training data, trains a base model for each subsample using a base learner, and takes majority vote among the base models when making predictions. Bagging has intrinsic randomness in the training process as it randomly creates subsamples. Our major theoretical results show that such intrinsic randomness already makes Bagging differentially private without the needs of additional noise. Moreover, we prove that if no assumptions about the base learner are made, our derived privacy guarantees are tight. We empirically evaluate Bagging on MNIST and CIFAR10. Our experimental results demonstrate that Bagging achieves significantly higher accuracies than state-of-the-art differentially private machine learning methods with the same privacy budgets.
Keywords:
Machine Learning: Ensemble Methods
Multidisciplinary Topics and Applications: Security and Privacy