TabKD: Tabular Knowledge Distillation Through Interaction Diversity of Learned Feature Bins

TabKD: Tabular Knowledge Distillation Through Interaction Diversity of Learned Feature Bins

Shovon Niverd Pereira, Krishna Khadka, Yu Lei

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence
Main Track. Pages 4723-4731. https://doi.org/10.24963/ijcai.2026/526

Data-free knowledge distillation enables model distillation without original training data, which is critical for privacy-sensitive tabular domains. However, existing methods do not perform well on tabular data because they do not explicitly address feature interactions, which are critical for encoding predictive knowledge. We identify interaction diversity, systematic coverage of feature combinations, as an important factor for effective tabular model distillation. To operationalize this insight, we propose TabKD, which learns adaptive feature bins aligned with teacher decision boundaries, then generates synthetic queries that ensure uniform pairwise interaction coverage. Across 4 benchmark datasets and 4 teacher architectures, TabKD achieves the highest student-teacher agreement in 14 of 16 configurations, outperforming 5 state-of-the-art baselines. We further show that interaction coverage strongly correlates with distillation quality, validating our core hypothesis. Our work establishes interaction-focused exploration as a principled framework for tabular model distillation.
Keywords:
Machine Learning: Adversarial machine learning
Machine Learning: Knowledge-aided learning