A Walkthrough for the Principle of Logit Separation

A Walkthrough for the Principle of Logit Separation

Gil Keren, Sivan Sabato, Björn Schuller

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence
Best Sister Conferences. Pages 6191-6195. https://doi.org/10.24963/ijcai.2019/861

We consider neural network training, in applications in which there are many possible classes, but at test-time, the task is a binary classification task of determining whether the given example belongs to a specific class. We define the Single Logit Classification (SLC) task: training the network so that at test-time, it would be possible to accurately identify whether the example belongs to a given class in a computationally efficient manner, based only on the output logit for this class. We propose a natural principle, the Principle of Logit Separation, as a guideline for choosing and designing loss functions that are suitable for SLC. We show that the Principle of Logit Separation is a crucial ingredient for success in the SLC task, and that SLC results in considerable speedups when the number of classes is large.
Keywords:
Machine Learning: Deep Learning
Machine Learning Applications: Big data ; Scalability
Machine Learning: Classification