Binary Classification from Positive Data with Skewed Confidence

1 Introduction

Predicting human behaviour and mental states is a key technology to promote well-being society [Matthews et al., 2014; Lane et al., 2014]. Some previous studies trained a predictor by utilizing labeled data collected in laboratory experiments with a relatively small number of subjects [AlHanai and Ghassemi, 2017; Shinoda et al., 2019]. On the other hand, large-scale data related to users' physical and physiological activities have been available as IoT devices and smartphones have been widely adopted these days [Weiss et al.

Liu et al., 2018]. However, the data collection and annotation processes are still highly expensive when the fully-supervised method is employed. The cost reduction is therefore a mandatory for the success of real-world applications [Bao and Intille, 2004; Kaji et al., 2019].

For instance, prediction of drowsy driving is one of the important industrial applications [Hachisuka et al., 2011]. A drowsy driving predictor is usually trained with datasets which include both Alert and Drowsy states collected in experiments using a driving simulator [Vural et al., 2007]. However, driving until drivers feel strong drowsiness is extremely time-consuming because they generally put much effort into staying awake to avoid accidents [Horne and Reyner, 1999]. The manual annotation of drivers' states also takes a lot of time and effort since human mental states are ambiguous, and it is hard even for experts to accurately distinguish different states. Hence, it is quite helpful in saving the cost of data collection and annotation if a drowsiness predictor can be constructed only from Alert samples.

Weakly-supervised classification [Zhou, 2018] is an essential approach to drastically reduce the annotation costs. Various problem formulations have been considered so far depending on what types of information is available as weaklylabeled data, which are less informative but less expensive than fully-labeled data [Mintz et al., 2009; Bao et al., 2018]. Semi-supervised classification [Chapelle et al., 2010; Zhu et al., 2003; Chapelle and Zien, 2005; Sakai et al., 2017], positive-unlabeled (PU) classification [Elkan and Noto, 2008; du Plessis et al., 2014], and unlabeled-unlabeled classification [Lu et al., 2019; du Plessis and Sugiyama, 2013] are the typical examples of weakly-supervised classification to utilize unlabeled data. There also have been studies in the other problem settings including multiple instance classification [Dietterich et al., 1997], partial label classification [Cour et al., 2011], complementary classification [Ishida et al., 2017], and similar-unlabeled classification [Bao et al., 2018]. In addition, noisy-label classification [Liu and Tao, 2016; Natarajan et al., 2013] and PU classification with selection bias [Kato et al., 2019] are methods for dealing with uncertainties in classification, which is an important research topic from a practical viewpoint.

In this paper, we pay special attention to positiveconfidence (Pconf) classification [Ishida et al., 2018], which is a unique weakly-supervised binary classification method that can learn only from positive data and the confidence. Compared with anomaly detection methods such as the oneclass support vector machine [Scholkopf and Smola, 2001], Pconf classification can practically achieve high classification accuracy and has a theoretical guarantee in the framework of empirical risk minimization. However, the performance of the Pconf classifier may be deteriorated when the confidence is skewed. We will use the term skew henceforth in this paper to indicate that the distribution of the positive confidence is deformed due to bias, which often occurs in an annotation process.

In practice, the confidence of positive samples may be obtained based on a tacit knowledge learned from past experience. The confidence can easily be skewed in this situation when there is bias caused by a lack of experience (the sample size experienced in the past was small) or imbalanced experience (abundant knowledge on a positive class but little knowledge on a negative class) for example.

[Ishida et al., 2018] has demonstrated that the Pconf classification was robust against noisy confidence. However, to the best of our knowledge, the more general case of the skewed confidence has not been considered for Pconf classification. The purpose of this paper is to extend Pconf classification to be able to cope with the skewed confidence, which will significantly enhance its practicality.

The challenge of this paper is to give a practical method to address the problem of the skewed confidence in the Pconf classification setting. In order to correct the skewed confidence, we introduce a parameterized model of the confidence. However, we may not be able to directly optimize the confidence model since we only have positive samples. Our key idea to overcome this difficulty is to assume a prior knowledge, the misclassification rate of positive samples, and optimize the confidence model through the minimization of the squared difference between the misclassification rate and an empirical validation error. Is this assumption reasonable? As typified by the examples below, some real-world problems fit well with our assumption:

Example 1 Store managers want to predict whether their customers will continue to come to their stores or not based on the customers' attributes data and a loyalty score. They cannot employ fully-supervised methods since they do not have data of rival stores while Pconf classification is available by using the loyalty score as the positive confidence. However, the evaluation of the loyalty score tends to be too optimistic because the store managers do not have the information on non-customers. On the other hand, they empirically know the overall churn rate.

Example 2 Engineers want to predict whether a person is sleepy or not from her physiological information. The self-report score of drowsiness can be used as the ground truth [Shinoda et al., 2019] to train a Pconf classifier. They empirically know the ratio of drowsiness in daytime, but the person may underreport the score because of embarrassment when she feels the strong drowsiness.

Then we experimentally demonstrate the effectiveness of the proposed method through synthetic toy problems with linearin-input models and benchmark problems with neural network models. Finally, we apply our method to a real-world problem of drivers' drowsiness prediction to show the practical usefulness.

2 ProblemFormulation

In this section, we review the original problem setting of Pconf classification [Ishida et al., 2018].

Let x ∈ Rd be a d-dimensional pattern and y ∈ {+1, -1} be its class label. Consider a situation where positive samples and the confidence are only available for training a classifier:

where xi is a positive sample independently drawn from p(x|y = +1), and ri = p(y = +1|xi) is the confidence. Because we do not have negative samples under this problem setting, the classification risk minimization represented by the following formulation cannot be directly executed:

where p(x, y) is the joint density that test data follows, g(x) : Rd → R is a binary classifier, and ℓ(z) is a loss function. Let π+ = p(y = +1) and r(x) = p(y = +1|x), and let E+ denote the expectation over p(x|y = +1). Then the classification risk R(g) is written as follows [Ishida et al., 2018]:

When minimizing R(g) with respect to g, π+ can be neglected since it is a proportional constant. Therefore, empirical risk minimization can be executed by using positive samples and the confidence as follows:

3 Adjusted Pconf Classification

We then introduce our problem setting, that is, Pconf classification with the skewed confidence, and propose the method for alleviating the negative impact of skew.

In order to correct the skewed confidence ˆ r, we employ the following formulation with adjusted confidence rk i for 0 < k < ∞:

Note that rk i ∈ (0, 1] and ri ≤ rj ⇔ rk i ≤ rk j . In this model, the skewed confidence is supposed as ˆ r = r-k 1 .

The hyperparameter k may be selected via cross validation if we have a validation set which includes both positive and negative samples. However, we cannot optimize k since we only have positive samples in the current setup. Thus it may not be possible to figure out the presence of skew in confidence and its magnitude only from positive samples.

To overcome this difficulty, we assume that we know the misclassification rate of positive samples φ as a prior knowledge:

1 In this paper, we consider the exponential model for the ease of controlling the flatness of the confidence distribution r(x). Other adjusted confidence models such as an additive model max (0, min(1, r + k)) , -1 ≤ k ≤ 1 can also be employed.

Under this assumption, optimal hyper-parameter k∗ may be selected by minimizing the squared error between the empirical classification error and φ:

where ℓ01(z) = (1-sign(z))/2 is the zero-one loss. Finally, the adjusted Pconf classifier is obtained via the following optimization problem:

4 Experiments

We examine the behaviour of our proposed method on simple synthetic data, and demonstrate the performance of our method on benchmark image datasets. We implemented the experiments on Python using Optuna [Akiba et al., 2019], PyTorch 2 , and Scikit-learn [Pedregosa et al., 2011].

4.1 Synthetic Experiments

In this toy experiment, a training set, a validation set, a test set, and a dataset for estimating confidence were created from positive and negative samples drawn independently from twodimensional Gaussian distributions. All the dataset except the validation set consisted of 1,000 positive samples and 1,000 negative samples. The validation set included only 1,000 positive samples. We computed the probabilistic output of ℓ2regularized linear logistic regression as positive confidence, and transforming it to skewed confidence by taking the b-th power (b ∈ {0.3, 0.5, 2.0, 4.0}). Confidence lower than 0.01 was rounded up to 0.01 to stabilize the optimization.

We compared original Pconf, adjusted Pconf, and the fullysupervised method. Linear-in-input model g(x) = α⊤x + β and the logistic loss ℓL (z) = log (1 + e-z ) were used for all of these methods. Then the empirical risk minimization for Pconf classification can be formulated as follows:

Adam [Kingma and Ba, 2015] with 5,000 epochs was used for optimization. Note that negative samples were not used for training the original Pconf and adjusted Pconf classifiers, and selecting the hyperparameter k.

Impact of Degree of Class Overlap and Skewed Confidence

We first examine the impact of the degree of class overlap and skewed confidence on the behaviour of Pconf classification. We fixed the mean of the positive distribution µ+ to [0, 0]⊤, and moved the mean of the negative distribution µ- from [1.0, 1.0]⊤ to [3.0, 3.0]⊤. We used the identity covariance for both the positive and negative distributions. In addition, we changed the degree of skewed confidence

2 https://pytorch.org/

.

b ∈ {0.3, 0.5, 2.0, 4.0}. We calculated the mean false negative (FN) rate over 10 trials with fully-supervised classification as ̂ φ (an estimate of φ).

The mean and standard deviation of the classification accuracy over 10 trials are reported in Table 1. Adjusted Pconf worked well with all the candidates of the degree of class overlap and skewed confidence. It was statistically significantly better than original Pconf in all the cases, but difference in accuracy got smaller as the negative distribution moved away from the positive distribution. This result indicates that skewed confidence has less impact on problems where two distributions are easily separable because a small change of a decision boundary does not affect the classification result in such a situation. The visual illustrations of the results are presented in Figure 1.

Impact of Estimation Error of φ

In the first experiment, we assumed that we had access to a precise estimate of the misclassification rate of positive samples φ. However, this assumption would not always hold in practice. We then conducted the experiments with different values of ̂ φ to see the robustness of our proposed method against the estimation error of φ. We used the mean FN rate reported in Table 1 multiplied by c ∈ {0.5, 0.7, 1.3, 1.5} as ̂ φ with estimation error.

Table 2 presents the results where b = 0.3 and 4.0. Although the classification accuracy decreased compared to the case with no estimation error in φ, the decrease in accuracy was insignificant in some cases. Additionally, the accuracy of adjusted Pconf in Table 2 was better than the accuracy of original Pconf reported in Table 1 even for the cases with 50% estimation error (c = 0.5 and 1.5). This result demonstrates the usefulness of our proposed method in a practical situation where the value of φ cannot be precisely estimated.

4.2 Benchmark Experiments

We conducted two benchmark experiments with deep neural network models to evaluate the performance of our proposed method on image datasets. The experimental setups were based on [Ishida et al., 2018].

Fashion-MNIST

The Fashion-MNIST dataset is a set of 28 × 28 gray-scale images each of which represents one of the following 10 fashion item classes: T-shirt/top, pullover, dress, coat, sandal, shirt, sneaker, bag, and ankle boot. Each class consists of 7,000 images. We chose T-shirt/top as the positive class and one of the others as the negative class to construct a dataset for binary classification. We then divided it into four stratified sub-datasets: a training set, a validation set, a test set, and a dataset for estimating confidence.

We compared the performance of original Pconf classification, adjusted Pconf classification and fully-supervised classification. Skewed confidence was used for training both original and adjusted Pconf classifiers. We also included in this experiment original Pconf classification with non-skewed confidence to evaluate the performance decline caused by skew in confidence. The logistic loss was used for all of these methods.

We used a three-layer fully-connected neural network (784-100-100-1) with ReLU [Nair and Hinton, 2010] as an activation function and 10-7 weight decay. Optimization was executed by Adam [Kingma and Ba, 2015] for 200 epochs with minibatch size 100.

We generated positive confidence from a neural network with a softmax output layer and the same architecture as the aforementioned binary classifiers, but we used 50% dropout [Srivastava et al., 2014] after each fully-connected layer instead of weight decay. Additionally, we set the number of epochs to 20 in order to prevent the probabilistic output from being too close to 0 or 1, which is not suitable for the representation of confidence. We made confidence skewed by removing 90% negative samples from a dataset for training a softmax network. As a result, the dataset became imbalanced (P : N = 10 : 1), and confidence was skewed towards the positive class. Confidence lower than 0.01 was rounded up to 0.01 to stabilize the optimization. We used the mean FN rate over 20 trials with fully-supervised classification as an estimate of φ.

The mean and standard deviation of the classification accuracy over 20 trials are reported in Table 3. Although skewed confidence severely affected the performance of original Pconf, it performed well with the accuracy over 90% in some cases where the supervised method had relatively high accuracy. This result was consistent with the synthetic experiment. Adjusted Pconf outperformed original Pconf in most of the cases, and was even competitive with fully-supervised classification and original Pconf with non-skewed confidence in some cases.

CIFAR-10

The CIFAR-10 dataset consists of 60,000 images in the 32 × 32× RGB format. It includes 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.

We chose airplane as the positive class and preprocessed the original dataset in the same way as the experiment with the Fashion-MNIST dataset.

In this experiment, we used a convolutional neural network model with the following architecture:

• Convolution (3 in- /18 out-channels, kernel size 5).
• Convolution (18 in- /48 out-channels, kernel size 5).
• Max-pooling (kernel size 2, stride 2).
• Max-pooling (kernel size 2, stride 2).
• Fully-connected (400 units) with ReLU.
• Fully-connected (800 units) with ReLU.
• Fully-connected (1 unit).

The results of the CIFAR-10 experiment are shown in Table 4. Adjusted Pconf outperformed original Pconf in all cases. Although the accuracy of adjusted Pconf was significantly lower than that of the fully-supervised method, adjusted Pconf was comparable to original Pconf with nonskewed confidence in some cases.

5 Real-World Application

In this section, we apply original and adjusted Pconf classification to the real-world problem of drivers' drowsiness prediction. Through the experiment, we show that the confidence may be skewed in a real scenario, and can be corrected by our proposed method.

The objective of this application is to predict drivers' drowsiness from the heartbeat information. Typically, a classifier for this problem can be trained with an experimental dataset including both Alert and Drowsy states collected by using a driving simulator. However, there are two issues with this way of building a classifier. The first one is derived from the fact that a driver reacts differently to a driving simulator and the real road situation [Hallvig et al., 2013]. A classifier trained with data from a driving simulator experiment may be less useful in the real traffic environment, but an experiment in the real environment is difficult and dangerous because drowsy driving easily leads to accidents. The second one is that driving until feeling strong drowsiness and the manual annotation of drivers' states are extremely timeconsuming even though we use a driving simulator. For these two issues, there is a strong motivation to learn a binary classifier without collecting Drowsy samples.

5.1 Drivers' Drowsiness Dataset

We used the drivers' drowsiness dataset [Kaji et al., 2018], which was constructed from the driving simulator experiment. Three healthy males engaged in the driving task with a driving simulator along an expressway around 100 km/h overtaking other cars if necessary. The task continued until the strong drowsiness was observed or a driver finished the whole course (about 150 km). The facial expressions and the electrocardiogram (ECG) signal were recorded during the driving task to construct the dataset which includes the feature vector x and the drowsiness score D. The feature vector x was composed of seven heart-rate-variability indicators computed in the frequency domain (the spectral power of the low frequency component, etc) and time domain (the variance of the peak-to-peak interval of the R-wave, which is the largest wave of ECG, etc.). These features were computed at 60 seconds intervals with 120-second sliding windows, and normalized to have zero mean and unit standard deviation for each driver. Expert j (j = 1, 2, 3) independently evaluated the drowsiness score Dj , which was rated from 1 ("Not sleepy") to 5 ("Very sleepy"), based on the recorded facial expressions [Kitajima et al., 1997] every 60 seconds.

Each driver performed the driving tasks 10 times, and only five trials were annotated and the remaining five trials were unlabeled. We used the annotated five trials in this study.

5.2 Experimental Setup

We divided the collected data into Alert (positive) and Drowsy (negative) classes to reframe the drowsiness prediction as a binary classification problem. The samples with the median of the drowsiness score Dj less than 3 were labeled as Alert, otherwise the samples were labeled as Drowsy. We also calculated the positive confidence from the drowsiness score Dj as follows, which is confined between 0 and 1:

We compared the performance of original Pconf classification, adjusted Pconf classification, and fully-supervised classification. The logistic regression with the Gaussian kernel of bandwidth 1.0 was used for all of these methods.

In this experiment, the evaluation of the classification performance was conducted independently for each driver. More specifically, the samples of four driving trials by the same driver were used for training a classifier, and the samples of the remaining one trial were used as a test set. We repeated this procedure five times for each driver so that all trials were tested once. The F-measure was used as the performance metric since the dataset was imbalanced.

We computed the mean FN rate over 20 trials with fullysupervised classification for each driver, and then used the average of the mean FN rate of two drivers as an estimate of φ for the remaining one driver. Only Alert samples and their confidence were used for training original Pconf and adjusted Pconf classifiers, and hyperparameter tuning.

5.3 Results and Discussion

The results are reported in Table 5. The F-measure could not be calculated for original Pconf because it predicted all the samples to be Alert. This result suggests that there existed strong skew towards the Alert class in the confidence. It can be explained by the fact that it is difficult even for the experts to find out the sign of drowsiness on driver's face because how the facial expressions change when she gets sleepy varies across drivers. Consequently, the drowsiness score tends to be lower than the actual drowsiness of drivers. Such bias can arise in any situation where there is asymmetric difficulty in evaluating confidence.

On the other hand, adjusted Pconf worked reasonably well with the drowsiness dataset. This result indicates that our method is also effective for the real-world problem where the positive confidence is given based on the subjective rating.

6 Conclusion

In this paper, we proposed adjusted Pconf classification to mitigate the influence of skewed confidence, which is not easy to handle but naturally emerges in real-world applications. The key idea is the assumption of the misclassification rate of positive samples, which is minimal statistical information as a prior knowledge. The hyperparameter to correct skew in confidence is optimized through the minimization of the squared difference between the misclassification rate and validation error. We demonstrated the effectiveness of the proposed method through the synthetic and benchmark experiments, and the real-world drowsiness prediction problem.