Calibrating a Deep Neural Network with Its Predecessors

1 Introduction

Deep neural networks (DNNs) have achieved great successes across a variety of domains, especially on classification related tasks such as object detection [Wang et al., 2022; Zhou et al., 2021; Qiao et al., 2021] and image classification [Pham et al., 2021; Wortsman et al., 2022], reaching prediction accuracy far beyond human beings. However, they still suffer from mis-calibrated predictions in the sense that the prediction probability cannot represent the ground-truth probability.This may lead to fatal problems when any safetycritical downstream tasks such as autonomous driving [Bojarski et al., 2016] and medical diagnosis [Caruana et al., 2015] rely heavily on the prediction probability.

The underlying cause for mis-calibrated predictions is associated with the capacity of modern neural networks that makes them vulnerable to overfitting [Guo et al., 2017]. [Mukhoti et al., 2020] show that overfitting in modern neural networks mostly results from the overconfidence on misclassified samples and they empirically verify the strong connection between the overfitting issue and calibration performance. Given this observation, some regularization techniques such as weight decay [Guo et al., 2017], label smoothing [M¨uller et al., 2019], and data augmentation [Thulasidasan et al., 2019; Hendrycks et al., 2019] are introduced to improve model calibration.

Early stopping [Prechelt, 1998] is another well-known regularization method, which suspends training once the model performance stops improving on a hold out validation dataset. [Mukhoti et al., 2020] conduct a series of empirical experiments and demonstrate that early stopping on training according to multiple criteria fails to yield a well-calibrated model. We mainly attribute this sub-optimal solution to the unitary strategy of conventional early stopping techniques which treat the entire network as a whole. Specifically, an early stopping technique takes a DNN as a black box without investigating the internal components, i.e., the blocks inside the network. However, the increasing depth of modern DNNs makes the optimization more challenging, which could lead to discrepancies of convergence speeds of different blocks in DNNs. Thus, any model calibration via a conventional early-stopping technique could be a sub-optimal solution.

In this paper, instead of taking a DNN as a whole, we consider a block in a DNN as the basic unit and explore the overfitting problem in each block. We empirically observe that blocks in a network overfit at different stages during training. Unlike the conventional early stopping approach [Prechelt, 1998] that stops the training of the whole network at a certain point to form a network predecessor, we propose to stop the training of each block at its own best-fitting block predecessor to improve model calibration. However, the blocks in DNNs are strongly coupled with each other, and the early stopping of an individual block independently does not ensure an optimal solution. To achieve an effective and adaptive early stopping for each block, we take into consideration all possible block predecessor combinations. Our objective is to discover the predecessor combination (PC) with better calibration performance. We propose a neural architecture search inspired approach, predecessor combination search (PCS) to calibrate the DNNs, which performs a differential search of the optimal block predecessor combination through a relaxation of the search space as well as a predecessors evaluation estimator.

Our contribution can be summarized as follows: (1) We study the overfitting problem of individual blocks empirically and show that different blocks reach their own overfitting points at different stages of training. (2) We propose a novel differential PCS method to search a better-calibrated model together with a sampling strategy to improve searching efficiency. (3) PCS achieves state-of-the-art results for both pre- and post-temperature scaling [Guo et al., 2017] on a variety of datasets and architectures via a large number of experiments. We show that PCS works well on outof-distribution (OoD) samples by shifting the dataset from CIFAR-10 to SVHN [Goodfellow et al., 2013] and CIFAR10-C [Hendrycks and Dietterich, 2018].

2 RelatedWorks

Due to the mis-calibration problem in modern neural networks [Guo et al., 2017] and the significant importance of calibration, many techniqueshave been proposed in recent years. The current calibration methods could be divided into three categories. The first category modifies the training loss by replacing the conventionally used cross-entropy loss with a mean square error loss [Hui and Belkin, 2020] or a focal loss [Gupta et al., 2020] or by adding an auxiliary regularization loss such as the MMCE loss [Kumar et al., 2018] and the AvUC loss [Krishnan and Tickoo, 2020]. [Bohdal et al., 2021] propose a differentiable surrogate for expected calibration error that improves the calibration performance directly. The recent work in [Karandikar et al., 2021] introduces a differentiable bin membership function and applies it on binbased metrics such as expected calibration error (ECE) [Guo et al., 2017] to make it become a differentiable auxiliary calibration loss.

Another category is the post-hoc calibration approaches that improve the calibration performance by modifying the prediction logits. Platt scaling [Platt et al., 1999] learns parameters to perform a linear transformation on the original prediction logits. Isotonic regression [Zadrozny and Elkan, 2002] learns piece-wise functions to transform the original prediction logits. Histogram binning [Zadrozny and Elkan, 2001] obtains calibrated probability estimates from decision trees and naive Bayesian classifiers. Bayesian binning into quantiles (BBQ) [Naeini et al., 2015] is an extension of histogram binning with Bayesian model averaging. Beta calibration [Kull et al., 2017] is proposed for binary classification and [Kull et al., 2019] generalize the beta calibration method from binary classification to multi-classification with Dirichlet distributions. [Wenger et al., 2020] employ a non-parametric representation using a latent Gaussian process. Among these methods, temperature scaling is the most popular post-hoc approach, which tunes the temperature parameter of the softmax function that minimizes the negative log likelihood (NLL) and does not change the prediction results. In this work, we present the calibration performance with both before and after temperature scaling.

All other regularization methods that can calibrate networks form the third category. Label smoothing [M¨uller et al., 2019] implicitly calibrates networks by artificially softening targets to prevent a model from overfitting to the "hard label". Mixup [Thulasidasan et al., 2019] and AugMix [Hendrycks et al., 2019] are two popular data augmentation techniques for calibration. The mix step in data augmentation increases the generality of datasets and reduces the influence of hard samples that can easily cause overconfidence problem. Weight decay, which dominated regularization methods for neural networks in the past, is now less often used by modern neural networks. However, it still plays an important role in improving model calibration [Guo et al., 2017]. Learning with Retrospection(LWR) [Deng and Zhang, 2021] makes use of the learned information in the past epochs to guide the subsequent training, which benefit the classification prediction and uncertainty.

3 ProblemFormulation

Considering a dataset D = ⟨(xi, yi)⟩Ni=1 with N samples from a joint distribution (X, Y), the ground-truth class label is yi ∈ {1, 2, ...,K}, where K denotes the number of classes. The probability for a class yi on a given input xi predicted by network F with model parameters Θ is denoted as ˆ pi,yi = FΘ(yi|xi). The predicted label ˆ yi and the corresponding confidence ˆ pi are defined as

When the model is perfectly calibrated, the prediction confidence ˆ p is expected to represent the real probability p for each sample xi with class label yi. In other words, the model accuracy P(ˆ y = y|ˆ p = p) is p, for all p ∈ [0, 1].

ECE is a widely-accepted metric to measure calibration performance. Formally, the ECE is defined as the expected absolute difference between the model's confidence and its accuracy, which can be formulated as

Due to the finite samples in datasets, [Guo et al., 2017] estimate ECE by dividing confidence p ∈ [0, 1] into B equalwidth bins. Bi denotes the set of samples with confidences within ( i-1 B , iB ]. Let Ii and Ci denote the accuracy and average confidence of all samples in bin Bi respectively. Accuracy of bin Bi is computed as Ii = 1 |Bi | ∑j∈Bi 1 (ˆ yj = yj), where 1 is the indicator function, and ˆ yj and yj are the predicted and ground-truth labels for the j th sample. Similarly, the confidence Ci of the ith bin is computed as Ci = 1 |Bi | ∑j∈Bi ˆ pj, i.e., Ci is the average confidence of all samples in the bin. Thus, in practice, ECE is formulated as the weighted average of accuracy-confidence difference for the bins:

Along with ECE, the maximum calibration error (MCE) is proposed to minimize the influence of worst-case confidence deviation, which is defined as the maximum difference of bins' accuracy and confidence: MCE = maxi∈1,...,B |Ii - Ci|.

[Guo et al., 2017] and [Mukhoti et al., 2020] state that the mis-calibration problem of networks is strongly related to overfitting on training sets. Early stopping, as a well-known regularization technique, mitigates the overfitting problem by stopping the whole network at the best-fitting epoch. According to previous observations, early stopping should yield a well calibrated model. However, the calibration performance of an early stopped model is often far from satisfactory. Thus, we hypothesize that shallow and deep blocks may suffer from different degrees of overfitting during optimization and a unified regularization criteria to the entire network FΘ could lead to a sub-optimal solution for model calibration. Instead, it would be more desirable to resolve the overfitting problems block-wise and explore an adaptive regularization approach to improve the model calibration performance.

3.1 Overfitting in Blocks

Most advanced CNN model architectures are the stack of blocks. Suppose there are M blocks in the network, the model parameters can be represented as Θ = {θi|i = 1, 2, . . . ,M}, where θi denotes the parameter of the ith block. Θj and θj are used to identify the network and block parameters at the j th training epoch. The degree of fitting of FΘ is represented by the trend of validation NLL loss LNLL(FΘj (x), y), j = 1, 2, ..., Ttrain, where Ttrain is the number of total training epochs over the training process. Conventional early stopping aims to find the weights at a "sweet point" 1 epoch to balance between underfitting and overfitting. However, there is no definition of overfitting of individual network blocks. In order to find the "sweet point" epoch of each block, we need to measure the degree of overfitting of an individual block fi. We formally define the predecessors of block fi as {f j i |j ∈ {1, 2, . . . , Ttrain}}, where f j i denotes the ith block with weight at the j th epoch. To achieve the overfitting measurement of fi, we treat the other blocks of the network ˜ Fi = {fi-|i- ∈ {1, 2, . . . ,M} \ {i}} as constant mappings. Then the overfitting degree of block fi could be represented by the trend of validation NLL loss over the training process LNLL(F(x), y; f j i ), j = 1, 2, ..., Ttrain, with the weights of other network blocks ˜ Fi fixed. Based on this idea, an empirical study is conducted to investigate the overfitting behaviors of individual network blocks.

1 A sweet point is defined as the balance point between underfitting and overfitting, which normally refers to the lowest point of the validation loss curve.

We present the empirical study in Figure 1, demonstrating the different overfitting behaviors of individual network blocks. A ResNet-50 is first trained on CIFAR-10 for Ttrain = 350 epochs. Then each sub-figure shows the evaluation of overfitting behavior of block fi and each block other than fi is fixed to a certain predecessor. Note that it is nontrivial to properly select the fixed predecessors for all other blocks ˜ Fi and ideally these blocks need to be at their own "sweet points". We adopt a simplified approach and fix ˜ Fi at the "sweet point" of the whole model as an approximation, i.e., ˜ Fi = {f ρ∗ i- |i- ∈ {1, 2, . . . ,M} \ {i}}, where ρ∗ is obtained either by early stopping the whole model based on the validation loss or classification error. Apart from the validation loss, the validation ECE and classification error are also plotted in Figure 1. The proportionally scaled values of the validation loss and ECE are reported for better visualisation. Through the variation of the validation loss of each block fi in this experiment as shown in Figure 1, we have the following intriguing observations on overfitting in blocks.

1. Deeper convolutional blocks tend to have more severe overfitting problems with training. From the first four columns in Figure 1, we observe that all convolutional blocks with the same block index share a similar overfitting pattern for both settings. However, when we look into each row, a different pattern shows on the convolutional blocks compared with each other. The deeper convolutional block, i.e., f4, starts rapid overfitting after the first learning rate scheduler point.

1. The overfitting problem of the final linear layer is much more complex and depends heavily on other blocks. When it comes to the final linear block f5 at the last column, the loss presents a totally reverse pattern with different ˜ Fi. The final block f5 with ˜ Fi fixed at predecessors early stopping on loss has a slight overfitting problem from epoch 50 to the first learning rate scheduler point at epoch 150, while f5 with ˜ Fi fixed at predecessors early stopping on classification error continues to overfit from the very first epoch until a few epochs after the first learning rate scheduler point while the error remains at the same level.
2. Mis-calibration is linked with block overfitting. When we look at the variation of ECE and validation loss, we observe that the overfitting trend of an individual block is consistent with the variation of ECE, which further extends the link between mis-calibration and overfitting at block-wise level.

According to these observations, it could be very difficult to identify the ideal PC due to the inter-dependency among blocks (the linear blocks in the two rows have totally different behaviours). In other words, simply finding the best-fitting blocks individually and combining them together without taking other blocks into account may not produce a best-fitting model. This motivates us to explore an automatic searching algorithm to find a group of predecessors at "sweet points" that allows F ∗ = {f ρ∗i i |i = 1, 2, . . . ,M} to best-fit to the training set. Here, ρ∗i denotes the optimal block predecessor choice of block fi.

4 Methodology

Based on the observation on Figure 1, we propose to explore a PC representation P = {ρi|i = 1, 2, . . . ,M} to tackle the overfitting issue in blocks and improve model calibration performance. The objective can be formulated as

where F is {f ρi i |i = 1, 2, . . . ,M}, λ denotes a hyperparameter, ρi ∈ {1, 2, . . . , Ttrain} indicates the predecessor selection of block fi, ERR and ECE are the classification error and ECE respectively. The direct optimization of Eq. (4) is not feasible via gradient descent due to two issues. First, the discrete representation P makes Eq. (4) an optimization problem over a discrete domain. Second, both terms ERR and ECE in our objective are not differentiable. Thus, we introduce a PCS framework to tackle the aforementioned issues. Specifically, we first introduce differentiable combination sampling through a continuous relaxation of the PC representation. Proxy classification error and ECE landscape are then introduced to achieve differential optimization of our objective through an estimator for predicting the classification error and ECE.

4.1 Differentiable Combination Sampling

In our PCS framework, the objective is to discover the optimal selection of predecessors from candidate sets for each block.

To learn the selection, we first exploit a K-dimensional trainable parameter αi ∈ RK for this predecessor selection, where K denotes the number of candidates. And ρi can be obtained by the argmax of the selection parameter αi and further represented in a one-hot encoding format ρi ∈ RK . The PC representation P can be written as:

To relax the discrete PC representation for gradient-based optimization, we use the Gumbel-Softmax trick [Jang et al., 2016] to approximate the one-hot distribution and introduce randomness. The ρi in Eq. (5) can be relaxed as:

where τ is the temperature parameter, ξ k i is an i.i.d sample from Gumbel(0, 1), k and k′ denote the kth and k′th logit of corresponding K-dimensional vector respectively. We denote the relaxed PC representation as ˜ P = {˜ ρi|i = 1, 2, . . . ,M}. With the learning of αi, ˜ P explores the random combination at the beginning of search and gradually converges to a relatively stable state.

For simplicity, we denote the model with ˜ P as F˜ P . The classification error and ECE on the validation set can be denoted as ERR(Dval, F ˜ P ) and ECE(Dval, F ˜ P ) respectively. Since the combination is hard-combined, we fine-tune F˜ P with one more epoch on Dtrain, denoted as F ∗˜ P . In PCS, the original objective (Eq. (4)) can be reformulated as:

where A = {αi|i = 1, 2, . . . ,M} is the collection of learnable selection parameters for all blocks.

4.2 Proxy Classification Error and ECE Landscape

For differential optimization of PC representation ˜ P, we use a trainable estimator ψ to obtain proxies ˆ ERR, ˆ ECE = ψ( ˜ P) to approximate the classification error and ECE. Since the input ˜ P is a sequential data, we utilize a one-layer long shortterm memory (LSTM) to build the estimator ψ: RM×K → Rd mapping ˜ P to a d-dimensional embedding vector and a linear layer: Rd →R2 outputting ˆ ERR and ˆ ECE. The estimator ψ is trained with a weighted mean squared error loss function:

where γ is a hyperparameter to control the loss ratio of ECE, Tse is the total searching steps and the superscript (t) indicates the evaluation results at the tth time step. All pairs of ˜ P and its corresponding classification error and ECE are stored in a memory Π to optimize estimator ψ. After each searching step t, memory Π is updated by Π = Π ∪ {( ˜ P (t) : (ECE(t) ,ERR(t) ))}. We can then use the optimized estimator ψ∗ to reformulate PCS objective Eq. (7):

The gradients of ˆ ERR∗ and ˆ ECE∗ can be used to optimize ˜ P and thus A:

where A′ is the new predecessor selection parameter and η is the learning rate. At the next searching time step, the corresponding ˜ P ′ is based on A′ , and memory Π is updated to Π = Π ∪ {( ˜ P ′ : (ECE′ ,ERR′ )}.

Remark: Search Procedure We first train model FΘ with Ttrain epochs and randomly store weight parameters at K different epochs. After that, we randomly initialize a warmup population H of size S and evaluate the classification error ERR(t) (Dval, F ∗˜ Pi ) and ECE ECE(t) (Dval, F ∗˜ Pi ) of

˜ Pi ∈ H. The combination-performance pairs are then stored to memory Π, which is used to warm up estimator ψ and equip ψ with prior knowledge about classification error and ECE before searching. After the initial training of model and warming up of ψ, the searching procedure is conducted with Tse steps. In each step, one PC representation ˜ P (t) is sampled based on current A(t) . The corresponding F ˜ P (t) is then finetuned with one epoch and evaluated to obtain a validation error and ECE for the training of ψ. A(t) is then optimized with the proxy classification error ˆ ERR and proxy ECE ˆ ECE.

Remark: Search Space Sampling Strategy Each selection ρ is an integer between 1 and training epoch Ttrain, and thus the candidate size K is Ttrain. However, storing weight parameters over all training epochs can be very storage-intensive. For instance, storing all candidates of a ResNet-50 training with 350 epochs can take up to 33GB. Considering that the model parameters of adjacent epochs are similar, especially at late training stages, we propose four sampling strategies to reduce the size of search space and improve storage efficiency, which are (1) Random Sampling, randomly sampling K different epochs ranging from 1 to Ttrain; (2) Uniform Sampling, uniformly sampling K epochs ranging from 1 to Ttrain; (3) Laplace Sampling, due to model parameters changing much faster at earlier epochs, sampling K epochs ranging from 1 to Ttrain with a Laplace distribution centering at 0; (4) Piece-Wise Laplace Sampling, due to model parameters changing much faster at earlier epochs of each learning schedule, sampling K epochs ranging from 1 to Ttrain with multiple Laplace distributions centering at 0 and other learning schedule epochs (150 and 250 in our case). Since the searching difficulty grows exponentially with the size of K, another benefit of small candidate size is that it improves the searching efficiency.

5 Experiments

5.1 Experimental Settings

Datasets We conduct experiments on various datasets, including CIFAR-10/100 [Krizhevsky, 2012] and TinyImageNet [Deng et al., 2009] to evaluate the calibration performance. We also include the robustness evaluation on Outof-Distribution (OoD) datasets, including SVHN [Goodfellow et al., 2013] and CIFAR-10-C [Hendrycks and Dietterich, 2018].

Baselines To verify the effectiveness of our proposed algorithm, we include different networks for evaluation, including ResNet-50, ResNet-110 [He et al., 2016], Wide-ResNet26-10 [Zagoruyko and Komodakis, 2016] and DenseNet121 [Huang et al., 2017], and compare with various approaches, including training with weight decay at 5 × 10-4 (we find that weight decay at 5 × 10-4 performs the best among multiple values), Brier Loss [Brier et al., 1950], MMCE loss [Kumar et al., 2018], Label smoothing [Szegedy et al., 2016] with a smoothing factor αLS = 0.05, focal loss [Mukhoti et al., 2020] with regularisation parameter γfocal = 3, and scheduled focal loss FLSD-53 [Mukhoti et al., 2020] which uses γfocal = 5 for ˆ p ∈ [0, 0.2) and γfocal = 3 for ˆ p ∈ [0.2, 1).

Other Calibration Metrics Recent works [Nixon et al., 2019; Kumar et al., 2019; Roelofs et al., 2022; Gupta et al., 2020] point out the defects of ECE. To evaluate our method comprehensively, we evaluate our method on three additional calibration metrics, i.e., MCE, Adaptive-ECE [Ding et al., 2020] and classwise-ECE [Kull et al., 2019] along with ECE. We also measure PCS with reliability plots in supplementary. Training Setup For training on CIFAR-10/100, we set Ttrain = 350. The learning rate is set to 0.1 for epoch 0 to 150, 0.01 for 150 to 250, and 0.001 for 250 until the end of training. For training on Tiny-ImageNet, we set Ttrain = 100. We follow the same training and validation set spilt setting as [Mukhoti et al., 2020]. The learning rate is set to 0.1 for epoch 0 to 40, 0.01 for epoch 40 to 60, and 0.001 for 60 until the end of training. The fine-tuning learning rate is set to 10-4 for CIFAR-10, 5 × 10-4 for CIFAR-100, and 10-3 for Tiny-ImageNet. The searching process is performed with Tse = 100 steps. The population size is S = 100. Experiments are conducted with ResNet-50 on CIFAR-10 if there is no other specification. All networks are optimized using the SGD optimizer with a weight decay at 5 × 10-4 and a momentum of 0.9. The training batch size is set to 128. All experiments are conducted on a single Tesla V-100 GPU with all random seeds set to 1. Our code and results of comparison method are based on the public code and the pre-trained weight provided by [Mukhoti et al., 2020].

Temperature Scaling Following the setting in the prior work [Mukhoti et al., 2020], the temperature parameter τ is optimized by grid searching with τ ∈ [0, 0.1, 0.2, . . . , 10] on the validation set and finding the one with the best posttemperature-scaling ECE, which is also applied on the additional calibration metrics.

5.2 Calibration Performance

We report ECE(%) (computed using 15 bins) along with optimal temperatures in Table 1. PCS achieves the state-ofthe-art ECE across all models and datasets and outperforms previous works by large margins, especially pre-temperaturescaling results. More specifically, most PCS pre-temperaturescaling results have already substantially exceeded the posttemperature-scaling results of previous works. The result of ResNet-110 on CIFAR-100 achieves the best calibration performance compared to previous works, with a 7% decrease in ECE. For comparison approaches, the model trained with focal loss is broadly better-calibrated than other methods. However, it fails in some cases such as the evaluation on TinyImageNet. In addition, the scheduled γfocal trick does not always work better than fixed γfocal and it is hard to ascertain which is the better between FL-3 and FL53. The MMCE auxiliary loss performs worst before temperature scaling. Another notable point of PCS is that multiple results such as those with ResNet-110 on CIFAR-10/100 achieve the innately calibrated model (T=1.0), which means that the PC itself has already yielded a well-calibrated model without temperature scaling.

We also evaluate PCS on other widely-accepted metrics including Adaptive ECE, Classwise-ECE and test set error.

PCS also achieves the state-of-the-art calibration results on almost all cases. The test set error on Tiny-ImageNet shows a 8.52% decrease from 49.81% to 41.29%, which is mainly because of PCS performing as an early stopping trick.

5.3 Robustness on Out-of-Distribution(OoD) Datasets

A well-calibrated model helps improve the model robustness on OoD datasets [Thulasidasan et al., 2019]. However, temperature scaling is known to be fragile under dataset distribution shift [Ovadia et al., 2019]. PCS form innately calibrated models and thus perform well on OoD datasets. We utilize AUROC (the higher the better) to evaluate the robustness under dataset shift. Table 2 shows the AUROC (%) computed for models trained on CIFAR-10 and tested on the OoD datasets SVHN and CIFAR-10-C. Our method achieves competitive results on almost all cases. The results after temperature scaling tend to drop generally, and approaches yielding better pre-temperature-scaling ECE have better robustness on OoD datasets. Although focal loss works well on calibration, it fails under dataset shift. Our PCS with ResNet-50 on CIFAR-10 achieves a 4% increase compared to previous methods.

5.4 Searching Results

We visualize the searching results of ResNet-50 on CIFAR10 in Figure 2. The searching results are obtained based on well-fitting (test set loss < 2) PCs. The result shows that the last convolutional block (Conv Block 4) tends to choose predecessors in the first half of training (epoch 50 to 180) while Conv Block 2 and Conv Block 3 prefer predecessors in the second half (epoch 150 to 350). This observation might indicate that the later Conv Blocks tend to overfit earlier than the former ones. The first Conv Block and the final linear block show no particular preference to certain predecessors.

This result is consistent with the evaluation of overfitting of individual blocks in Figure 1. The Conv Block 1 suffers from little overfitting throughout training and thus has no preference to certain predecessors, while the middle blocks (Conv Block 2, 3, 4) prefer predecessors with a low validation loss as shown in Figure 1. We visualize the searching results of other models and observe the similar pattern. We also test our algorithm on other networks such as ViT [Dosovitskiy et al., 2020] and MLP-mixer [Tolstikhin et al., 2021], which show the similar calibration effect.

5.5 Ablation Study

Comparison with Other Searching Methods In Table 3, our method is compared with different searching methods as well as early stopping methods. When early stopping on the model as a whole, it is hard to ensure a low error and good calibration performance at the same time. Early stopping on loss and ECE shows a large performance drop on classification error. The random search is conducted 5 times to make the results stable and achieve a relatively high performance on classification error but not ECE. We also compare multiple objectives in Eq. (9) with the performance of a single objective optimization on validation NLL loss, i.e.,minA ˆ NLL. Searching on ECE or error individually could lead to extremely unbalanced results. PCS achieves a better result on both test set error and ECE. The hyperparameter λ is tuned on models and datasets.

Weight Sampling Strategy To compare sampling strategies discussed in the previous section, a scatter plot in Figure 3 shows the searching results. Note that the "Full Sampling" indicates searching on all possible predecessors without sampling, i.e., K = Ttrain. We use the same K = 50 for all sampling strategies. From Figure 3, we observe little difference between different sampling strategies, which indicates that we save storage space with little loss of performance. Thus, we use the random sampling strategy throughout the paper due to its simplicity.

Warm-up Population A larger population indicates more searching time. To find a balance between searching time and performance, we compare the searching results with different population sizes. We use the number of well-fitting results (test loss under 0.2) to measure the searching performance.

All experimental results are averaged over 5 runs. According to Table 4, the larger the population size, the more well-fitting results can be found since estimator ψ can have better prior knowledge on error and ECE landscape. However, we use a smaller population size as long as the searching provides satisfactory results.

6 Conclusion

In this paper, we address a common problem, the miscalibration in modern neural networks. We observe that different blocks in a network have different overfitting patterns. Our proposed predecessor combination search, as a regularization method, is very effective for calibrating models and can also be potentially applied to other tasks such as learning with noisy labels and improving model robustness.

Acknowledgments

This work was supported in part by the Australian Research Council under Project DP210101859 and the University of Sydney Research Accelerator (SOAR) Prize. The authors acknowledge the use of the National Computational Infrastructure (NCI) which is supported by the Australian Government, and accessed through the NCI Adapter Scheme and Sydney Informatics Hub HPC Allocation Scheme.