Improving Prediction Certainty Estimation for Reliable Early Exiting via Null Space Projection

1 Introduction

Recently, the increasing scale of pre-trained language models (PLMs) has hindered their deployment on resourceconstrained devices and latency-sensitive applications due to the high computational costs and long inference time. To address this issue, early exiting [Xin et al., 2020; Zhou et al., 2020; Liao et al., 2021; Xin et al., 2021; Balagansky and Gavrilov, 2022; Zeng et al., 2024; He et al., 2025a; He et al., 2025b], a kind of adaptive inference strategy, has been proposed to accelerate the inference of PLMs. Specifically, each intermediate layer of PLMs is paired with an internal classifier to provide an early prediction. Once the certainty level of an early prediction reaches the threshold, the forward inference is terminated, bypassing the computation of the following deeper layers. By conducting a sample-wise inference procedure, PLMs can deal with easy samples with shallow layers and handle hard samples with deep layers. This effectively improves the inference efficiency of PLMs without sacrificing accuracy.

Typically, an early exiting method involves devising an exiting signal to estimate the certainty level of early predictions, determining whether samples exit from early layers. Researchers have developed quite a few exiting signals by measuring and seeking the characteristics of predictions with high certainty levels, including the entropy [Xin et al., 2020; He et al., 2024], the softmax score [Schwartz et al., 2020], the patience [Zhang et al., 2023; Zhu et al., 2023], the energy score [Akbari et al., 2022], and the patience-confidence score [Zhang et al., 2022]. These methods employ heuristic exiting strategies, enabling the direct computation of exiting signals without involving any learning process and accordingly facilitating easy deployment for inference acceleration.

Notably, existing methods primarily use logits to devise their exiting signals 1 . Unfortunately, these methods exhibit a high Premature Exiting Rate, which suggests a tendency to overestimate prediction certainty based on logits. It significantly impairs task performance and hinders the model's acceleration. This is attributed to the fact that logits contain only class-relevant information in the features while ignoring the detrimental influence of class-irrelevant information on prediction certainty. Theoretically, any feature x can be orthogonally decomposed into x = xW + xW ⊥ (per Figure 1), where W is the column space of the classifier's weight matrix W, and W ⊥ is the null space of W. We can yield that logits (i.e. W T x) extract class-relevant information (i.e. the feature similarity to each class) from the component xW but fail to capture class-irrelevant information from xW ⊥ due toW T xW ⊥ = 0. Notably, xW ⊥ is closely related to prediction certainty since a large proportion of redundant

1 Given that probabilities are calculated from logits, typically through operators like softmax, we collectively label methods based on logits or probabilities as logit-based methods in this paper.

(class-irrelevant) information in the feature interferes with the model's classification and reduces prediction certainty. Nevertheless, logit-based methods completely disregard the detrimental impact of class-irrelevant information on prediction certainty, inevitably resulting in an overestimation of prediction certainty. The observations raise an intriguing question: Can we enhance prediction certainty estimation by integrating both class-relevant and class-irrelevant information for more reliable exiting decisions?

In this paper, we propose a novel early exiting method based on the Certainty-Aware Probability (CAP) score, which complements the logit-based methods by considering the proportion of class-irrelevant information in the features. To this end, we first define the ratio of ∥xW ⊥ ∥ and ∥x∥ as the NSP (null space projection) score, which provides an estimation of prediction certainty by leveraging the proportion of classirrelevant information in the features. A higher NSP score indicates a lower level of prediction certainty. Then, we introduce the scaled NSP score as a new logit corresponding to a constructed virtual UNK (unknown) class. After appending this new logit to the original logits, the CAP score is finally defined as the softmax probability corresponding to the constructed UNK class, which can serve as a proxy for prediction uncertainty. The exiting condition is met once the CAP score falls below the threshold. From the formation of CAP, it is evident that our method integrates the class-relevant original logits (indicating the feature similarity to each class) with the class-irrelevant NSP score (indicating the proportion of classirrelevant information in the features) to enhance prediction certainty estimation, enabling more reliable exiting decisions.

Our contributions are summarized as follows:

• We reveal that current logit-based early exiting methods neglect the detrimental influence of class-irrelevant information in the features on prediction certainty, leading to an overestimation of prediction certainty and premature exiting of samples with incorrect early predictions.
• We define a class-irrelevant NSP score to estimate prediction certainty by considering the proportion of classirrelevant information in the features.
• We propose a novel early exiting method based on the CAP score, which integrates class-relevant logits and the class-irrelevant NSP score to enhance prediction certainty estimation for more reliable exiting decisions.

Extensive experiments on the GLUE benchmark verify that our method outperforms the SOTA ConsistentEE [Zeng et al., 2024] by 28% in model acceleration, with negligible extra computational or storage overhead. Further analysis confirms the generality of our method across various backbones and demonstrates its effectiveness in enhancing prediction certainty estimation and delivering reliable exiting decisions.

2 RelatedWorks

Heuristic Exiting Strategy. DeeBERT [Xin et al., 2020], FastBERT [Liu et al., 2020], Right-Tool [Schwartz et al., 2020], and E-LANG [Akbari et al., 2022] employ entropy, softmax score, and energy score to indicate prediction certainty/uncertainty, respectively. The exiting criterion is met when the certainty level exceeds a threshold. PABEE [Zhou et al., 2020] relies on cross-layer consistency (i.e. patience) to determine exiting, allowing samples to exit once a sufficient number of consecutive classifiers provide the same answer. F-PABEE [Gao et al., 2023], LECO [Zhang et al., 2023], and BADGE [Zhu et al., 2023] enhance the flexibility of PABEE by introducing softer cross-layer comparison strategies. PCEE-BERT [Zhang et al., 2022] proposes a hybrid exiting strategy based on entropy and patience to guarantee reliable and flexible early exiting. These logit-based methods neglect the detrimental influence of class-irrelevant information on prediction certainty, compromising the reliability of exiting decisions. To remedy this, we propose to consider the proportion of class-irrelevant information in the features for more reliable exiting decisions.

Learning-based Exiting Strategy. BERxiT [Xin et al., 2021], PALBERT [Balagansky and Gavrilov, 2022], and ConsistentEE [Zeng et al., 2024] train neural networks to generate exiting signals. HASHEE [Sun et al., 2022] and BE3R [Mangrulkar et al., 2022] train networks to route each sample (or token) to an appropriate exiting layer, requiring no layer-by-layer exiting judgments. These methods incorporate the learning process to formulate their exiting strategies, introducing additional parameters and training costs. They also depend on intricate parameter tuning, and carefully designed network architectures or training objectives. In contrast, our method employs a heuristic exiting strategy where exiting signals can be computed directly, facilitating easy deployment.

Architecture/Loss of Multi-Exit Networks. CascadeBERT [Li et al., 2021] performs early exiting in multiple cascaded complete models to provide comprehensive representations for accurate predictions. It also introduces a difficulty-aware objective to calibrate the model's predictions. GPFEE [Liao et al., 2021] facilitates early predictions by integrating both past and future states. It incorporates imitation loss to train the model to approximate future states based on all past states. LeeBERT [Zhu, 2021] and GAMLBERT [Zhu et al., 2021] introduce cross-layer distillation objectives to encourage mutual learning among classifiers. LECO [Zhang et al., 2023] enhances the network architecture based on neural architecture search. BADGE [Zhu et al., 2023] introduces block-wise bypasses to alleviate optimization conflicts among multiple classifiers. DisentangledEE [Ji et al., 2023] introduces adapters to decouple generic language representations from task-specific representations. It also proposes a non-parametric classifier for improvements. Different from these methods that enhance training objectives or network architectures for early exiting models, our method focuses on improving exiting strategies. Combining our method with these orthogonal works would be an intriguing direction for further research.

3 Background andMotivation

3.1 ProblemDefinition

Given a BERT-style PLM with M layers, we denote the hidden states at the m-th layer as h(m) . To enable early exiting on a classification task with C classes during the inference stage, each intermediate layer is coupled with an internal classifier fm where m ∈ {1, 2, · · · ,M - 1} to provide an early prediction p(m) = fm(h(m) ), i.e., the probability distribution over the C classes. Classifiers in different layers do not share parameters. The exiting criterion is met when the estimated prediction certainty (or uncertainty) of an internal classifier exceeds (or falls below) the threshold τ.

3.2 Are Exiting Decisions Reliable?

We utilize two types of error rates to evaluate the reliability of exiting decisions from different aspects:

• Premature Exiting Rate, which is the frequency of making "exit" decisions when the internal classifier provides an incorrect early prediction.
• Delayed Exiting Rate, which is the frequency of making "continue" decisions when the internal classifier provides the correct early prediction.

The exiting decision-making can be formalized as a binary classification task, where correct early predictions are positive samples and incorrect ones are negative. The Premature Exiting Rate thus corresponds to the false positive rate, indicating the model's failure to identify negative samples (i.e., incorrect early predictions) due to overestimating prediction certainty. A higher rate suggests that the model is prone to prematurely emit samples with incorrect early predictions, ultimately impairing task performance. In contrast, the Delayed Exiting Rate corresponds to the false negative rate, indicating the model's failure to identify positive samples (i.e., correct early predictions) due to underestimating prediction certainty. As its value escalates, the model tends to delay the exiting of samples with correct early predictions, resulting in prolonged inference time. Therefore, exiting decisions are considered reliable only when both the Premature and Delayed Exiting Rates are sufficiently low.

From Figure 3, we observe that the current logit-based early exiting methods exhibit a much higher Premature Exiting Rate compared to the Delayed Exiting Rate. This indicates that these methods primarily suffer from an overestimation of prediction certainty, leading to premature exiting of samples with incorrect early predictions. We believe this is because logits represent the feature similarity to each class, namely they are class-relevant. However, there is classirrelevant information related to prediction certainty in the feature space that is not contained in logits. This affects the estimation of prediction certainty, leading to sub-optimal exiting decisions. These observations trigger our further exploration on the missing information from features to logits.

3.3 The Missing Information in Logits

Given a classification task with C-classes, each internal classifier consists of a fully connected layer with weight W ∈ RN×C and bias b ∈ RC , which transforms the feature x ∈ RN to the logits l ∈ RC , i.e. l = W T x + b, thus providing an early prediction p = softmax(l). To omit the bias term from the calculation of logits, we offset the original feature space by a vector o = (W T )+b, where (·)+ denotes the Moore-Penrose inverse of the matrix:

For convenience, in the rest of this paper, the feature x refers to the offset result. Let W denote the column space of W and W ⊥ denote the null space of W. The feature x can be orthogonally decomposed into x = xW + xW ⊥ , where xW and xW ⊥ are the projections of x to W and W ⊥, respectively. Note that xW ⊥ satisfies W T xW ⊥ = 0, which is class-irrelevant. Therefore, Eq.(1) can be rewritten as:

where each logit li is the inner product of xW and the class vector wi (the i-th column of W), indicating the feature similarity to the i-th class. Obviously, logits l extracts classrelevant information from the component xW using W but neglect the component xW ⊥ that carries class-irrelevant information. Indeed, xW ⊥ is closely related to prediction certainty. Precisely, given a feature x, a larger component xW ⊥ indicates more redundant (class-irrelevant) information in the feature, which can interfere with the model's classification and lead to reduced prediction certainty. Based on the above analysis, the detrimental influence of class-irrelevant information on prediction certainty is completely ignored by logitbased methods, which leads to an overestimation of prediction certainty, causing premature exiting of samples.

To remedy this, we define the ratio of the norms of xW ⊥ and x as the NSP (null space projection) score:

The NSP score can provide an estimation of prediction certainty by considering the proportion of class-irrelevant information in the features. Geometrically, it equals to the cosine similarity between the feature and its projection to the null space. The NSP score lies between 0 and 1, and a higher value indicates a lower certainty level.

Logits reflect the feature similarity to each class, while the NSP score indicates the proportion of class-irrelevant information in the features, both of which are closely related to prediction certainty. We hypothesize that combining classrelevant logits and the class-irrelevant NSP score could enhance prediction certainty estimation, thus enabling more reliable exiting decisions. Such a solution is proposed in Section 4 by introducing the scaled NSP score as a new logit.

4 Methods

We propose a novel early exiting method based on the Certainty-Aware Probability (CAP) score, which integrates the class-irrelevant NSP score with class-relevant logits for reliable exiting decisions. Figure 1 provides an overview of our method. During inference, we first compute the NSP score in Eq.(3), which is the ratio of the norms of the feature x and its projection xW ⊥ onto the null space W ⊥. Then, we introduce the scaled NSP score as a new logit corresponding to a constructed virtual UNK class. Finally, the exiting signal of our method, i.e. the CAP score, is defined as the softmax probability corresponding to the new logit.

4.1 Null Space Projection

This subsection provides the computation of the NSP score. We offset the feature space by o = (W T )+b to omit the bias term in the computation of logits, as shown in Eq.(1). Additionally, for computational efficiency, we compute the component xW , thus obtaining ∥xW ⊥ ∥ = √∥x∥2 - ∥xW ∥2 .

As xW is the projection of x onto the column space of W, it can be expressed as xW = Wµ, where µ ∈ RC denotes the combination coefficients of w1 ∼ wC. Considering that xW ⊥ satisfies W T xW ⊥ = 0, we have:

Thus, we construct a system of linear equations regarding the combination coefficients µ:

Solving Eq.(5), we get µ = (W T W)-1W T x, and consequently obtain:

According to Eq.(3), the NSP score can be computed as:

4.2 The CAP Score

In this subsection, we introduce the Certainty-Aware Probability (CAP) score, an exiting signal that combines the classirrelevant NSP score and class-relevant logits. To this end, we introduce the scaled NSP score as a new logit corresponding to a constructed virtual UNK (unknown) class:

where α is the scaling parameter. Note that the NSP score cannot be directly used as a new logit since the subsequent softmax is highly sensitive to the scale of logits. Hence, we introduce the hyper-parameter α to align the scale of the new logit with each original logit, preventing the final CAP score from being dominated by either side due to scale mismatches.

We append l0 to the original logits l1 ∼ lC and derive an extended probability distribution through softmax:

where i ∈ {0, 1, · · · ,C}. Specifically, l1 ∼ lC suggest the feature similarity to each original class, thus the corresponding p1 ∼ pC indicate the probability of the sample belonging to each original class, respectively. In contrast, the new logit l0 reflects the proportion of class-irrelevant information in the feature. A larger l0 value indicates a higher proportion of class-irrelevant information in the feature, making it more challenging for the model to identify which class the sample belongs to. Hence, the corresponding p0 denotes the probability of the sample belonging to the constructed UNK class, which indicates the model's inability to provide an exact answer and can serve as a proxy for prediction uncertainty.

Based on the above analysis, the CAP score is defined as:

The CAP score lies in (0, 1), which is affected by both classrelevant original logits and the class-irrelevant NSP score. We notice that samples with lower NSP scores (less classirrelevant information) and larger original logits (higher similarity to each class) typically have lower CAP scores, showing higher prediction certainty. The exiting criterion is met when the CAP score falls below a predefined threshold. Using an information fusion strategy, our method improves prediction certainty estimation to deliver more reliable exiting decisions. The computational overhead from the CAP score is negligible compared to the encoder layer (see Section 6).

4.3 More Insights into CAP

Rationale for Logit Concatenation. Intrinsically, both the new and original logits capture feature similarity. Specifically, we treat all original classes as known classes, while the constructed virtual UNK class represents all unknown classes. Since the component xW ⊥ is orthogonal to all known class vectors, it can be naturally interpreted as the class vector for the UNK class. Accordingly, the NSP score captures the cosine similarity between the feature and the UNK class vector (per Figure 1), while the original logits measure the feature's similarity to each known class vector. This semantic alignment justifies concatenating the new and original logits, ensuring that the extended logits and probability distribution carry clear and coherent physical meaning.

Effectiveness of CAP. Current logit-based methods only consider the feature's similarity to known class vectors, disregarding the possibility of the sample belonging to an unknown class. This leads to an overly optimistic estimation of prediction certainty. In contrast, our CAP score considers the feature's similarity to both known and unknown classes, enabling a more objective estimation of prediction certainty.

5 Experiments

5.1 Datasets

We evaluate our method on six classification tasks of the GLUE benchmark [Wang et al., 2019], including SST-2, MRPC, QNLI, RTE, QQP, and MNLI. Data statistics are shown in Table 1.

5.2 Backbones and Baselines

We choose the widely used BERT-base [Devlin et al., 2019] as the backbone model for convincing comparison. We compare our method with two groups of representative and competing baselines. The first group encompasses all existing heuristic exiting strategies, including DeeBERT [Xin et al., 2020], Right-Tool [Schwartz et al., 2020], PABEE [Zhou et al., 2020], PCEE-BERT [Zhang et al., 2022], E-LANG [Akbari et al., 2022], and F-PABEE [Gao et al., 2023]. The second group encompasses competitive early exiting methods relevant to our study, including BERxiT [Xin et al., 2021], PALBERT [Balagansky and Gavrilov, 2022], DisentangledEE [Ji et al., 2023], and ConsistentEE [Zeng et al., 2024]. For fair comparisons, HASHEE [Sun et al., 2022] is not included since it employs token-level early exiting, whereas our method focuses on sentence-level early exiting.

For the first group of baselines, DeeBERT, E-LANG, and Right-Tool employ entropy, energy scores, and softmax scores as exiting signals, respectively. PABEE uses crosslayer consistency (i.e. the patience score) to determine exiting. F-PABEE combines PABEE with softer cross-layer consistency measures to ensure flexible early exiting. PCEEBERT adopts a hybrid exiting strategy based on entropy and patience. For the second group of baselines, BERxiT introduces learning-to-exit networks to generate exiting signals that score the certainty level of early predictions. PALBERT adopts a deterministic Q-exit criterion that evaluates the cumulative distribution function of the exiting layer's probability distribution from neural networks for exit decisionmaking. DisentangledEE incorporates adapters to decouple generic language representations from task-specific language representations and further proposes a non-parametric classifier for improvements. It employs the patience-based exiting strategy in PABEE during inference. ConsistentEE introduces policy networks to predict the probability distribution of exiting decisions.

5.3 Experimental Settings

Speed Measurement. Following previous studies [Zhang et al., 2022; Zeng et al., 2024], we evaluate the model acceleration with saved layers:

where M is the total number of layers and N m is the number of samples exiting from the mth layer.

Training. Our implementation is based on Hugging Face's Transformers [Wolf et al., 2020]. All internal classifiers are jointly trained with the backbone by minimizing the sum of their cross-entropy losses. Following previous studies [Zhou et al., 2020; Zhang et al., 2022], we perform a grid search over learning rates of {1e-5, 2e-5, 3e-5, 5e-5}, and batch sizes of {16, 32, 128}. The maximum sequence length is fixed at 128. We employ a linear decay learning rate scheduler and the AdamW optimizer [Loshchilov and Hutter, 2019]. We conduct experiments on two RTX4090 GPUs with 24GB.

Inference. Following previous studies [Zhang et al., 2022; Gao et al., 2023], the batch size for inference is set to 1, which mimics a common industry scenario where requests from different users arrive sequentially. We select α in Eq.(8) from {0.01, 0.1, 1.0, 10.0} for each task.

5.4 Overall Performance Comparison

In Table 2, we report the test results of each early exiting method on the GLUE benchmark with BERT-base as the backbone model. The speed-up ratio is approximately 2.00×. Overall, our method outperforms all baseline methods in nearly all tasks, which demonstrates the effectiveness of our design. Notably, compared to the backbone, our method achieves an average speed-up ratio of 2.19× across all tasks with negligible performance degradation, surpassing the SOTA baseline ConsistentEE by 28%.

Figure 2 shows the performance-efficiency curves of our method and three competitive logit-based baselines on a representative subset of GLUE using the same fine-tuned multi-exit network. Our method exhibits a superior tradeoff between task performance and inference efficiency across different tasks compared to the baseline methods. This demonstrates that our method effectively completes the current logit-based methods by considering the proportion of class-irrelevant information in the features suggested by NSP scores. Further analysis confirms our method's advantage in estimating prediction certainty and making reliable exiting decisions (see Section 6), offering an intrinsic explanation for its performance gains.

6 In-depth Analysis

DIS Analysis for Exiting Signals. Following previous work [Li et al., 2021], we use the Difficulty Inversion Score (DIS) to evaluate the quality of prediction certainty estimation. DIS measures the consistency between the exiting signal and the sample difficulty. A higher value indicates a stronger capability of exiting signals in estimating prediction certainty. Table 3 shows the DIS of various exiting signals on the SST-2 and QNLI development sets. We observe that by incorporating the class-irrelevant NSP score, the CAP score consistently outperforms the baselines across different layers, demonstrating a more accurate estimation of prediction certainty. This is crucial for making reliable exiting decisions.

Statistics of Exiting Decisions. Figure 3 illustrates the two error rates for exiting decisions using different early exiting methods on the SST-2 and QNLI development sets. Compared to the current logit-based methods, utilizing the NSP score as the exiting signal can effectively reduce the Premature Exiting Rate while exhibiting a relatively high Delayed Exiting Rate. We attribute this to the NSP score's neglect of feature similarity to each class, resulting in an underestimation of prediction certainty and delayed exiting for samples with correct early predictions. In contrast, by fusing information from both class-relevant logits and the class-irrelevant NSP score, our method significantly reduces both types of error rates, demonstrating more reliable exiting decisions. This explains the superiority of our method in model acceleration.

Impact of α. Figure 4 shows the impact of α in Eq.(8) on model acceleration. α is used to align the scale of the new and original logits, and a higher value indicates a stronger emphasis on the class-irrelevant NSP score compared to the class-relevant original logits. Overall, we observe that both excessively large and small values of α can impair the model acceleration under various speed-up ratios. This suggests an optimal trade-off between class-irrelevant and class-relevant information, which enables reliable exiting decisions. Additionally, the incorporation of the NSP score yields more significant performance improvements under high acceleration scenarios. We attribute this to the limited classification capacity of shallow classifiers. This aggravates the overestimation of prediction certainty when relying solely on the class-relevant original logits, exacerbating premature exiting of samples with erroneous early predictions. Hence, incorporating the class-irrelevant NSP score becomes increasingly crucial for reliable exiting decisions. Finally, for parameter selection, values of α between 0.1 and 1.0 typically deliver satisfactory results across different tasks and speed-up ratios.

Computational and Storage Costs. Table 4 presents the computational complexity for each module in our model. It is noteworthy that, at each layer, the offset vector o in Eq.(1) and the projection matrix in Eq.(6) are shared across samples during inference, requiring no repetitive calculations. Consequently, for each sample, the computational overhead introduced by making exiting decisions primarily arises from incorporating internal classifiers and calculating CAP, totaling less than 1.21M FLOPs per layer. This overhead is negligible compared to the 1813.5M of an encoder block, leading to a minimal impact on inference time. The inference time analysis presented in Table 5 further reinforces this conclusion. Additionally, the results in Table 6 indicate that our model only requires less than 0.03% additional parameters compared to the backbone due to incorporating internal classifiers. The analysis above confirms our method's efficiency regarding computation and storage, suggesting strong scalability to larger datasets and backbones. Furthermore, as the model's computational complexity is primarily dominated by encoder blocks, its inference costs are approximately proportional to the number of executed layers, validating the rationale for the speed measurement in Eq.(11).

Generality on Different PLMs. To verify the generality of our information fusion strategy, we apply our method to ALBERT [Lan et al., 2020], which is an optimized version of BERT with reduced parameters and improved efficiency. The results at around 2.00× speed-up ratio are listed in Table 7. Our method still outperforms all competitive baseline methods in general, validating its generality on various PLMs.

7 Conclusion

In this paper, we propose a novel early exiting method based on the CAP score, which integrates insights from both classrelevant logits and the class-irrelevant NSP score to address the overestimation of prediction certainty in current logitbased early exiting methods, enabling more reliable exiting decisions. Our method is simple yet effective. Extensive experiments on the GLUE benchmark validate its superiority across different backbones. Further analysis confirms the interpretability of our method and its efficiency in terms of storage and computation.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 62376198, No. 62406225), the National Key Research and Development Program of China (No. 2022YFB3104700), and the Shanghai Baiyulan Pujiang Project (No. 08002360429).

Contribution Statement

Duoqian Miao serves as the corresponding author and is responsible for all communications related to this manuscript.