Differentiable Model Selection for Ensemble Learning

1 Introduction

Model selection involves the process of identifying the most suitable models from a set of candidates for a given learning task. The chosen model should ideally generalize well to unseen data, with the complexity of the model playing a crucial role in this selection process. However, striking a balance between underfitting and overfitting is a significant challenge.

A variety of techniques have been presented in the machine learning literature to address this issue. Of particular relevance, ensemble learning [Witten et al., 2005] is a meta-algorithm that combines the outputs of individually pretrained models, known as base learners, to improve overall performance. Despite being trained to perform the same task, these base learners may exhibit error diversity, meaning they fail on different samples, and their accuracy profiles complement each other across an overall distribution of test samples. The potential effectiveness of an ensemble model strongly depends on the correlation between the base learners' errors across input samples and their accuracy; those with higher accuracy and error diversity have a higher potential for improved ensemble accuracy [Mienye and Sun, 2022].

However, the task of identifying the optimal aggregation of ensemble model predictions for any particular input sample is nontrivial. Traditional approaches often aggregate predictions across all base learners of an ensemble, aiming to make predictions more robust to the error of individual base learners. While these techniques could be enhanced by selectively applying them to a subset of base learners known to be more reliable on certain inputs, the design of algorithms that effectively select and combine the base learners' individual predictions remains a complex endeavor. Many consensus rule-based methods apply aggregation schemes that combine or exclude base learners' predictions based on static rules, thereby missing an opportunity to inform the ensemble selection based on a particular input's features.

Recently, the concept of differentiable model selection has emerged, aiming to incorporate the model selection process into the training process itself [Dona and Gallinari, 2021; Sheth and Fusi, 2020; Fu et al., 2016]. This approach leverages gradient-based methods to optimize model selection, proving particularly beneficial in areas like neural architecture search. The motivation behind differentiable model selection lies in the potential to automate and optimize the model selection process, thereby leading to superior models and more efficient selection procedures. Despite its promises, however, it remains non-trivial how to design effective differentiable model selection strategies and the use of gradientbased methods alone further enhances the risk of converging to local optima which can lead to suboptimal model selection.

In light of these challenges, this paper proposes a novel framework for differentiable model selection specifically tailored for ensemble learning. The framework integrates machine learning and combinatorial optimization to learn the selection of ensemble members by modeling the selection process as an optimization problem leading to optimal selections within the prescribed context.

Contributions. In more detail, this paper makes the following contributions: (1) It proposes end-to-end Combinatorial Ensemble Learning (e2e-CEL), a novel ensemble learning framework that exploits an integration of ML and combinatorial optimization to learn specialized consensus rules for a particular input sample. (2) It shows how to cast the selection and aggregation of ensemble base learner predictions as a differentiable optimization problem, which is parameterized by a deep neural network and trained end-to-end within the ensemble learning task. (3) An analysis of challenging learning tasks demonstrates the strengths of this idea: e2eCEL outperforms models that attempt to select individual ensemble members, such as the optimal weighted combination of the individual ensemble members' predictions as well as conventional consensus rules, implying a much higher ability to leverage error diversity.

These results demonstrate the integration of constrained optimization and learning to be a key enabler to enhance the effectiveness of model selection in machine learning tasks.

2 RelatedWork

Ensemble learning involves two steps: training individual base learners and combining their outputs for accurate predictions. The composition of an ensemble from base learners with complementary error profiles is commonly done through bagging (randomly partitioning training sets for each member) and boosting (adaptively creating datasets based on error distributions to increase error diversity). A survey of training individual base learners can be found in [Mienye and Sun, 2022]. The second step is typically handled by classical aggregation rules over the predictions or activation values of ensemble members, such as majority or plurality voting. Some works have also attempted to mathematically model more effective aggregation rules, such as the Super Learner algorithm [Ju et al., 2018] which forms a weighted combination of base learner models that maximizes accuracy over a validation set. This algorithm has been proven to be asymptotically optimal for combining ensemble members predictions.

This paper addresses the latter, challenging, aspect of ensemble modeling: optimizing the aggregation of predictions from individual ensemble base learners. The proposed e2eCEL approach aims to learn aggregation rules adaptively at the level of individual input samples, rather than a single rule for all samples. While heuristic-based selection rules to derive input-dependent ensembles are not new to the literature, to the best of our knowledge, this is the first proposal of a method that learns such conditional rules in an end-to-end manner. A discussion on additional work is deferred to [Kotary et al., 2022], Appendix A.

3 Setting and Goals

The paper considers ensembles as a collection of n models or base learners represented by functions fi, 1 ≤ i ≤ n, trained independently on separate (but possibly overlapping) datasets (Xi,Yi), all on the same intended classification task. On every task studied, it assumed that (Xi,Yi) are given, along with a prescription for training each base learners, so that fi are assumed to be pre-configured. This setting is common in federated analytic contexts, where base learners are often trained on diverse datasets with skewed distributions [Kairouz et al., 2021], and in ML services, where providers offer a range of pre-trained models with different architectural and hyperparametrization choices [Ribeiro et al., 2015].

Let n∈N be the number of base learners, c∈N the number of classes and d ∈N the input feature size. Given a sample z∈Rd , each base learner fj : Rd →Rc computes fj(z) = ˆ yj . For the classification tasks considered in this paper, each ˆ yj is the direct output of a softmax function Rc →Rc ,

Explicitly, each classifier fi(φi, x) is trained with respect to its parameters φi to minimize a classification loss L as

The goal is then to combine the base learners into an ensemble, whose aggregated classifier g performs the same task, but with greater overall accuracy on a master dataset (X,Y), where Xi ⊂ X and Yi ⊂ Y for all i with 0 ≤ i ≤ n:

As is typical in ensemble learning, the base learners may be trained in a way that increases test-error diversity among fi on X - see Section 5. In each dataset there is an implied train/test/validation split, so that evaluation of a trained model is always performed on its test portion. Where this distinction is needed, the symbols Xtrain, Xvalid, Xtest are used. A list of symbols used in the paper to describe various aspects of the computation, along with their meanings is provided in [Kotary et al., 2022], Table 4.

4 End-to-end Combinatorial Ensemble Learning

Ideally, given a pretrained ensemble fi, 1 ≤ i ≤ n and a sample z ∈ X, one would select from the ensemble a classifier which is known to produce an accurate class prediction for z. However, a performance assessment for each base learners' predictions is not available at test time. Thus, conventional ensemble learning schemes resort to selection criteria such as plurality voting (see Section 5 for a description of the aggregation rules here used as a benchmark).

The end-to-end learning scheme in this work is based on the idea that a more accurate ensemble prediction can be made by using predictions based on z, and that selecting a well-chosen subset of the ensemble, rather than the entire ensemble, can provide more reliable results than a single base learner. The size of the subset, k, is treated as a hyperparameter. While it may seem logical to only choose the best predicted base learner for a given input sample (setting k = 1), it is consistently observed in Section 5 that the optimal performance is achieved for 1 < k < n.

The proposed mechanism casts the sub-ensemble selection as an optimization program that is end-to-end differentiable and can thus be integrated with a learning model gθ to select a reliable subset of the ensemble base learners to combine for predictions. An end-to-end Smart Selection Ensemble (e2eSSE), or simply, smart ensemble, consists of an ensemble of base learners along with a module that is trained by e2e-CEL to select the sub-ensemble of size k, which produces the most accurate combined prediction for a given input. The model g is called the selection net, and the end-to-end ensemble model is trained by optimizing its parameters θ.

E2e-CEL overview. E2e-CEL is composed of three main steps:

1. Predict a vector of scores gθ(z) = ˆ c, estimating the prediction accuracy for each base learner on sample z.
2. Identify the base learner indices E ⊂ [n] which correspond the the top k predicted scores.

3. Collect the predictions of the selected sub-ensemble fj(z) and perform an approximate majority voting scheme over those predictions to determine the z's class.

By training on the master set Xtrain, the smart ensemble learns to make better predictions by virtue of learning to select better sub-ensembles to vote on its input samples. However, note that subset selection and plurality voting are discrete operations, and in plain form do not offer useful gradients for backpropagation and learning. The next sections discuss further details of the e2e-CEL framework, including differentiable approximations for each step of the overall model.

Figure 1 illustrates the e2e-CEL model and its training process in terms of its component operations. Backpropagation is shown with red arrows, and it only applies to the operations downstream from the selection net g, since the e2e-CEL is parameterized by the parameters of g alone.

4.1 Differentiable Model Selection

The e2e-CEL system is based on learning to select k < n predictions from the master ensemble, given a set of input features. This can be done by way of a structured prediction of binary values, which are then used to mask the individual base learner predictions.

Consider the unweighted knapsack problem

which can be viewed as a selection problem whose optimal solution assigns the value 1 to the elements of b associated to the top k values of ˆ c. Relaxing constraint (4c) to 0 ≤ b ≤ 1 results in an equivalent linear program (LP) with discrete optimal solutions b ∈ {0, 1}n , despite being both convex and composed of continuous functions. This useful property holds for any LP with totally unimodular constraints and integer right-side coefficients [Bazaraa et al., 2008].

This optimization problem can be viewed as a mapping from ˆ c to a binary vector indicating its top k values, and represents thus a natural candidate for selection of the optimal sub-ensemble of size k given the individual base learners' predicted scores, seen as ˆ c. However, the outputs of Problem (4) define a piecewise constant function, K(ˆ c), which does not admit readily informative gradients, posing challenges to differentiability. For integration into the end-to-end learning system, the functionK(ˆ c) must provide informative gradients with respect to ˆ c. In this work, this challenge is overcome by smoothing K(ˆ c) based on perturbing ˆ c with random noise.

As observed by [Berthet et al., 2020], any continuous, convex linear programming problem can be used to define a differentiable perturbed optimizer, which yields approximately the same solutions but is differentiable with respect to its linear objective coefficients. Given a random noise variable Z with probability density p(z) ∝ exp (-v(z)) where v is a twice differentiable function,

is a differentiable perturbed optimizer associated to K. The temperature parameter ϵ > 0 controls the sensitivity of its gradients (or properly, Jacobian matrix), which can itself be represented by the expected value [Abernethy et al., 2016]:

In this work, Z is modeled as a standard Normal random variable. While these expected values are analytically intractable (due to the constrained argmax operator within the knapsack problem K), they can be estimated to arbitrary precision by sampling in Monte Carlo fashion. This procedure is a generalization of the Gumbel Max Trick [Gumbel, 1954].

Note that simulating Equations (5) and (6) requires solving Problem (4) for potentially many values of z. However, although the theory of perturbed optimizers requires the underlying problem to be a linear program, only a blackbox implementation is required to produce K(ˆ c) [Berthet et al., 2020], allowing for an efficient algorithm to be used in place of a (more costly) LP solver. The complexity of evaluating the differentiable perturbed optimizer K(ˆ c) is discussed next.

Theorem 1. The total computation required for solving Problem (4) is O(n log k), where n and k are, respectively, the ensemble and sub-ensembles sizes.

Proof. This result relies on the observation that K(ˆ c) can be computed efficiently by identifying the top k values of ˆ c in O(n log k) time using a divide-and-conquer technique. See, for example, [Cormen et al., 2022].

Generating m such solutions for gradient estimation then requires O(mn log k) operations. Note, however, that these can be performed in parallel across samples, allowing for sufficient noise samples to be generated for computing accurate gradients, especially when GPU computing is available.

For clarity, note also that the function K, as a linear program mapping, has a discrete output space since any linear program takes its optimal solution at a vertex of its feasible region [Bazaraa et al., 2008], which are finite in number. As such, it is a piecewise constant function and is differentiable except on a set of measure zero [Folland, 1999]. However, ∂K/∂ˆ c = 0 everywhere it is defined, so the derivatives lack useful information for gradient descent [Wilder et al., 2019]. While ∂K/∂ˆ c is not the true derivative of K at ˆ c, it supplies useful information about its direction of descent.

In practice, the forward optimization pass is modeled as K(ˆ c), and the backward pass is modeled as ∂K(ˆ c)/∂ˆ c. This allows further downstream operations, and their derivatives, to be evaluated at K(ˆ c) without approximation, which improves training and test performance [Kotary et al., 2021]. These forward and backward passes together are henceforth referred to as the Knapsack Layer. Its explicit backward pass is computed as

where zi ∼ N (0, 1)n are m independent samples each drawn from a standard normal distribution.

4.2 Combining Predictions

Denote as P ∈ Rc×n the matrix whose j th column is the softmax vector ˆ yj of base learner j,

For the purpose of combining the ensemble base learner predictions, K(ˆ c) is treated as a binary masking vector b ∈ {0, 1}n , which selects the subset of base learners for making a prediction. Denote as B ∈ {0, 1}c×n the matrix whose ith column is Bi = -→1 bi; i.e.,

This matrix is used to mask the base learner models' softmax predictions P by element-wise multiplication. Next, define

Doing so allows compute the sum of predictions over the selected sub-ensemble E, but in a way that is automatically differentiable, that is:

The e2e-SSE prediction comes from applying softmax to this sum:

viewing the softmax as a smooth approximation to the argmax function as represented with one-hot binary vectors. This function is interpreted as a smoothed majority voting to determine a class prediction: given one-hot binary class indicators hi, the majority vote is equal to argmax(∑i hi). An illustration of the process is given in Figure 1.

At test time, class predictions are calculated as

Combining predictions in this way allows for an approximated majority voting over a selected sub-ensemble, but in a differentiable way so that selection net parameters θ can be directly trained to produce selections that minimize the classification task loss, as detailed in the next section.

4.3 Learning Selections

The smart ensemble mechanism learns accurate class predictions by learning to select better subensembles to vote on its input samples. In turn, this is done by predicting better coefficients ˆ c which parameterize the Knapsack Layer.

The task of predicting ˆ c based on input z is itself learned by the selection net, a neural network model g so that ˆ c = g(z). Since g acts on the same input samples as each fi, it should be capable of processing inputs from z at least as well as the base learners' models; in Section 5, the selection net in each experiment uses the same CNN architecture as that of the base learner models. Its predicted values are viewed as scores over the ensemble members, rather than over the possible classes. High scores correspond to base learners which are well-qualified to vote on the sample in question.

In practice, the selection net's predictions ˆ c are normalized before input to the mapping K:

This has the effect of standardizing the magnitude of the linear objective term in (4a), and tends to improve training. Since scaling the objective of an optimization problem has no effect on its solution, this is equivalent to standardizing the relative magnitudes of the linear objective and random noise perturbations in Equations (5) and (6), preventing ϵ from being effectively absorbed into the predicted ˆ c.

For training input x, let ˆ yθ(x) represent the associated e2eSSE prediction given the selection net parameters θ. During training, the model minimizes the classification loss between these predictions and the ground-truth labels:

Generally, the loss function L is chosen to be the same as the loss used to train the base learner models, as the base learners are trained to perform the same classification task.

4.4 e2e-CEL AlgorithmDetails

Algorithm 1 summarizes the e2e-CEL procedure for training a selection net. Note that only the parameters of the selection net are optimized in training, and so only its downstream computations are backpropagated. This is done by the standard automatic differentiation employed in machine learning libraries [Paszke et al., 2019], except in the case of the Knapsack Layer, whose gradient transformation is analytically specified by Equation (7).

For clarity, Algorithm 1 is written in terms of operations that apply to a single input sample. In practice, however, minibatch gradient descent is used. Each pass of the training begins evaluating the base learner models (line 4) and sampling standard Normal noise vectors (line 5). The selection net predicts from input features x a vector of base learner scores gθ(x), which defines an unweighted knapsack problem K(gθ(x)) that is solved to produce the binary mask b (line 6).

Masking is applied to the base learner predictions before being summed and softmaxed for a final ensemble prediction ˆ y (line 8). The classification loss L is evaluated with respect to the label y and backpropagated in 3 steps: (1) The gradient ∂L ∂b is computed by automatic differentiation backpropagated to the Knapsack Layer's output (line 9). (2) The chain rule factor ∂b ∂ˆ c is analytically computed by the methodology of Section 4.1 (line 10). (3) The remaining chain rule factor ∂ˆ c ∂θ is computed by automatic differentiation (line 11). Note that as each chain rule factor is computed, it is also applied toward computing ∂L ∂θ (line 12). Finally, a SGD step [Ruder, 2016] or one of its variants ([Diederik and Jimmy, 2014], [Zeiler, 2012]) is applied to update θ (line 13).

The next section evaluates the accuracy of ensemble models trained with this algorithm, on classification tasks using deep neural networks.

5 e2e-CEL Evaluation

The e2e-CEL training is evaluated on several vision classification tasks: digit classification on MNIST dataset [Deng, 2012], age-range estimation on UTKFace dataset [Zhifei Zhang, 2017], image classification on CIFAR10 dataset [Krizhevsky et al., 2009], and emotion detection on FER2013 dataset [Liu et al., 2016].

Being an optimized aggregation rule, e2e-CEL is compared with state-of-the-art Super Learner algorithm [Ju et al., 2018] along with the following widely adopted baseline aggregation rules when paired with a pre-trained ensemble :

• Super Learner: a fully connected neural network that, given the base learners' predictions, learns the optimal weighted combinations specialized for any input sample.

Algorithm 1: Training the Selection Net

• ·Unweighted Average: averages all the base learners' softmax predictions and then compute the index of the corresponding highest label score as the final prediction.
• Plurality Voting: makes a discrete class prediction from each base learner and then returns the most-predicted class.
• Random Selection: randomly selects a size-k sub-ensemble of base learners for making prediction and then applies the unweighted average rule to the selected base learners' soft predictions.

Experimental settings. As described in Section 1 and in [Kotary et al., 2022] Appendix A, ensemble learning schemes are most effective when base learner models are accurate and have high error diversity. In this work, base learners are deliberately trained to have high error diversity with respect to input samples belonging to different classes. This is done by composing for each base learner model fi (1 ≤ i ≤ n) a training set Xi in which a subset of classes is overrepresented, resulting base learners that specialize in identifying those classes. The exact class composition of each dataset Xi depends on the particular classification task and on the base learner's intended specialization.

For each task, each base learner is designed to be specialized for recognizing either one or two particular classes. To this end, the training set of each base learner is partitioned to have a majority of samples belonging to a particular class, while the remaining part of the training dataset is uniformly distributed across all other classes by random sampling. Specifically, to compose the smart ensemble for each task, a single base learner is trained to specialize on each possible class, and on each pair of classes (e.g., digits 1 and 2 in MNIST). When c is the number of classes, the experimental smart ensemble then consists of c+ (c2) total base learners. Training a specialized base learner in this way generally leads to high accuracy over its specialty classes, but low accuracy over all other classes. Therefore in this experimental setup, no single base learner is capable of achieving high overall accuracy on the master test set Xtest. This feature is also recurrent in federated analytic models [Kairouz et al., 2021].

Table 1 shows the average accuracy of individual base learner models on their specialty classes and their nonspecialty classes; reported, respectively as specialized accuracy and complementary accuracy. The reported overall accuracy is measured over the entire master test set Xtest. This sets the stage for demonstrating the ability of e2e-CEL training to compose a classifier that substantially outperforms its base learner models on Xtest by adaptively selecting subensembles based on input features; see Section 5.2.

Note that, in each experiment, the base learner models' architecture design, hyperparameter selection, and training methods have not been chosen to fully optimize classification accuracy, which is not the direct goal of this work. Instead, the base learners have been trained to maximize error diversity, and demonstrate the ability of e2e-CEL to leverage error diversity and compose highly accurate ensemble models from far less accurate base learner models, in a way that is not shared by conventional aggregation rules. Note also that improving base learner model accuracies would, of course, tend to improve the accuracy of the resulting ensemble classifiers. In each case, throughout this section, the e2e-SSE selection net is given the same CNN architecture as the base learner models which form its ensemble.

5.1 Datasets and Settings

For each task, the base learners are trained to specialize in classifying one or two particular classes, which allows the selection program to leverage their error diversity. Additional details about the base learners' models and the dataset split can be found in [Kotary et al., 2022], Appendix B.

Digit classification. MNIST is a dataset of 28x28 pixel greyscale images of handwritten digits. It contains 60000 images for training and 10000 images for testing. The ensemble consists of 55 base learners, 10 of which specialize on one class and (10 2 ) = 45 of which specialize on two classes.

Image classification. CIFAR10 is a 32x32 pixel color images dataset in 10 classes: airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. It contains 6000 images of each class. The ensemble consists of 55 base learners, 10 of which specialize on 1 class and (10 2 ) = 45 of which specialize on 2 classes.

Age estimation. UTKFace is a face images dataset consisting of over 20000 samples and different version of images format. Here 9700 cropped and aligned images are split in 5 classes: baby (up to 5 years old), teenager (from 6 to 19), young (from 20 to 33), adult (from 34 to 56) and senior (more than 56 years old). The classes are not uniformly distributed per number of ages, but each class contains the same number of samples. The goal is to estimate a person's age given the face image. The ensemble consists of 15 base learners, 5 of which specialize on 1 class and (52) = 10 on 2 classes.

Emotion detection. Fer2013 is a dataset of over 30000 48x48 pixel grayscale face images, which are grouped in 7 classes: angry, disgust, fear, happy, neutral, sad, and surprised. The goal is to categorize the emotion shown in the facial expression into one category. The ensemble consists of 21 base learners, 7 of which specialize on 1 class and (72) = 21 of which specialize on 2 classes.

5.2 e2e-CEL Analysis

The e2e-CEL strategy is tested on each experimental task for sub-ensemble size k varying between 1 and n, and compared to the baseline methods described above. Note in each case that accuracy is defined as the percentage of correctly classified samples over the master test set.

Table 2 reports the best accuracy over all the ensemble sizes k of ensembles trained by e2e-CEL along with that of each baseline ensemble model, where each are formed using the same pre-trained base learners. Note how the proposed e2e-CEL scheme outperforms all the baseline methods, in each task, for all but the lowest values of k.

Figure 2 reports the test accuracy found by e2e-CEL along with ensembles based on the Super Learner, weighted average, majority voting, or random selection scheme. We make two key observations: (1) Note from each subplot in Figure 2 that smart ensembles of size k > 1 provide more accurate predictions than baseline models that randomly select subensembles of the same size, a trend that diminishes as k increases and base learner selections have less consequence (the two perform equally when k = n). (2) In every case, the subensemble size which results in optimal performance is strictly between 1 and n. Importantly, this illustrates the motivating intuition of the e2e-CEL ensemble training. Neither the full ensemble (k = n), nor smart selection of a single base learner model (k = 1) can outperform models that use smart selection of a sub-ensemble of any size. A well-selected subensemble has higher potential accuracy than the master ensemble, and is, on average, more reliable than a well-selected single base learner.

Next, Table 3 (left) reports the accuracy of the e2e-SSE model trained on each task, along with the sub-ensemble size that resulted in highest accuracy. In two cases, for the digit classification and the image classification task, the e2e-SSE performs best when the sub-ensemble size is equal to the number of classes. In the remaining tasks, this observation Ensemble size holds approximately. This is intuitive, since the number of base learners specializing on any class is equal to the number of classes, and e2e-CEL is able to increase ensemble accuracy by learning to select these base learners for prediction.

Table 3: Left: Best ensemble size (Best k) and associated e2e-CEL test accuracy attained on each dataset. Right: Average accuracy for the constituent ensemble base learners.

Finally, observe the accuracy of e2e-CEL in Table 3 (left) and the performance of the individual base learners predictors of the ensemble tested on both the labels in which their training was specialized as well as the other labels. Note how e2eSSE predictions outperform their constituent base learners by a wide margin on each task. For example, on UTKFace, the e2e-SSE ensemble reaches an accuracy 40 percentage points higher than its average constituent base learner. This illustrates the ability of e2e-CEL to leverage the error diversity of base learners to form accurate classifiers by composing them based on input features, even when the individual base learner's accuracies are poor.

6 Conclusion

This study addresses a significant issue in model selection and ensemble learning: determining the best models for the classification of distinct input samples. The presented solution is an innovative approach for differentiable model selection, and tailored to ensemble learning, merging machine learning with combinatorial optimization. This framework constructs precise predictions, adaptively selecting sub-ensembles based on input samples.

The study show how to transform the ensemble learning task into a differentiable selection process, trained cohesively within the ensemble learning model. This approach allows the proposed framework to compose accurate classification models even from ensemble base learners with low accuracy, a feature not shared by existing ensemble learning approaches. The results on various tasks demonstrate the versatility and effectiveness of the proposed framework, substantially outperforming state-of-the-art and conventional consensus rules in a variety of settings.

This work demonstrates that the integration of machine learning and combinatorial optimization is a valuable toolset for not only enhancing but also combining machine learning models. This work contributes to the ongoing efforts to improve the efficiency and effectiveness of model selection in machine learning, particularly in the context of ensemble learning, and hopes to motivate new solutions where decision-focused learning may be used to improve the capabilities of machine learning systems.

Ethics Statement

While the proposed e2e-CEL method has the potential to improve the performance of ensemble learning, it is important to consider the ethical implications of its use and to take steps to mitigate any potential negative impacts. One possible concern is the potential to select sub-ensembles in a way that could perpetuate or amplify biases present in the ensemble base learners. This could lead to unfair or discriminatory predictions for certain groups of people.

It is also important to consider the potential benefit of this study. This approach allows for the composition of accurate classification models even from ensemble base learners with low accuracy or strong biases, which is a feature not shared by existing ensemble learning approaches. The proposed solution thus aims to enhance the performance of ensemble learning models and may advance the development of more effective predictors which exhibit fewer biases.

Acknowledgements

This research is partially supported by NSF grant 2232054 and NSF CAREER Award 2143706. Fioretto is also supported by an Amazon Research Award and a Google Research Scholar Award. Its views and conclusions are those of the authors only.

Contribution Statement

James Kotary and Vincenzo Di Vito have equal contribution.