Evaluating and Aggregating Feature-based Model Explanations

1 Introduction

There has been great interest in understanding blackbox machine learning models via post-hoc explanations. Much of this work has focused on feature-level importance scores for how much a given input feature contributes to a model's output. These techniques are popular amongst machine learning scientists who want to sanity check a model before deploying it in the real world [Bhatt et al., 2020]. Many feature-based explanation functions are gradient-based techniques that analyze the gradient flow through a model to determine salient input features [Shrikumar et al., 2017; Sundararajan et al., 2017]. Other explanation functions perturb input values to a reference output and measure the change in the model's output [ ˇStrumbelj and Kononenko, 2014; Lundberg and Lee, 2017].

With many candidate explanation functions, machine learning practitioners find it difficult to pick which explanation function best captures how a model reaches a specific output for a given input. Though there has been work in qualitatively evaluating feature-based explanation functions on human subjects [Lage et al., 2019], there has been little exploration into formalizing quantitative techniques for evaluating model explanations. Recent work has created auxiliary tasks to test if attribution is assigned to relevant inputs [Yang and Kim, 2019] and has developed tools to verify if the features important to an explanation function are relevant to the model itself [Camburu et al., 2019].

∗Contact Author

Borrowing from the humanities, we motivate three criteria for assessing a feature-based explanation: sensitivity, faithfulness, and complexity. Philosophy of science research has advocated for explanations that vary proportionally with changes in the system being explained [Lipton, 2003]; as such, explanation functions should be insensitive to perturbations in the model inputs, especially if the model output does not change. Capturing relevancy faithfully is helpful in an explanation [Ruben, 2015]. Since humans cannot process a lot of information at once, some have argued for minimal model explanations that contain only relevant and representative features [Batterman and Rice, 2014]; therefore, an explanations should not be complex (i.e., use few features).

In this paper, we first define these three distinct criteria: low sensitivity, high faithfulness, and low complexity. With many explanation function choices, we then propose methods for learning an aggregate explanation function that combines explanation functions. If we want to find the simplest explanation from a set of explanations, then we can aggregate explanations to minimize the complexity of the resulting explanation. If we want to learn a smoother explanation function that varies slowly as inputs are perturbed, we can leverage an aggregation scheme that learns a less sensitive explanation function. To the best of our knowledge, we are the first to rigorously explore aggregation of various explanations, while placing explanation evaluation on an objective footing. To that end, we highlight the contributions of this paper:

• We describe three desirable criteria for feature-based explanation functions: low sensitivity, high faithfulness, and low complexity.
• We develop an aggregation framework for combining explanation functions.
• We create two techniques that reduce explanation complexity by aggregating explanation functions.
• We derive an approximation for Shapley-value explanations by aggregating explanations from a point's nearest neighbors, minimizing explanation sensitivity and resembling how humans reason in medical settings.

2 Preliminaries

Restricting to supervised classification settings, let f be a black box predictor that maps an input x ∈ Rd to an output f(x) ∈ Y. An explanation function g from a family of explanation functions, G, takes in a predictor f and a point of interest x and returns importance scores g(f, x) = φx ∈ Rd for all features, where g(f, x)i = φx,i (simplified to φi in context) is the importance of (or attribution for) feature xi of x. By gj, we refer to a particular explanation function, usually from a set of explanation functions Gm = {g1, g2, . . . , gm}.

We denote D : Rd × Rd ↦→ R≥0 to be a distance metric over explanations, while ρ : Rd × Rd ↦→ R≥0 denotes a distance metric over the inputs. An evaluation criterion µ takes in a predictor f, explanation function g, and input x, and outputs a scalar: µ(f, g; x). D = {(xi , yi )}ni=1 refers to a dataset of input-output pairs, and Dx denotes all xi in D.

3 Evaluating Explanations

With the number of techniques to develop feature level explanations growing in the explainability literature, picking which explanation function g to use can be difficult. In order to study the aggregation of explanation functions, we define three desiderata of an explanation function g.

3.1 Desideratum: Low Sensitivity

We want to ensure that, if inputs are near each other and their model outputs are similar, then their explanations should be close to each other. Assuming f is differentiable, we desire an explanation function g to have low sensitivity in the region around a point of interest x, implying local smoothness of g. While [Melis and Jaakkola, 2018] codified the property, [Ghorbani et al., 2019] empirically tested explanation function sensitivity. We follow the convention of the former and define max sensitivity and average sensitivity in the neighborhood of a point of interest x.

Definition 1 (Max Sensitivity). Given a predictor f, an explanation function g, distance metrics D and ρ, a radius r, and a point x, we define the max sensitivity of g at x as:

Definition 2 (Average Sensitivity). Given a predictor f, an explanation function g, distance metrics D and ρ, a radius r, a distribution Px(·) over the inputs centered at point x, we define the average sensitivity of g at x as:

3.2 Desideratum: High Faithfulness

Faithfulness has been defined in [Yeh et al., 2019]. The feature importance scores from g should correspond to the important features of x for f; as such, when we set particular features xs to a baseline value ¯xs, the change in predictor's output should be proportional to the sum of attribution scores of features in xs. We measure this as the correlation between the sum of the attributions of xs and the difference in output when setting those features to a reference baseline. For a subset of indices S ⊆ {1, 2, . . . d}, xs = {xi, i ∈ S} denotes a sub-vector of input features that partitions the input, x = xs ∪ xc. x[xs=¯xs] denotes an input where xs is set to a reference baseline while xc remains unchanged: x[xs=¯xs] = ¯xs ∪ xc. When |S| = d, x[xs=¯xs] = ¯x.

Remark (Reference Baselines). Recent work has discussed how to pick a proper reference baseline ¯x. [Sundararajan et al., 2017] suggests using a baseline where f(¯x) ≈ 0, while others have proposed taking the baseline to be the mean of the training data. [Chang et al., 2019] notes that the baseline can be learned using generative modeling.

Definition 3 (Faithfulness). Given a predictor f, an explanation function g, a point x, and a subset size |S|, we define the faithfulness of g to f at x as:

For our experiments, we fix |S| then randomly sample subsets xs of the fixed size from x to estimate correlation. Since we do not see all ( [d] |S|) subsets in our calculation of faithfulness, we may not get an accurate estimate of the criterion. Though hard to codify and even harder to aggregate, faithfulness is desirable, as it demonstrates that an explanation captures which features the predictor uses to generate an output for a given input. Learning global feature importances that highlight, in expectation, which features a predictor relies on is a challenging problem left to future work.

3.3 Desideratum: LowComplexity

A complex explanation is one that uses all d features in its explanation of which features of x are important to f. Though this explanation may be faithful to the model (as defined above), it may be too difficult for the user to understand (especially if d is large). We define a fractional contribution distribution, where | · | denotes absolute value:

Note that Pg is a valid probability distribution. Let Pg(i) denote the fractional contribution of feature xi to the total magnitude of the attribution. If every feature had equal attribution, the explanation would be complex (even if it is faithful). The simplest explanation would be concentrated on one feature. We define complexity as the entropy of Pg.

Definition 4 (Complexity). Given a predictor f, explanation function g, and a point x, the complexity of g at x is:

4 Aggregating Explanations

Given a trained predictor f, a set of explanation functions Gm = {g1, . . . , gm}, a criterion to optimize µ, and a set of inputs Dx, we want to find an aggregate explanation function gagg that satisfies µ at least as well as any gi ∈ Gm. Let h(·) represent some function that combines m explanations into a consensus gagg = h(Gm). We now explore different candidates for h(·).

4.1 Convex Combination

Suppose we have two different explanation functions g1 and g2 and have chosen a criterion µ to evaluate a g. Consider an aggregate explanation, gagg = h(g1, g2). A potential h(·) is a convex combination where gagg = h(g1, g2) = wg1 +(1w)g2 = w⊺Gm.

Proposition 1. If D is the ℓ2 distance and µ = µA (average sensitivity), the following holds:

Proof. Assuming Px(z) is uniform, we can apply the triangle inequality and the convexity of D to arrive at the above.

A convex combination of explanation functions thus yields an aggregate explanation function that is at most as sensitive as any of the explanation functions taken alone. In order to learn w given g1 and g2, we set up an objective as follows.

Assuming a uniform distribution around all x ∈ Dx, we can rewrite this as:

By Cauchy-Schwartz, we get the following:

where a = wg1(f, x) + (1 - w)g2(f, x) and b = wg1(f, z) + (1 - w)g2(f, z). This implies that w∗ will be minimal when one element of w∗ is 0 and the other is 1. Therefore, a convex combination of two explanation functions, found by solving Equation (1), will be at most as sensitive as the least sensitive explanation function.

4.2 Centroid Aggregation

Another sensible candidate for h(·) to combine m explanation functions is based on centroids with respect to some distance function D : G × G ↦→ R, so that:

where p is a positive constant. The simplest examples of distances are the ℓ2 and ℓ1 distances with real-valued attributions where G ⊆ Rd .

Proposition 2. When D is the ℓ2 distance and p = 2, the aggregate explanation is the feature-wise sample mean.

Proposition 3. When D is the ℓ1 distance and p = 1, the aggregate explanation is the feature-wise sample median.

Propositions 2 and 3 follow from standard results in statistics that the mean minimizes the sum of squared differences and the median minimizes the sum of absolute deviations [Berger, 2013].

We could obtain rank-valued attributions by taking any quantitative vector-valued attributions and ranking features according to their values. If D is the Kendall-tau distance with rank-valued attributions where G ⊆ Sd (the set of permutations over d features), then the resulting aggregation mechanism via computing the centroid is called the KemenyYoung rule. For rank-valued attributions, any aggregation mechanism falls under the rank aggregation problem in social choice theory for which many practical "voting rules" exist [Bhatt et al., 2019a].

We analyze the error of a candidate gagg. Suppose the optimal explanation for x using f is g∗ (f, x) and suppose gagg is the mean explanation for x in Equation (2). Let ǫi,x = ||g∗ (f, x) - gi(f, x)|| be the error between the optimal explanation and the ith explanation function.

Proposition 4. The error between the aggregate explanation gagg(f, x) and the optimal explanation g* (f, x) satisfies:

Proof. For a fixed x, we have:

Averaging across Dx, we obtain the result.

Hence, by aggregating, we do better than when using one explanation function alone. Many gradient-based explanation functions fit to noise [Hooker et al., 2019]. One way to reduce noise would be to aggregate by ensembling or averaging. As proven in Proposition 4, the typical error of the aggregate is less than the expected error of each function alone.

5 Lowering Complexity Via Aggregation

In this section, we describe iterative algorithms for aggregating explanation functions to obtain gagg(f, x) with lower complexity whilst combining m candidate explanation functions Gm = {g1, . . . , gm}. We desire a gagg(f, x) that contains information from all candidate explanations gi(f, x) yet has entropy less than or equal to that of each explanation gi(f, x). As discussed, a reasonable candidate for an aggregate explanation function is the sample mean given by Equation (2). We may want gagg(f, x) to approach the sample mean, gavg(f, x); however, the sample mean may have greater complexity than that of each gi(f, x).

For example, let g1(f, x) = [-1, 0]T and g2(f, x) = [0, 1]T . The sample mean is gavg(f , x) = [-0.5, 0.5]T . Both g1 and g2 have the minimum possible complexity of 0, while gavg has the maximum possible complexity, log(2). Our aggregation technique must ensure that gagg(f, x) approaches gavg(f , x) while guaranteeing gagg(f , x) has complexity less than or equal to that of each gi(f, x). We now present two approaches for learning a lower complexity explanation, visually represented in Figure 1.

5.1 Gradient-Descent Style Method

Our first approach is similar to gradient descent. Starting from each gi(f, x), we iteratively move towards gavg(f, x) in each of the d directions (i.e., changing the kth feature by a small amount) if the complexity decreases with that move. We stop moving when the complexity no longer decreases or gavg(f, x) is reached. Simultaneously, we start from gavg(f, x) and iteratively move towards each gi(f, x) in each of the d directions if the complexity decreases. We stop moving when the complexity no longer decreases or any of the gi(f, x) are reached. The final gagg(f, x) is the location that has the smallest complexity from these 2d different walks. Since we only move if the complexity decreases and start from each gi(f, x), the entropy of gagg(f, x) is guaranteed to be less than or equal to the entropy of all gi(f, x).

5.2 Region ShrinkingMethod

In our second approach, we consider the closed region, R, which is the convex hull of all the explanation functions, gi(f, x). Notice region R initially contains gavg. We consider an iterative approach to find the global minimum in the region R. As before, we consider the convex combination formed by two explanation functions, gi and gj. Using convex optimization, we find the value on the line segment between gi and gj that has the minimum complexity; essentially, we iteratively shrink the region. For the region shrinking method, the convex combination formed by gi and gj is:

For every pair of functions in Gm, we find the functions that produces the minimum complexity in the convex combination of the functions, producing a new set of candidates G ′m. gagg is the element in set G ′m with minimal complexity after K iterations. In each iteration, a function is chosen if it has the minimum complexity of all the functions in a convex combination. Thus, the minimum complexity of the set G ′m decreases or remains constant with each iteration.

6 Lowering Sensitivity Via Aggregation

To construct an aggregate explanation function g that minimizes sensitivity, we would need to ensure that a test point's explanation is a function of the explanations of its nearest neighbors under ρ. This is a natural analog for how humans reason: we use past similar events (training data) and facts about the present (individual features) to make decisions [Bhatt et al., 2019b]. We now contribute a new explanation function gAVA that combines the Shapley value explanations of a test point's nearest neighbors to explain the test point.

6.1 Shapley Value Review

Borrowing from game theory, Shapley values denote the marginal contributions of a player to the payoff of a coalitional game. Let T be the number of players and let v : 2T → R be the characteristic function, where v(S) denotes the worth (contribution) of the players in S ⊆ T . The Shapley value of player i's contribution (averaging player i's marginal contributions to all possible subsets S) is:

Let Φ ∈ RT be a Shapley value contribution vector for all players in the game, where φi(v) is the ith element of Φ.

6.2 Shapley Values as Explanations

In the feature importance literature, we formulate a similar problem to where the game's payoff is the predictor's output y = f(x), the players are the d features of x, and the φi values represent the contribution of xi to the game f(x). Let the characteristic function be the importance score of a subset of features xs, where EY [·|x] is an expectation over Pf (·|x):

This characteristic function denotes the negative of the expected number of bits required to encode the predictor's output based on the features in a subset S [Chen et al., 2019]. Shapley value contributions can be approximated via Monte Carlo sampling [ˇStrumbelj and Kononenko, 2014] or via weighted least squares [Lundberg and Lee, 2017].

6.3 Aggregate Valuation of Antecedents

We now explore how to explain a test point in terms of the Shapley value explanations of its neighbors. Termed Aggregate Valuation of Antecedents (AVA), we derive an explanation function that explains a data point in terms of the explanations of its neighbors. We do the following: suppose we want to find an explanation function gAVA(f , xtest) for a point of interest xtest. First we find the k nearest neighbors of xtest under ρ denoted by Nk(xtest, D).

We define gAVA(f, xtest) = Φxtest as the explanation function where:

In essence, we weight each neighbor's Shapley value contribution by the inverse distance from the neighbor to the test point. AVA is closely related to bootstrap aggregation from classical statistics, as we take an average of model outputs to improve explanation function stability.

Theorem 5. gAVA(f, xtest) is a Shapley value explanation.

Proof. We want to show that gAVA(f, xtest) = Φxtest is indeed a vector of Shapley values. Let gSHAP(f, z) = Φz be the vector of Shapley value contributions for a point z ∈ Nk. By [Lundberg and Lee, 2017], we know gSHAP(f, z)i = φi(vz) is a unique Shapley value for the characteristic function vz. By linearity of Shapley values [Shapley, 1953], we know that:

This means that the Φz1 + Φz2 will yield a unique Shapley value contribution vector for the characteristic function vz1 + vz2 . By linearity (or additivity), we know for any scalar α:

This means αΦz will yield a unique Shapley value contribution vector for the characteristic function αvz. Now define:

We can conclude that Φxtest is a vector of Shapley values

While [Sundararajan et al., 2017] takes a path integral from a fixed reference baseline ¯x and [Lundberg and Lee, 2017] only considers attribution along the straight line path between ¯x and xtest, AVA takes a weighted average of attributions along paths from training points in Nk to xtest. AVA can similarly be thought of as a convex combination of explanation functions where the explanation functions are the explanations of the neighbors of xtest and the weights are ρ(xtest, z)-1 . Though the weights are guaranteed to be non-negative, we normalize the weights to sum to 1 and edit the AVA formulation to be: gAVA(f, xtest) = ρtotΦxtest where ρtot = ∑z∈Nk(xtest) ρ(xtest, z)-1 . Notice this formulation is a specific convex combination as described before; therefore, AVA will result in a lower sensitivity than gSHAP(f, x) alone.

6.4 Medical Connection

Similar to how a model uses input features to reach an output, medical professionals learn how to proactively search for risk predictors in a patient. Medical professionals not only use patient attributes (e.g., vital signs, personal information) to make a diagnosis but also leverage experiences with past patients; for example, if a doctor treated a rare disease over a decade ago, then that experience can be crucial when attributes alone are uninformative about how to diagnose [Goold and Lipkin Jr, 1999]. This is the analogous to "close" training points affecting a predictor's output. AVA combines the attributions of past training points (past patients) to explain an unseen test point (current patient). When using the MIMIC dataset [Johnson et al., 2016], AVA models the aforementioned intuition.

7 Experiments

We now report some empirical results. We evaluate models trained on the following datasets: Adult, Iris [Dua and Graff, 2017], MIMIC [Johnson et al., 2016], and MNIST [LeCun et al., 1998]. We use the following explanation functions: SHAP [Lundberg and Lee, 2017], Shapley Sampling (SS) [ˇStrumbelj and Kononenko, 2014], Gradient Saliency (Grad) [Baehrens et al., 2010], Grad*Input (G*I) [Shrikumar et al., 2017], Integrated Gradients (IG) [Sundararajan et al., 2017], and DeepLift (DL) [Shrikumar et al., 2017].

For all tabular datasets, we train a multilayer perceptron (MLP) with leaky-ReLU activation using the ADAM optimizer. For Iris [Dua and Graff, 2017], we train our model to 96% test accuracy. For Adult [Dua and Graff, 2017], our model has 82% test accuracy. As motivated in Section 6.4, we use MIMIC (Medical Information Mart for Intensive Care III) [Johnson et al., 2016]. We extract seventeen real-valued features deemed critical, per [Purushotham et al., 2018], for sepsis prediction. Our model gets 91% test accuracy on the task. For MNIST [LeCun et al., 1998], our model is a convolutional neural network and has 90% test accuracy.

For experiments with a baseline ¯x, zero baseline implies that we set features to 0 and average baseline uses the average feature value in D. Before doing aggregation, we unit norm all explanations. For the complexity criterion, we take the positive ℓ1 norm. We set D = ℓ2 and ρ = ℓ∞.

7.1 Faithfulness µF

In Table 2, we report results for faithfulness for various explanation functions. When evaluating, we take the average of multiple runs where, in each run, we see at least 50 datapoints; for each datapoint, we randomly select |S| features and replace them with baseline values. We then calculate the Pearson's correlation coefficient between the predicted logits of each modified test point and the average explanation attribution for only the subset of features. We notice that, as subset size increases, faithfulness increases until the subset is large enough to contain all informative features. We find that Shapley values, approximated with weighted least squares, are the most faithful explanation function for smaller datasets.

7.2 Max and Avg Sensitivity µM and µA

In Table 3, we report the max and average sensitivities for various explanation functions. To evaluate the sensitivity criterion, we sample a set of test points from D and an additional larger set of training points. We then find the training points that fall within a radius r neighborhood of each test point and find the distance between each nearby training point explanation and the test point explanation to get a mean and max. We average over ten random runs of this procedure. Sensitivity is highly dependent on the dimensionality d and on the radius r. We find that as sensitivity decreases as r increases. Empirically, for MIMIC, Shapley values approximated by weighted least squares (SHAP) are the least sensitive.

7.3 MNIST Complexity µC

In Table 1, we provide a qualitative example for the gradient descent-style and region-shrinking methods for lowering complexity of explanations from a model trained on MNIST. We show an example with images since it illustrates the notion of lower complexity well; however, other data types (tabular) might be better suited for complexity optimization.

7.4 AVA

Our empirical findings support use of an AVA explanation if low sensitivity is desired. [Ghorbani et al., 2019] note that perturbation-based explanations (like gSHAP) are less sensitive than their gradient-based counterparts. In Table 4, we show that AVA explanations not only have lower sensitivities in all experiments but also have less complex explanations (depending on the radius r and number of features d). After finding the average distance between pairs of points, we use r = 1 for Adult, r = 0.3 for Iris, and r = 10 for MIMIC.

8 Conclusion

Borrowing from earlier work in social science and the philosophy of science, we codify low sensitivity, high faithfulness, and low complexity as three desirable properties of explanation functions. We define these three properties for featurebased explanation functions, develop an aggregation scheme for learning combinations of various explanation functions, and devise schemes to learn explanations with lower complexity (iterative approaches) and lower sensitivity (AVA). We hope that this work will provide practitioners with a principled way to evaluate feature-based explanations and to learn an explanation which aggregates and optimizes for criteria desired by end users. Though we consider one criterion at a time, future work could further axiomatize our criteria, explore the interaction between different evaluation criteria, and devise a multi-objective optimization approach to finding a desirable explanation; for example, can we develop a procedure for learning a less sensitive and less complex explanation function simultaneously?

Acknowledgements

We thank reviewers for their feedback. We thank Pradeep Ravikumar, John Shi, Brian Davis, Kathleen Ruan, Javier Antoran, and James Allingham for their comments. UB acknowledges support from DeepMind and the Leverhulme Trust via the CFI. AW acknowledges support from the David MacKay Newton Research Fellowship at Darwin College, The Alan Turing Institute under EPSRC grant EP/N510129/1 & TU/B/000074, and the Leverhulme Trust via the CFI.