Even If Explanations: Prior Work, Desiderata & Benchmarks for Semi-Factual XAI

1 Introduction

With the emergence of deep learning there has been rising concern about the opacity of Artifical Intelligence (AI) systems and their impact on public and private life [Adadi and Berrada, 2018; Guidotti et al., 2018]. Currently, governments are taking steps to protect people's rights, to regulate the AI industry and ensure that these technologies are not abused (e.g., the EU's GDPR [Goodman and Flaxman, 2017]). Research on eXplainable AI (XAI) tries to address such issues using automated explanations to improve the transparency of black-box models, to audit datasets and ensure fairness, accountability and trustworthiness [Gunning and Aha, 2019; Sokol and Flach, 2019; Birhane et al., 2022].

Recently, significant research effort have been expended on counterfactual explanations for XAI [Byrne, 2019; Miller, 2019; Keane et al., 2021; Karimi et al., 2022]; a recent survey paper reports 350 papers on the topic [Verma et al., 2020]. In this paper, we survey a less-researched special-case of the counterfactual, semi-factual explanations. In this review, we survey the literature on semi-factuals, we define desiderata for this strategy, identify key evaluation metrics and implement baselines to provide a solid base for future work.

Counterfactuals aim to explain algorithmic decisions in a post-hoc fashion, as an after-the-fact justification. So, in XAI, counterfactuals are typically used to explain what changes to the input-features of an AI system will change the outputdecision. For example, when a customer is refused a loan (i.e., the negative-class outcome for Q in Fig.1), the counterfactual might say "if you asked for a loan with a shorter term, it would have been approved" (the red C in Fig.1 that has a positive-class outcome). Technically, these could be called "outcome-counterfactuals" as they capture changes to the world that change the outcome (here, to be consistent with the literature, we will mostly call them "counterfactuals").

Semi-factuals are a special-case of the counterfactual; they differ from outcome-counterfactuals in that they show endusers the feature changes that do not change a decisionoutcome. They are "counterfactual" in that they convey possibilities that "counter" what actually occurred, even though the outcome does not change. So, when the customer is refused the loan, the semi-factual might say "even if you doubled your stated income, you would still be refused the loan" or, indeed, "trebled your stated income" (the blue A and B in Fig.1 with the unchanged outcomes). Indeed, the larger the featuredifferences asserted in the semi-factual, the better (more convincing) the explanation (e.g., B is better than A in Fig.1). Philosophers have argued over whether semi-factuals really differ from outcome-counterfactuals (see [Bennett, 2003; Goodman, 1947]), but they have been shown to differ in their psychological impacts [McCloy and Byrne, 2002].

All the benefits accruing to counterfactuals in XAI also seem to accrue to semi-factuals; namely, that they have many legal [Wachter et al., 2017], psychological [Byrne, 2019] and technical benefits [Keane et al., 2021]. For example, in a medical domain, an intern reviewing X-rays of tumors could be told "even if this tumour was half its current size, it would still require a surgical intervention". Similarly, semi-factuals can convey key aspects of a casual model (e.g., a farmer might be told "even if you doubled your fertiliser use, your yield would not increase" because of soil factors). However, as we shall see, semi-factuals also differ significantly in many respects from counterfactuals (see desiderata, section 4).

Outline of Paper & Contributions. In this paper, we systematically review prior work on semi-factuals (henceforth, SFs) in the Cognitive Sciences and AI, beginning with a discussion of key examples from the early literature in Philosophy and Psychology (see section 2). From this work we define desiderata for SFs (section 3). In section 4, we report the results of a systematic survey before sketching the brief history of semi-factual algorithms for explanation (section 5). We then report a benchmarking study implementing key historical algorithms along with a newly-proposed na¨ıve benchmark (see section 6), before closing with some conclusions (see section 7). As such, the paper makes several novel contributions to this emerging area of XAI, providing:

• A comprehensive survey of the relevant literature.
• A first statement of desiderata for semi-factual XAI.
• A na¨ıve benchmark algorithm, based on the new idea of Most Distant Neighbors (MDNs).
• Novel comparative tests of historical benchmarks, to identify the best for future use.
• A publically-available repository of metrics, data, results for these benchmarks and an annotated bibliography (see https://github.com/ itsaugat/sf survey).

2 Philosophy &Psychology of Semi-Factuals

Semi-factuals have been studied under different guises in Philosophy and Psychology for several decades. In Philosophy, counterfactuals (if only...) and semi-factuals (even if...) are often compared to conditionals (if...then) with a view to analysing their logic, truth conditions and role in causation [Chisholm, 1946; Goodman, 1947; Bennett, 1982; Barker, 1991; Bennett, 2003]. For example, [Bennett, 1982] and [Barker, 1991] argue about how the words "even" and "still" affect the interpretation of examples, such as:

1. Even if the United States had used nuclear weapons in Vietnam, it would still have lost the war.

where the semi-factual asserts that even if the military-force expended by United States significantly increased, the Vietnam War would still have been lost. In AI terms, the semifactual says increasing the feature-value of military-force would not change the outcome. Hence, [Iten, 2002] proposes "scalar" analyses of even and even if ;"Even Neville passed the exam" puts Neville low on an academic-ability scale.

In Psychology, semi-factual research has grown out of studies on counterfactual thinking in human cognition [Kahneman and Tversky, 1982; Byrne, 2007; Byrne, 2011; Handley and Feeney, 2007; Epstude and Roese, 2008]. Byrne [2007] proposed a mental model theory of semi-factuals that has been tested in several psychological studies (see e.g., [McCloy and Byrne, 2002; Parkinson and Byrne, 2017]). McCloy & Byrne's [2002] seminal work explicitly compared people's reasoning using matched scenarios for counterfactuals and semi-factuals, akin to the case of Philip who has an allergic reaction to an ice-cream sundae:

1. If only Philip had not chosen the ice-cream sundae, he wouldn't have had an allergic reaction. (Counterfactual)
2. Even if Philip had chosen the banana split, he would still have had an allergic reaction 1 . (Semi-factual)

McCloy & Byrne found that counterfactuals lead people to judge the antecedent event (i.e., the choice of dessert) to be more causally-related to the outcome, but semi-factuals had the opposite effect, leading people to judge the antecedent event to be less causally-related to the outcome. So, semi-factuals weaken the causal link between the inputs and outcome, convincing people that outcome would have occurred anyway (people also differ in their emotional reactions to these events). In another experiment, they also found that counterfactuals lead people to focus on alternative antecedents that undo the outcome (e.g., "If only Philip had chosen the cheese cake he would not have had a reaction"), whereas semi-factuals lead people to focus on alternative antecedents that do not undo the outcome (e.g., "Even if Philip had chosen the baked-alaska he would still have had a reaction"). Subsequent studies test other psychological aspects of semi-factuals [Parkinson and Byrne, 2017; Moreno-Rios et al., 2008; Santamar´ıa et al., 2005; Espino et al., 2022].

Taken together these psychological findings show that semi-factuals have very different psychological effects than counterfactuals. Unlike counterfactuals, semi-factuals convince people of the status quo, they dissuade them from questioning outcomes [Green, 2008], and weaken the causal link between features and outcomes.

3 Desiderata for Semi-Factuals

Several desiderata are suggested by these analyses of semifactuals. These desiderata cover computational (i.e., "what needs to be computed") and psychological requirements (i.e., the response to be elicited in users) and are defined as follows.

Assume (i) a query instance, Q, that has a vector, x, and an outcome, y, that occurs when x holds and (ii) a semi-factual instance, SF, that has a vector, x′ , and an outcome, y′ , that occurs when x′ holds. SF will be a good explanation of Q if:

1. Q is factually the case and SF counters some of Q's facts but not Q's outcome; so the vectors x and x′ differ, diff(x, x′ ), with no outcome change, y = y′ .

1 Because Philip is allergic to the ice-cream in both desserts.

1. Ideally, SF relies on sparse changes to a key-feature(s), f, of Q, with other features being equal 2 , ideally, one feature change (i.e., diff(x, x′ )=1); though many featuredifferences could be proposed, fewer is assumed to be better for psychological reasons.
2. The key-feature(s) changed should be plausible/mutable/actionable; that is, the SF produced by the change should be within the data-manifold.
3. People should find the SF convincing even though it may seem to be unexpected/surprising/counter-intuitive; for instance, they may expect the key-feature change to change the outcome, where y ̸= y′ .
4. If people accept SF, it will change their perception of the causal role of the key-feature(s), f, in the domain. So, their causal model of the domain will change (e.g., causes may be updated/deleted/refined).
5. For fairness and ethical reasons, the asserted differences between Q and SF, should not be misleading. For instance, (i) the key-feature should not be a proxy variable, (ii) the change asserted should be robust (e.g., resistant to adversarial attacks in that local region of the decision space), (iii) though the change may be unexpected it should not violate the domain's causality, (iii) the change assumes ceteris paribus (i.e., "other things being equal"), verifiably so (i.e., the unchanged-outcome shown should not depend on subtle interactions with other variables).

These desiderata present a high bar for semi-factual explanation methods; indeed, it is unclear whether any current method meets all of them. Furthermore, some of them may require further computational specification (e.g., how keyfeatures are selected) and psychological specification in operational definitions for user studies (e.g., for the notions of plausibility, convincingness and surprise).

4 Systematic Survey: Even if Explanations

A systematic search of the AI, Philosophy and Psychology literatures on semi-factuals was conducted using a bottomup citation-search and top-down keyword-searches (see Table 1). Ten searches were carried out between October 12th , 2022 and December 19th , 2022, consisting of (i) a bottomup search checking GoogleScholar citations to three key papers (i.e., [Cummins and Bridge, 2006; Nugent et al., 2009; Kenny and Keane, 2021], (ii) nine top-down searches using keywords in GoogleScholar (see Table 1). The papers found (N=1,150) were title-and-abstract screened to check whether they were just citing semi-factuals or substantially researching them as a topic. Subsequent selections then identified the core papers of relevance (see here for PRISMA diagram).

4.1 Survey Results

Of the 1,150 original results checked, 92 potentially-relevant papers were selected to be read in depth from which 62 core papers were identified (41 cited here; note, 145 duplicates were removed). As we shall see in the next section on history (section 5), from a low base semi-factual research in AI has expanded considerably in the last two years. Note, many semi-factual papers in Philosophy, Psychology and Linguistics were checked but few are specifically relevant to explanation (e.g., in Philosophy the focus tends to be on the truth conditions of counterfactual statements and the linguistic functions of "even" and "still"). Finally, it should also be said that many excluded papers were from closely-related areas that do not cover semi-factuals per se, but which could provide insights for future work; areas that include research on (i) case difference learning (e.g., [Hanney and Keane, 1996; Ye et al., 2021]), (ii) feature selection using near misses (e.g., [Kira et al., 1992; Herchenbach et al., 2022]), (iii) counterfactual explanation (e.g., [Keane et al., 2021; Verma et al., 2020]), (iv) flip-points in learning (e.g., [Yousefzadeh and O'Leary, 2019]), (v) dynamic critiquing in recommenders (e.g., [Reilly et al., 2004]) and computational argumentation (e.g., [ ˇCyras et al., 2021]. These papers are recorded in a publically-available annotated biblography (see https: //github.com/ itsaugat/sf survey).

2 Equal may not mean the features have identical values, they may just be within some threshold difference.

5 A Brief History of Semi-Factual XAI

In AI, semi-factuals have only been considered recently, relative to the philosophical and psychological literatures. Much of the initial work emerged from Case-Based Reasoning (CBR) research on post-hoc, example-based explanations [Sørmo et al., 2005; Keane and Kenny, 2019]. In this AI research, semi-factual explanations have been variously cast as a fortiori arguments [Nugent et al., 2005; Nugent et al., 2009] and precedent-based explanations [Cummins and Bridge, 2006; Bridge and Cummins, 2005]. More recently, Kenny & Keane [2021] re-connected this work to the older literatures by calling them "semi-factuals". Arguably, there are four distinct phases in the development of semi-factual explanation in AI: (i) initial utility-based proposals, (ii) proximity-based methods, (iii) local-region methods and (iv) the more recent "modern-era" of counterfactuallyinspired proposals. In the following sub-sections, we describe each in turn and the intuitions behind them. We end this section by defining a new benchmark-method based on the notion of Most Distant Neighbors (MDNs).

5.1 Semi-Factuals Based on Feature-Utility

Doyle et al. [2004] appear as the first AI paper in our searches to propose using semi-factuals to explain automated decisions, under the rubric of a fortiori reasoning. An a fortiori argument is defined as one that uses a stronger version of an already-convincing proposition (i.e., "EU countries cannot afford standing armies, sure even the US can hardly afford its standing army"). Doyle et al. [2004] noted that nearestneighbor, example-based explanations can often be less convincing than neighbors that have more extreme feature-values within the same class. For example, if patient-x with a moderate temperature is judged to be dischargeable then a semifactual past case, patient-y with a much higher temperature who was discharged is more convincing than pointing to another patient with the same moderate temperature being discharged [Doyle et al., 2006]. So, this semi-factual method computes a set of nearest neighbours as explanatory cases and then re-ranks them using utility functions on selected features to find a more convincing a fortiori case, as follows:

where q is the query, x is an instance, c is a class label and ξ() measures the contribution to explanation utility of the feature f. The ξ() function uses relative-differences in feature-values to assign utilities. For example, for the temperature feature, the measure might assign higher utility to a 10◦C difference than to a 5◦C difference between a query and semi-factual case. This method priorities explanatory instances with more convincing feature-values, and may compute these over multiple features. Indeed, these utilities are seen as being classspecific and, even, user-specific, depending on what a given user may find convincing. Furthermore, [Doyle et al., 2004] argued that these utility values often decreased as instances approach the decision boundary, as they were more likely to be outliers in the class and, therefore, less convincing. They user tested this method in a medical domain, showing that semi-factuals provided better explanations than the top-3 nearest neighbors for queries tested.

However, this method was knowledge-intensive, the utility values for each feature had to be hand-coded for each class (and, presumably, for each end-user). Indeed, in one of their user tests, the utility measures had to be re-defined half-way through the study to better reflect end-users' assessments [Doyle et al., 2006]. This is a major drawback for the technique, as it begs the critical question about what featuredifferences will actually be more convincing. Accordingly, this utility method is not a plausible benchmark, though we do use their intuition about feature-differences to define a new, useful benchmark method (see section 5.4).

5.2 NUN-Related Semi-Factuals

Cummins & Bridge's [2006] "Knowledge-Light based Explanation-Oriented Retrieval" (KLEOR) approach proposed three methods based on similarity to Nearest Unlike Neighbors (NUNs). These KLEOR variants use the NUN to find the best semi-factual for a given query (n.b., they called the NUN, a Nearest Miss). In modern parlance, the NUN is the closest instance in the dataset bearing a counterfactual relationship to the query (see [Keane and Smyth, 2020]).

The first variant, Sim-Miss, selects an instance to be the semi-factual which is most similar to the NUN but in the same class as the query q:

where q is the query, x is the instance, G represents the set of all instances in the same class as the query, and nun is the Nearest Unlike Neighbor, with Sim being Euclidean Distance or Cosine Similarity. This variant is the most na¨ıve as it assumes a simple decision boundary. The second variant, Global-Sim method, is more sophisticated in that it requires the semi-factual be closer to q than to the nun (to avoid SFs far from the query but close to the NUN):

using the global similarity between instances. Finally, the third variant, Attr-Sim, computes more fine-grained similarities for each feature-attribute, ensuring that the semi-factual lies between the q and nun across the majority of features:

where F is the feature-dimension set and a is a featureattribute. These methods rely on the interesting intuition that a known counterfactual can be guide to finding a good semifactual explanation; that is, the semi-factual is to be found between the query and the NUN in the feature space.

Cummins & Bridge's carried out a computational evaluation of these three KLEOR variants using test-instances from a single dataset, showing that each method could find semifactuals for most queries. The less restrictive SFSim-Miss method had the best coverage and, counter-intuitively, the most restrictive SFAttr-Sim did better than SFGlobal-Sim. They also performed a psychological evaluation showing that SFSim-Miss and SFAttr-Sim found semi-factuals that people thought to be as good as those found by SFUtility, notably without the latter's knowledge engineering overheads. Accordingly, all three of these KLEOR variants were used in the present benchmarking study (see section 6).

5.3 Semi-Factuals &Local-Region Boundaries

Nugent et al. [2009] proposed another a fortiori method, by finding marginal instances in the local region around the query. Here, a surrogate model, specifically, logistic regression was used to capture the local neighborhood of the query, built using subset of instances located around it (akin to LIME [Ribeiro et al., 2016]). This use of known instances ensures that the surrogate model learns about the local space surrounding the query, finding a reliable proxy justifying query's global behaviour. Finally, candidate nearest neighbors are tested using this local model to give a probability, with the marginal-probability instance closest to the decision boundary, being chosen as the semi-factual explanation, as follows:

where, C is the set of candidate neighbors and LR() is the local logistic regression model providing the probability score.

The intuition here is that good semi-factuals should be close to the query's local decision boundary, while being as far as possible from it in this local space (as in Fig.1). So, a convincing semi-factual explanation should be locally close to the query but as distant from it as possible within this local region. Unfortunately, the authors did not computationally evaluate the method, though they did ask users to evaluate the explanations output. Their results showed that users had higher satisfaction and confidence in the semi-factual explanations compared to conventional nearest-neighbor examplebased explanations. Accordingly, this method is also used in the present benchmarking study (see section 6).

5.4 ANew Benchmark: Most Distant Neighbors

Analogies between counterfactual XAI and semi-factuals suggest another na¨ıve benchmark that has not been proposed before in the literature. Early counterfactual methods often used Nearest Unlike Neighbors (NUNs), the nearest classdifferent instance in the dataset to the query, as counterfactual explanations [Cunningham et al., 2003; Wexler et al., 2019]. NUNs can be a reasonable first-pass at counterfactuals that are guaranteed to be within-domain (though they have other weaknesses). An analogous solution for semi-factual explanations relies on the notion of Most Distant Neighbors (MDNs); namely, the most distant same-class instance in the dataset to the query on some key-feature. MDNs should be good semi-factuals because they reflect many of the desiderata and are, by definition, within domain.

To compute MDNs, for a given feature of q, its neighbours on the dimension are partitioned into instance-sets that have higher values (i.e., HighSet) or lower values (i.e., LowSet) than the query. Each of these sets are ranked-ordered separately using a Semi-Factual Scoring (sfs) function, a distance messure that prioritises instances that are sparse (few feature differences) while also having the highest value-differences on a key-feature, as follows:

where S is HigherSet or LowerSet and x ∈ S, same() counts the features that are equal between q and x, F is the total

Algorithm 1 MDN Semi-factual

number of features, diff() gives the difference-value of keyfeature, f, and diffmax () is the maximum difference-value for that key-feature in the HighSet/LowSet. Basically, the instance with the highest overall sfs value from the HighSet/LowSet is the best candidate for that feature. This computation is done for each feature of q, independently, with the best of the best instances (i.e., with the highest sfs value across all features) being chosen as the overall semi-factual for the query (see Algorithm 1).

The intuition behind MDNs is that if one can find a instance that has some features in common with the query but is as far from it on a key-feature, then it will make a good semifactual (see desiderata). This new method was also added to benchmarking study to compare it to the historical methods.

5.5 The Modern Era: Post-2020Methods

Kenny & Keane [2021] instigated, what could be called, the modern-era of semi-factual AI research with their GANbased counterfactual method, called PIECE, that also computed semi-factuals. PIECE finds "exceptional" and "normal" features for a given class and then modifies the query's "exceptional" features to create instances that have the "normal" features of the counterfactual class, using a GAN to generate image visualisations. As successive exceptionalfeatures are changed the generated instances move away from the query towards the counterfactual class, with the instance generated just before the decision boundary being chosen as the semi-factual. Kenny & Keane showed these semi-factuals to be more distant from the query than those found by other perturbation techniques (see Expt.2). In one sense, this solution re-imagines the Cummins-Bridge intuition that good semi-factuals can be found somewhere between the query and a counterfactual, close to the decision boundary.

PIECE kicked off a renewed interest in semi-factual XAI as researchers looked to improve it and to apply semi-factuals in different application contexts. So, Zhao et al. [2022] have proposed a class-to-class variational encoder (C2CVAR) which is less computationally expensive than PIECE that can generate semi-factuals (and counterfactuals; see also [Ye et al., 2020]). Vats et al. [2022] have used StyleGAN2 [Karras et al., 2020] to find semi-factual explanations for classifications of medical images of ulcers. Though these works try to explain model capabilities, others have proposed using semi-factuals to explain model limits. Artelt & Hammer [2022] use semi-factuals to explain the "reject option"; where an AI system rejects inputs because "a prediction with an unacceptable lower certainty" can only be made. Their perturbation-based optimisation method uses a loss function that promotes diverse semi-factuals that are (i) in the same class as the query (they are also rejected), (ii) sparse (they aim for 1-feature-difference), (iii) "sufficiently distant" from the query, and (iv) of higher certainty than the query (to make themmore convincing). Notably, here, the key-feature being varied is the certainty of the instance's prediction. In a similar vein, Lu et al. [2022] argue that semi-factuals may be used to explain spurious patterns in human-in-the-loop ML (see also [Hagos et al., 2022]). Finally, Mertes et al. [2022] propose an (apparently) wholly new type of counterfactual, called "alterfactuals", to explore the"irrelevant feature" space of the model; they describe these as semi-factuals that "move parallel to the decision boundary, indicating which features would not modify the model's decision".

Other similar proposals have also been made, though they suffer from a poor knowledge of the literature. [Fern´andez et al., 2022] propose a framework for contrastive explanations, called "explanation sets", in which they use a very loose, non-standard definition of semi-factuals (i.e., as basically any instance in some sub-region of the query's class), perhaps because they seem to be unaware of the prior literature. [Herchenbach et al., 2022] also overlook prior work in their proposal of a broad framework involving near hits and near misses (counterfactuals) as image-explanations, though it is not clear whether the former are really semi-factuals (or just nearest neighbors).

Finally, from the user perspective, Mueller et al. [2021] include a semi-factual module in their cognitive tutorial for training users about "cognitively-challenging aspects of an AI system" and [Salimi, 2022] reports user-tests for trustworthiness after using semi-factuals.

These recent papers reflect a rapidly-expanding interest in semi-factual XAI. In time, these modern-era methods will need to be comparatively evaluated relative to the benchmarks and metrics proposed here, to determine which fare best in explaining predictions to end-users.

6 Benchmarking Study

To provide a firm empirical basis for future work on semifactual XAI, we ran a benchmark study of five methods, the four historical methods [i.e., the three KLEOR methods (SimMiss, Global-Sim, Attr-sim) and the Local-Region one] and the newly-proposed MDN method. Standard evaluation metrics from prior XAI work were used to compare these methods, using the five measures detailed below.

Query-to-SF Distance. The L2-norm from the Query to the SF, where higher scores are better, as the semi-factual should be far from the query.

Query-to-SF kNN (%). This is a measure of the percentage of instances (within the whole dataset) in the k-NN set surrounding the Query that occur before the SF is included

(i.e., as k is successively increased upto the appearance of the SF); it is an alternative measure for how far the SF is from the Query in the dataset, so higher values are better.

SF-to-Query-Class Distance. A within-distribution measure for the closeness of the SF to the distribution of the Query-Class using Mahalanobis distance [Chandra and others, 1936], where lower values indicate that the SF is closer to the query-class distribution.

SF-to-NUN Distance. The L2-norm from the SF to the NUN, where lower scores are better as the semi-factual is closer to the class boundary.

MDN Distance. The sfs function, a semi-factual-oriented distance from a Query to a SF, can used to determine how far the SFs selected by historical methods are from the Query; this metric allows us to assess whether historical methods find "better" MDNs than the MDN-method itself, where higher sfs values indicate the SF is a better MDN for the Query.

Sparsity (%). The L0-norm counting the number of feature-differences between the Query and SF, divided into three levels (i.e., 1-diff, 2-diff and >3-diff) where the percent of SFs selected by the method at each level is recorded; obviously, methods with higher percentage at lower difference levels are better (ideally, high-percentage at the 1-diff level).

6.1 Method

We performed leave-one-out cross-validation for each of the five methods on seven datasets to find a semi-factual for every instance in the dataset, treating each as a query. We used 3-NN model to implement the KLEOR variants. For the Local Region method, we consider a minimum of 200 instances from each class to build the local model for a query. In the MDN method, a "20% of the standard deviation" threshold was used to determine whether values for a given feature were essentially "the same". The seven datasets were benchmark, publically-available, tabular datasets commonly used in the counterfactual literature, all binary-classed: Adult-Income (N=26,540, 12 features), Blood Alcohol (N=2,000, 5 features), Default Credit Card (N=30,000, 23 features), Pima Diabetes (N=392, 8 features), German Credit (N=1,000, 20 features), HELOC (8,291 instances, 20 features), Lending Club (N=39,239, 8 features). All the experiments were carried out in Python 3.9 on Ubuntu 16.04 machine with 40 core Intel Xeon(R) processor with an approximate run-time of 40 hours. All programs, data and results are available at https://github.com/ itsaugat/sf survey.

6.2 Results &Discussion

Figure 2 summarises the overall results for the five methods (as mean ranks over datasets) on the six benchmark measures (Figures 3, 4 and 5 show results by-dataset). The summary shows that MDN does best on three of the six measures (i.e., Query-to-SF Distance, Query-to-SF kNN, MDN Distance), with the Local Region method being a close second; performance on the other metrics (SF-to-Query-Class Distance, SFto-NUN Distance and Sparsity) requires some interpretation.

On the Query-to-SF Distance metric (Figure 3a) it can be seen that MDN produces the highest Query-to-SF distances for 4 of the 7 datasets, showing that it tends to find the furthest SF-instances from the query. On the Query-SF kNN metric (Figure 3b) MDN again scores the highest in 3 of 7 datasets with overall percentages that stand out; so, MDN finds SFs separated from the Query by many instances. On the MDNDistance metric (Figure 4) the four historical methods mainly produce lower scores across datasets (except for the HELOC dataset) showing that the MDN method is finding the furthest SFs from the Query in the dataset (i.e., it is finding the best MDNs in each dataset).

With respect to the other metrics MDN does less well. On the SF-to-Q-Class Distance measure (Figure 3c) MDN is ranked 4th; though all its SFs are by-definition within distribution (as valid datapoints), MDN probably scores lower as it is finding outliers in the distribution. On the SF-to-NUN Distance metric (Figure 3d), the KLEOR variants perform relatively better than the MDN and Local Region methods; however, this result is not that surprising as these methods are optimised to find SFs close to NUNs. Finally, on the sparsity measure (see Figure 5) MDN markedly differs from all other methods. In Figure 5, the higher the blue-portion of the bars [i.e., the % of 1-diff SFs] for a given method-dataset pair, the better the performance. MDN has very little blue, it is the worst of all the methods in three datasets where 100% of its SFs have >3-feature-differences (though in three others it fares better). This performance could probably be improved by fine-tuning the sfs function [see formula (7)]. Recall, that this function has two equally-weighted components, that compute (i) same-features and (ii) relative-differences in the key-feature. If a higher weight was given to the same-features component, then the method should select sparser SFs (perhaps also aided by a scoring threshold). For the present work, we felt it was better to provide a vanilla sfs function to get a clear sense of how a baseline-MDN method might work.

Overall, in conclusion, though it seems that the MDN and the Local Region methods provide the best candidates for semi-factual baselines. The Local Region method provides reasonable, solid results with decent sparsity, whereas the MDNmethod shows the furthest SF-point in the dataset from the Query (as type of upper limit to beat).

7 Conclusion

In recent years, counterfactual explanations have been heavily researched as a significant explanation strategy in XAI. Yet, very little attention has been given to an, arguably, equally useful method that relies on semi-factuals (where changes to input features do not lead to output changes). In this paper, from a systematic survey, we aim to remedy this deficit and place this topic area on a firm footing with defined desiderata, benchmarked methods and suitable metrics. In conclusion, several limitations and caveats are to be noted.

With respect to limitations, it is to be noted that in the current benchmark study we have concentrated on tabular data, largely to respect the focus of historical methods. However, the desiderata and evaluation metrics should equally apply to image dataset (and possibly time-series data), albeit relying more on latent features (as has been demonstrated in [Kenny and Keane, 2021]). The paucity of user studies is another severe limitation; until some carefully-controlled studies are carried out, we do not really know how users will respond to these explanations in the AI context.

With respect to caveats, we believe that it is important to reiterate the ethical point about semi-factual use (a point that also applies to counterfactuals [Asher et al., 2022]). These explanatory methods have significant cognitive impacts on people's understanding of AI systems and domains, they convince and dissuade people. But, they could be misused if certain assumptions are violated (e.g., if the SF is not robust). So, future implementations of these methods will need to provide metrics to audit these assumptions, to ensure they are being properly and fairly applied in advice to end-users.

Acknowledgments

This publication has emerged from research conducted with the financial support of Science Foundation Ireland (SFI) to the Insight Centre for Data Analytics under Grant Number 12/RC/2289 P2.