Twin-Systems to Explain Artificial Neural Networks using Case-Based Reasoning: Comparative Tests of Feature-Weighting Methods in ANN-CBR Twins for XAI

1 Introduction

The recent explosion in the use of AI for prediction and decision making in our daily lives has raised questions about the ability of these systems to explain what they do [Ribeiro et al., 2016]; this problem is especially evident in artificial neural networks (ANN). Although the problem of eXplainable AI (XAI) is not new [Clancey, 1983], there is a renewed urgency to solving this problem because of the emerging ubiquity of these systems and new data regulations that encourage AI systems to explain their decisions (e.g., GDPR). This paper proposes the twin-system approach to address the XAI problem, in which a less interpretable system is mapped to one generally considered more interpretable, with both using the same dataset. We demonstrate this twinning approach has potential to help explain the decisions of "black box" AI systems. This twin-system approach has some commonality with white box testing [Pei et al., 2017], however the twin-system idea differs in its mapping of the ANN into a separate "twin" model that is more interpretable.

∗Contact Author

ANN-CBR twins. Twin-systems are based on the premise that if meaningful mappings can be computed from a black box model to its white box "twin", then we can understand and explain the former using the latter. Whilst there are many options for how different AI techniques could be twinned, this paper explores the twinning of opaque ANN systems with arguably more transparent case-based reasoning (CBR) systems (i.e., ANN-CBR twins). CBR systems have qualitatively different interpretable properties to ANNs, as they employ an intuitive reasoning from precedent or example method based on similarity between cases, where these cases can provide useful explanations for predictions [Cunningham et al., 2003]. In an ANN-CBR twinning both systems work from an identical dataset, and the ANN is analyzed to find the feature weights that contributed to the outputs produced, these are then used in the CBR system to retrieve cases which can explain the ANN's outputs. Assessing the quality of extracted ANN feature weights has historically been a difficult process [Shin et al., 2000; Olden et al., 2004], but here we use a novel evaluation method that quantitatively measures predictive agreement between the twinned systems (see Section 3.1).

Explanation. InAI, as in Psychology and Philosophy, there is no agreed definition for the notion of "explanation" (see e.g., [Sørmo et al., 2005]), and a wider discussion of this is not possible here. However, often a distinction is made between explanations that make the system more transparent (by directly describing the reasoning/decision process of the system that led to its outputs) or that justify the system (i.e., by evidentially providing some basis for the system's outputs). The twin-system approach proposed here, is arguably, much closer to justification than transparency (e.g., [Lipton, 2018] calls it "post-hoc explanation by example").

Paper outline. This paper explores ANN-CBR twinning as a solution to the XAI problem, by comparatively testing different feature-weighting schemes. In Section 2, previous approaches to twinning are reviewed. Section 3 describes the feature-weighting techniques tested (including our novel method COLE) and the new evaluation methodology used. Sections 4 and 5 report two experiments testing MLP-CBR and CNN-CBR twins, respectively, on benchmark datasets. Section 6 concludes and considers future directions for this approach (see code at https://github.com/EoinKenny/ Twin-Systems).

2 PriorWork

2.1 Artificial Neural Networks (ANN)

Biologically inspired, ANNs deal well with both structured and unstructured data [Shin et al., 2000; Chen et al., 2018]. A division is often made between traditional multilayer perceptron networks (MLPs), and the more recent deep learning models [such as, convolutional neural networks (CNNs)]. However, the non-linear nature of both types make them difficult to interpret and poor at explaining their outputs [Olden and Jackson, 2002; Selvaraju et al., 2017]. MLPs trained on tabular data have explicit input features, but their contribution to outputs is opaque. Statistical methods based on sensitivity analysis [Bai et al., 2011] have dominated attempts to explain how these systems operate, but we demonstrate here that recent techniques developed specifically for ANNs are more useful in twin-systems. CNNs can be even more opaque with regards to their features and respective contributions to classifications. Attempts to improve the interpretability of CNNs have largely focused on visualizing what specific neurons in the network have learned (see e.g., [Zeiler and Fergus, 2014; Erhan et al., 2009]). However, arguably, such visualizations still have to be contextualized.

2.2 Case-Based Reasoning (CBR)

CBR is an AI technique that has been applied in a wide variety of domains for many different tasks (e.g., diagnosis, design, classification, recommendation, and law) [De Mantaras et al., 2005; Bench-Capon, 2017]. CBR's intuitive method involves reasoning from example or precedent via four main processing steps: retrieval, reuse, revise and retrain. Accordingly, CBR systems can explain their outputs using cases [Cunningham et al., 2003]. At its simplest, when a query case is presented, the most similar cases to it are retrieved before being used directly (or adapted) to make a prediction. Typically, the retrieval step finds cases by matching the features of the query against the case base using k-nearest neighbor (k-NN). Retrieval accuracy can heavily depend on the weights given to these features, which reflect their importance in the domain. For example, a CBR system predicting house prices might weight the location feature over floorspace, because location is much more important for predicting the house price (see Fig. 1). So, the extraction of accurate feature weights is a key step in twin-systems because they determine the cases retrieved.

2.3 Twin-Systems

Hybrid systems typically combine several AI techniques to meet a task requirement [Medsker, 2012]. For example, Reategui et al. [1997] combined an ANN and CBR system for medical diagnosis, where the ANN generated hypotheses to direct the search for cases in a CBR system (for a recent example using deep learning see Amin et al. [2018]). However, these systems are not twin-systems in our sense, as the motivation for combining techniques is to build a processing pipeline to achieve some task, where each technique is typically insufficient on its own. Furthermore, the primary motivation is not explanation. In contrast, the twin-system idea uses two models alongside one another where the purpose for using both techniques is, specifically, explanation. In effect, twin-systems are a distinct, special case of hybrid systems (see Keane and Kenny [2019a], [2019b] for a survey and definitions).

Previous Twin-Systems and MLPs

Examples of hybrid systems using MLPs exist that meet the twin-system definition (see [Keane and Kenny, 2019a] for a review). Shin et al. [2000] built MLP-CBR twins to test the accuracy of different feature-weighting schemes, mapping between the MLP and CBR systems to retrieve explanatory cases using k-NN. They found that a weighted k-NN worked better than an unweighted one, provided good explanatory cases, and was more resilient in noisy feature spaces. This work mainly explored global weighting methods, which assume that the input space is isotropic, and use a single ubiquitous feature-weight vector for each domain.

In contrast, Park et al. [2004] examined local weighting by accounting for varying feature weights across the instance space, but their solution cannot be applied to find posthoc explanations for standard MLPs. Nugent and Cunningham [2005] used local-weighting methods by perturbing a query to build a localized dataset, fitting a linear model to this new data to approximate the MLP function locally, and then using its coefficients to weight the CBR's k-NN search. The present paper builds on these previous efforts by considering the weighting of the entire training set and the individual weighting of query instances (see Section 3).

Recent Twin-Systems and Deep Learning

Beyond this earlier work, we have found few recent papers that could be called twin-systems in deep learning. Here, visual explanations are typically used such as activity maximization [Erhan et al., 2009], deconvolution [Zeiler and Fergus, 2014], activation maps [Zhou et al., 2015], and saliency maps [Bach et al., 2015]. However, recently k-NN has been combined with CNNs using nearest neighbor approaches [Papernot and McDaniel, 2018; Sani et al., 2017], but they rely on neuron activations to fit their k-NN "twins", an approach which we show is not always applicable (see Section 5).

Since the above approaches are often seen as compromising interpretability for accuracy, recent attempts have built interpretability into the network itself using prototypes and CBR ideas. Li et al. [2017] used an encoder to represent features in the latent space and judged nearby cases via distance metrics. However, a decoder is required for visualizing prototypes as explanations, and although this solution performed well on MNIST, it reportedly fails in more complex domains, such as CIFAR-10 [Chen et al., 2018]. Chen et al. [2018] expanded this work by allowing their prototypes to represent parts of images, that were projected onto representations in training data for visualization and explanation. These are promising approaches, but at present are difficult to train, achieve lower accuracy than standard models, and require additional hyperparameters (e.g., number and size of prototypes). In Expt. 2, we develop CNN-CBR twins in which higher computational costs are avoided, and accuracy is maintained, while still finding plausible explanations (see Section 5).

3 Techniques for Extracting FeatureWeights

In the above review, we saw that global and local techniques are distinguished for extracting feature weights from an ANN. Global techniques use a single weight-vector ubiquitously, whereas local techniques generate instance-specific weight-vectors. In Expt. 1, we explore MLP-CBR twinsystems by competitively testing seven different weighting methods: three global and four local techniques (see Section 4). These methods represent the best techniques identified in the literature. However, they have not all been comparatively tested before.

Global Weighting Methods

All global weighting methods use the following formula when calculating distances in k-NN searches:

where |wf | is the absolute weight derived for feature f, xf is the feature f in training instance x, qf is the feature f in the query instance q, and n is the total number of features. The three global methods are:

Sensitivity (SENS). Used by Shin et al. [2000], this works by setting a feature's input to zero and measuring the difference between the network's new and old predictions to gauge the feature's importance. This technique has been found to be very promising in their research.

where Si is the sensitivity (feature weight) of input i, P 0 is the original prediction of the network, P i is the prediction after said input is removed, L is the set of training data and n the number of training set instances.

Perturbation (PTB). A popular method for extracting feature weights from MLPs [de O˜na and Garrido, 2014], the method works by perturbing input features to observe the effect on the output. Previous work shows the optimal perturbation range to be perhaps 20% [Bai et al., 2011], which we use. This technique has not been tested in MLP-CBR systems, we propose the following formulation to do so:

where Wi represents the global weight of feature i, σ the perturbation range, n the number of training set instances, L the training data, and δ a function returning the absolute summed change in two predictions with a positive and negative perturbation respectively to the feature i in instance Lj.

ConnectionWeights (CW). Advanced by Olden and Jackson [2002], as a possible improvement on Garson's Algorithm [Garson, 1991] (which considers the absolute values of a network's weights to derive feature weighting), this considers if the network's weights are positive or negative:

where Rij is the relative importance of the variable xi with respect to the output neuron j, H is the number of neurons in the hidden layer, Wik is the connection weight between the input neuron i and the hidden neuron k, and Wkj is the weight between the hidden neuron k and the output neuron j.

Local Weighting Methods

Of the four local methods, the first uses (1) but computes the weight vector individually for each test instance. The last three use a standard k-NN algorithm, but it is fit to a different dataset of "contributions" to weight the search. The four methods examined are described below.

Local Linear Model (LLM). This is a name we coin for Nugent and Cunningham's [2005] method for finding feature weights from an MLP to inform case retrieval for explanation. This method perturbs the features of a query randomly to generate a new local dataset, the MLP is then used to generate labels for this new data. Using the new dataset a linear model is then built which works to approximate the MLP function locally around the query. The linear model's coefficients are then used to weigh features in the k-NN search. Recently, Ribeiro et al. [2016] generalized a similar approach in Local Interpretable Model-Agnostic Explanations (LIME), whose code library is used here to implement LLM. As LIME is model agnostic, it can be applied to any classifier, so this method could be used with other twinning options, other than ANN-CBR twins (e.g., SVM-CBR twin-systems).

Contributions Oriented Local Explanations (COLE: CLPR, C-IG, C-DeepLIFT). Our method COLE is based on the premise that the contributions of features in a model's classification represent the most sensible basis to inform casebased explanations. Consider a linear model f(⃗x) with n input features and a weight vector ⃗w ∈ IRn . The contribution vector ⃗c of an instance ⃗x is:

where xi.wi calculates ci (i.e., the contribution of feature i). Intuitively, it makes more sense to find explanatory cases using ⃗c, rather than ⃗x or ⃗w, because ⃗c more closely represents the final classification logic of f(⃗x) (as the final output is typically influenced by ∑ni=1 ci +Bias). In f(⃗x), the weights ⃗w are known, but in an MLP determining ⃗w is nontrivial, and it is likely to mutate at every area of the input space. To deal with this issue, we used saliency map techniques, to heuristically estimate ⃗c directly (avoiding the need for ⃗w), as they have proven useful in deep learning [Lundberg and Lee, 2017]. These saliency map methods backpropagate contributions (usually from the output to input layer) and, hence, can provide feature weights for finding explanatory cases using COLE in the context of MLP-CBR twins. We used three saliency map techniques (see Algorithm 1): Layer-wise Relevance Propagation (LRP) [Bach et al., 2015], Integrated Gradients (IG) [Sundararajan et al., 2017] and DeepLIFT [Shrikumar et al., 2017], which we denote as CLPR, C-IG and C-DeepLIFT to show they are COLE variants of these methods. Specifically, we used Epsilon-LRP (ϵLRP), a black reference point for IG, and a reference activation of zero for DeepLIFT. In Expt. 1 the contribution scores for all the training and testing data were generated by attributing scores for the output classification neuron to the inputlayer features (see Algorithm 1); these were then used in a k-NN classifier for the CBR system. The training and testing instances are all individually weighted to represent the MLP locally, enabling the weighted k-NN (i.e., the CBR system) to find explanatory cases based on how discriminating the MLP finds certain features.

3.1 Twin-Systems: EvaluationMethodology

In the present experiments, we want to find the best featureweighting method for MLP-CBR twin-systems. To meet this aim, we developed a novel evaluation methodology that assesses predictive agreement between the twinned systems. This "agreement" score is calculated by dividing the number of agreed predictions on testing data from both models by the number of test instances. As such, 0.0 and 1.0 represent zero and perfect agreement in the twin-system, respectively. Note, this aim is different to determining whether the system can deliver good explanatory cases (but, see Expt. 2 for some qualitative evidence). However, knowing the best feature-weighting scheme clearly will enable better testing of such explanatory capabilities in the future.

To evaluate the weighting, a situation is setup were an increase in agreement between the systems will correlate with an increase the fidelity of the extracted weighting. This is done by making k-NN closely imitate the MLP decision boundary, and then applying the extracted weighting from the

Algorithm 1 COLE weighting

Input: train, training data; test, testing data; MLP, MLP model; δ(⃗x,MLP), function (e.g., DeepLIFT) returning a 2D array A ∈ IRm,n of contributions for all input features to all classification outputs of MLP for instance ⃗x where m is the number of neurons in the last layer, and n is the number of features in the input layer.

Output: trainC, testC, the contributions of input features to the training and testing data classifications of MLP, respectively.

1: trainC, testC = emptyArray, emptyArray 2: for i = 0; to len(train) do 3: idx = 0 // For Sigmoid Output 4: if MLP.outputLayer == Sof tMax then 5: idx = MLP.predictClass(train[i]) 6: end if 7: trainC.append(δ(train[i], MLP )[idx]) 8: end for 9: for i = 0; to len(test) do 10: idx = 0 11: if MLP.outputLayer == Sof tMax then 12: idx = MLP.predictClass(test[i]) 13: end if 14: testC.append(δ(test[i], MLP )[idx]) 15: end for 16: return trainC, testC

MLP to the k-NN to complete the systemmapping. In this situation a high agreement score will signify feature-weights of high quality. Formally, the evaluation has the following steps:

1. Datasets with a large number of features are chosen such that an unweighted k-NN (henceforth k-NN*) is found to have relatively low accuracy on testing data.
2. An MLP trained on the same data is found to have superior accuracy on the testing data (by finding good weights in the feature space).
3. k-NN* is made to closely imitate the MLP decision boundary by temporarily removing outlier cases (defined here as training cases the MLP miss-classifies).
4. Featureweighting is extracted from theMLP and used in the weighted k-NN. So, a high agreement score between the weighted k-NN and MLP signifies that the extracted MLP feature-weighting is of high fidelity.

In this situation, provided the weighting is credible, similar cases will "cluster" together in a weighted k-NN and it will closely mimic the MLP's classifications (which our results confirm), and nearest neighbors will represent cases with similar discriminating features used in a classification. In this evaluation method, outliers were only removed to allow a "clean" test of agreement and, by extension, the different feature-weighting methods 1 .

1 Both experiments were run with and without the outlier data and, as expected, while overall agreement was worse with the outliers left in, there was no appreciable change to the rank ordering of feature-weighting techniques.

4 Experiment 1: MLP-CBR Twins

In this experiment, we measure the agreement for MLP-CBR twins using seven different feature-weighting methods for kNN compared to a baseline unweighted k-NN (k-NN*). The aim being to discover the feature-weighting method that best represents the MLP function for a given domain. In these tests, were are only interested in mapping one system to the other, irrespective of the data split, so cross-validation was not used. The splits ranged from 10-33% for testing data and were selected to challenge the unweighted k-NN* baseline. Values for k were chosen to give the best general performance across all weighting techniques, whilst trying to keep it as low as possible.

4.1 Method: Architecture, Datasets and Procedure

All MLPs used had a single input, hidden, and output layer. The hidden layer used ReLU activation functions and the output used Sigmoid and SoftMax for binary and multi-class classification, respectively. Seven popular datasets were selected and modified as described above (comparable results were found for five regression datasets, that are not reported here). Thus, the MLP and CBR systems were tested on the same datasets using eight different k-NN variants (k-NN* and the seven weighting methods described above). The Connection Weights (CW) method was an exception as it was not developed originally for SoftMax outputs, so its average agreement is over four datasets rather than seven (see Table 1). The measure is the agreement in predictions between the model-pairs (see Section 3.1). The CBR system is a retrievalonly system (there is no adaptation) that simply uses k-NN to find cases from which predictions are derived.

4.2 Results and Discussion

Table 1 summarizes the agreement scores for all tests in Expt. 1. Overall, a feature-weighted k-NN does better than an unweighted k-NN* (Mean Agreement = 0.91). Moreover, the local weighting methods (C-DeepLIFT, C-IG, C-LPR, and LLM) do better on average (Mean Agreement = 0.93-0.99) than global methods (PTB, SENS, CW; Mean Agreement = 0.94-0.95). Within the local methods, the contribution-based methods (i.e., C-DeepLIFT, C-IG, C-LPR) generally do better than the non-contribution LLMmethod, the former often achieving perfect agreeemnt with the MLP, with C-DeepLIFT being the best of all these methods (Mean Agreement = 0.99).

However, LLM is model agnostic (whereas the other local methods are not), meaning it may have wider applicability in other twin-systems. Finally, it should be noted that the multiclass classification problems achieve higher mean agreement scores than binary ones when using the contribution methods. This is possibly due to additional weight parameters in the SoftMax layer allowing for greater discretization of the classes, as the saliency map techniques used to realize COLE here utilize these weight values when calculating contribution scores.

5 Experiment 2: CNN-CBR Twins

This experiment tests the prediction agreement for a deep learning model (a CNN) twinned with a CBR system using a control, k-NN*, and a weighted k-NN computed from the best contributions-based method found in Expt.1 (i.e., CDeepLIFT; see Section 4.2).

5.1 Method: Architecture, Datasets and Procedure

All CNN models used had three convolutional layers with two max pooling layers following the first two convolutions. The models then contained three fully-connected (FC) layers, the last of these being a SoftMax output. Consequently, the first FC layer represents the latent features extracted by the CNN, analogous to the input features of the MLPs tested in Expt. 1. The exception to this is the final test which used the InceptionResNetV2 CNN. This model was used for feature extraction, so a standard three-layer MLP could be trained on these extracted features to discriminate the classes.

When using a CNN for image classification, the pixel space is transformed into a convolutional output before being used for classification. Formally, let the transformation be X → C ∈ IR(h,w,d) where the image tensor X is transformed into a new representation C of the image, where h, w, and d represent the height, width and number of feature maps, respectively. After this a CNN will typically consist of a FC or global average pooling (GAP) layer. Either way, let the next layer be a vector representation ⃗x ∈ IRn of the image, where n is the number of latent features. Provided there are no more layers before the SoftMax output, computing contribution scores can be trivially done by multiplying the feature activations generated for a specific instance in ⃗x ∈ IRn by the weight connecting them to the predicted class output node. In this situation, we can examine the activation map of the class and it's discriminating features [Zhou et al., 2015] (see Fig. 2b).

However, when using additional FC layers as we do here, it is necessary to follow the process outlined in Expt. 1 where contribution scores are generated using (for example) DeepLIFT by attributing scores from the output classification neuron to the feature layer ⃗x ∈ IRn . In Expt. 2, the control kNN* used neuron activations from ⃗x to retrieve cases and the weighted k-NN used contributions from ⃗x with C-DeepLIFT. Selvaraju et al. [2017] proposed a solution to interpretability with multiple FC layers by generalizing the class activation maps (discussed by Zhou et al. [2015]) into more CNN architectures, such as those with several FC layers. However, this solution does not focus on the use of CBR or twin-systems to explain outputs.

Three popular image datasets were selected and modified as described earlier (see Section 3.1). We compared fitting k-NN with activations in the latent feature layer ⃗x ∈ IRn as a control (i.e., k-NN*), and the contributions derived by C-DeepLIFT for the same layer. The first four tests used a CNN trained locally and the last used a pre-trained (on ImageNet) CNN InceptionResNetV2 for prior feature extraction to train a standard MLP. This final model was chosen because it is among the most complex, publicly available CNN models (572 layers), and allows tests using image transfer learning. The agreement measure used is the same as Expt. 1. Finally, as before, the CBR systems were retrieval-only systems using variants of k-NN.

5.2 Results and Discussion

Table 2 summarizes the agreement scores for all the tests made in Expt. 2. Overall, the k-NN using the contribution scores (i.e., C-DeepLIFT) did markedly better than the baseline k-NN*. Furthermore, the mean agreement for CDeepLIFT in CNN-CBR twins was very high (0.996 out of 1.0). Notably, the agreement scores decrease in k-NN* as complexity of the task increases, particularly in InceptionResNetV2, which is a model designed to discriminate between a large number of classes with large number of features; hence it is not surprising that more intricate weighting details are required for mapping accurately between the systems. However, if the CNN parameters had been fined-tuned to CIFAR10, it is likely agreement would be closer to 1.0 for both kNN* and C-DeepLIFT. Lastly, echoing Expt. 1, the SoftMax outputs here achieve very high mean agreement, suggesting that they may always be the optimal choice, even in a binary classification domain when Sigmoid is typically used.

5.3 Qualitative Evidence

The focus here is on finding the best feature-weighting methods for ANN-CBR twin-systems rather than assessing the explanatory value of retrieved cases for users. However, CNNs are often "visually explained" when they are applied to image datasets, mainly because people have sufficient expertise to evaluate the image-predictions made. Fig. 2 shows some of the explanatory cases retrieved from queries to the CNN-CBR twin-system from Expt. 2, trained on the MNIST dataset, using C-DeepLIFT to weight the k-NN search.

Fig. 2a shows a miss-classification by the CNN, where an image of a "6" is categorized as a "0", along with the three nearest neighbors found to explain this prediction. Note these are all cases of a "0", but resemble a "6", making the missclassification of the CNN reasonable. Fig. 2b shows a correct classification of a "4" by the CNN, along with the three explanatory, nearest-neighbor cases. These images also have a feature activation map (FAM) 2 showing the most discriminating feature across the cases. Interestingly, these FAMs show that the presence of a gap across the top of the number "4" is the primary feature making it a "4". Note, the nearestunlike-neighbor found is the class "9" (using the CBR twin), showing the importance of the "link" in discriminating it as a "4". Fig. 3 shows some results for CNN-CBR twins on the CIFAR-10 dataset. So, twin-systems potentially provide justification, scrutability and transparency for the operations of the CNN (as defined by [Tintarev and Masthoff, 2007]).

2 These FAMs resemble the class activation maps of Zhou et al. [2015]. However, here FAMs show the most discriminating feature for a query rather than the whole class. These are computed by multiplying the activations of the convolutional output C ∈ IR(h,w,d) by the weights W ∈ IR(h,w,d) connecting them to the feature contributing most to the classification. The resulting matrix M ∈ IR(h,w,d) is then summed along d into M ∈ IR(h,w) and up-sampled to the original image size before being superimposed.

5.4 Computational Costs

To answer concerns about the computational costs of these methods, it should be said that Algorithm 1 took ∼ 40 seconds to run Expt. 1 with Connect-4 (67, 557 training and testing instances with n=126), whilst in Expt. 2 it took ∼6613 seconds for CIFAR-10 (60, 000 training and testing instances with n=1000). The experiments used a MacBook Pro; processor: 2.9 GHz Intel Core i5; memory: 16 GB 2133 MHz LPDDR3. Hence, the costs of deriving the weights for the k-NN are quite modest. Moreover, this initial computation is only required once, subsequent queries took <30 seconds to derive a single query's weighting and <0.05 seconds to find explanatory cases for both tabular and image data.

6 Conclusions and FutureWork

This paper offers four novel contributions. First, a description of the twin-system framework for XAI where opaque AI black boxes are explained by interpretable white boxes. Second, a promising demonstration of this framework, twinning ANNs and CBR systems for case-based explanation. Third, the identification of the best feature-weighting techniques for such ANN-CBR twin-systems (testing both MLP-CBR and CNN-CBR twins) using a novel evaluation method and showing that our contribution-based method, COLE, does best (using DeepLIFT). Finally, qualitative evidence shows that CNN-CBR twins provide very plausible visual explanations on benchmark image datasets (regardless of the number of FC layers in the model, a point of importance in image transfer learning [Zhang et al., 2018]). For future research, user studies assessing the explanatory value of the cases retrieved are planned, alongside generalizing COLE to be applicable to any machine learning model beyond just ANNs (e.g., for SVM-CBR twin-systems).

Acknowledgments

This publication has emanated from research conducted with the financial support of (i) Science Foundation Ireland (SFI)

to the Insight Centre for Data Analytics under Grant Number 12/RC/2289 and (ii) SFI and the Department of Agriculture, Food and Marine on behalf of the Government of Ireland to the VistaMilk SFI Research Centre under Grant Number 16/RC/3835.