CRAFT: Concept Recursive Activation FacTorization for Explainability

Abstract

Attribution methods, which employ heatmaps to identify the most influential regions of an image that impact model decisions, have gained widespread popularity as a type of explainability method. However, recent research has exposed the limited practical value of these methods, attributed in part to their narrow focus on the most prominent regions of an image – revealing “where” the model looks, but failing to elucidate “what” the model sees in those areas. In this work, we try to fill in this gap with CRAFT – a novel approach to identify both “what” and “where” by generating concept-based explanations. We introduce 3 new ingredients to the automatic concept extraction literature: (i) a recursive strategy to detect and decompose concepts across layers, (ii) a novel method for a more faithful estimation of concept importance using Sobol indices, and (iii) the use of implicit differentiation to unlock Concept Attribution Maps.

We conduct both human and computer vision experiments to demonstrate the benefits of the proposed approach. We show that the proposed concept importance estimation technique is more faithful to the model than previous methods. When evaluating the usefulness of the method for human experimenters on a human-centered utility benchmark, we find that our approach significantly improves on two of the three test scenarios. Our code is freely available: github.com/deel-ai/Craft

1. Introduction

Interpreting the decisions of modern machine learning models such as neural networks remains a major challenge. Given the ever-increasing range of machine learning applications, the need for robust and reliable explainability methods continues to grow [11, 35]. Recently enacted European laws (including the General Data Protection Regulation (GDPR) [37] and the European AI act [43]) require the assessment of explainable decisions, especially those made by algorithms.

In order to try to meet this growing need, an array of explainability methods have already been proposed [14, 50, 59, 61, 62, 66, 69, 71, 76]. One of the main class of methods called attribution methods yields heatmaps that indicate the importance of individual pixels for driving a model’s decision. However, these methods exhibit critical limitations [1, 27, 63, 65], as they have been shown to fail – or only marginally help – in recent human-centered benchmarks [8, 26, 41, 49, 60, 64]. It has been suggested that their limitations stem from the fact that they are only capable of explaining where in an image are the pixels that are critical to the decision but they cannot tell what visual features are actually driving decisions at these locations. In other words, they show where the model looks but not what it sees. For example, in the scenario depicted in Fig. 1, where an ImageNet-trained ResNet mistakenly identifies an image as containing a shovel, the attribution map displayed on the left fails to explain the reasoning behind this misclassification.

Figure 1. The “Man on the Moon” incorrectly classified as a “shovel” by an ImageNet-trained ResNet50.

Heatmap generated by a classic attribution method [55] (left) vs. concept attribution maps generated with the proposed CRAFT approach (right) which highlights the two most influential concepts that drove the ResNet50’s decision along with their corresponding locations. CRAFT suggests that the neural net arrived at its decision because it identified the concept of “dirt” commonly found in members of the image class “shovel” and the concept of “ski pants” typically worn by people clearing snow from their driveway with a shovel instead the correct concept of astronaut’s pants (which was probably never seen during training).

A recent approach has sought to move past attribution methods [40] by using so-called “concepts” to communicate information to users on how a model works. The goal is to find human-interpretable concepts in the activation space of a neural network. Although the approach exhibited potential, its practicality is significantly restricted due to the need for prior knowledge of pertinent concepts in its original formulation and, more critically, the requirement for a labeled dataset of such concepts. Several lines of work have focused on trying to automate the concept discovery process based only on the training dataset and without explicit human supervision. The most prominent of these techniques, ACE [24], uses a combination of segmentation and clustering techniques but requires heuristics to remove outliers. However, ACE provides a proof of concept that it might be possible to discover concepts automatically and at scale – without additional labeling or human supervision. Nevertheless, the approach suffers several limitations: by construction, each image segment can only belong to a single cluster, a layer has to be selected by the user to be used to retrieve the relevant concepts, and the amount of information lost during the outlier rejection phase can be a cause of concern. More recently, Zhang et al. [77] proposes to leverage matrix decompositions on internal feature maps to discover concepts.

Here, we try to fill these gaps with a novel method called CRAFT which uses Non-Negative Matrix Factorization (NMF) [46] for concept discovery. In contrast to other concept-based explanation methods, our approach provides an explicit link between their global and local explanations (Fig. 2) and identifies the relevant layer(s) to use to represent individual concepts (Fig. 3). Our main contributions can be described as follows:

Figure 2. CRAFT results for the prediction “chain saw”.

First, our method uses Non-Negative Matrix Factorization (NMF) to extract the most relevant concepts used by the network (ResNet50V2) from the train set (ILSVRC2012 [10]). The global influence of these concepts on the predictions is then measured using Sobol indices (right panel). Finally, the method provides local explanations through concept attribution maps (heatmaps associated with a concept, and computed using grad-CAM by backpropagating through the NMF concept values with implicit differentiation). Besides, concepts can be interpreted by looking at crops that maximize the NMF coefficients. For the class “chain saw”, the detected concepts seem to be: the chainsaw engine, the saw blade, the human head, the vegetation, the jeans and the tree trunk.

Figure 3.

(1) Neural collapse (amalgamation). A classifier needs to be able to linearly separate classes by the final layer. It is commonly assumed that in order to achieve this, image activations from the same class get progressively “merged” such that these image activations converge to a one-hot vector associated with the class at the level of the logits layer. In practice, this means that different concepts get ultimately blended together along the way. (2) Recursive process. When a concept is not understood (e.g. $𝒞$), we propose to decompose it into multiple sub-concepts (e.g., ${𝒞}_1,{𝒞}_2,{𝒞}_3$) using the activations from an earlier layer to overcome the aforementioned neural collapse issue. (3) Example of recursive concept decomposition using CRAFT on the ImageNet class “parachute”.

(i) A novel approach for the automated extraction of high-level concepts learned by deep neural networks. We validate its practical utility to users with human psychophysics experiments.

(ii) A recursive procedure to automatically identify concepts and sub-concepts at the right level of granularity – starting with our decomposition at the top of the model and working our way upstream. We validate the benefit of this approach with human psychophysics experiments showing that (i) the decomposition of a concept yields more coherent sub-concepts and (ii) that the groups of points formed by these sub-concepts are more refined and appear meaningful to humans.

(iii) A novel technique to quantify the importance of individual concepts for a model’s prediction using Sobol indices [34, 67, 68] – a technique borrowed from Sensitivity Analysis.

(iv) The first concept-based explainability method which produces concept attribution maps by backpropagating concept scores into the pixel space by leveraging the implicit function theorem in order to localize the pixels associated with the concept of a given input image. This effectively opens up the toolbox of both white-box [15, 59, 62, 66, 69, 71, 78] and black-box [14, 47, 55, 57] explainability methods to derive concept-wise attribution maps.

2. Related Work

Attribution methods

Attribution methods are widely used as post-hoc explainability techniques to determine the input variables that contribute to a model’s prediction by generating importance maps, such as the ones shown in Fig.1. The first attribution method, Saliency, introduced in [62], generates a heatmap by utilizing the gradient of a given classification score with respect to the pixels. This method was later improved upon in the context of deep convolutional networks for classification in subsequent studies, such as [66, 69, 71, 76] . However, the image gradient only reflects the model’s operation within an infinitesimal neighborhood around an input, which can yield misleading importance estimates [22] since gradients of large vision models are notoriously noisy [66]. That is why several methods leverage perturbations on the input image to probe the model and create importance maps that indicate the most crucial areas for the decision, such as Rise [55], Sobol [14], or more recently HSIC [50].

Unfortunately, a severe limitation of these approaches – apart from the fact that they only show the “where” – is that they are subject to confirmation bias: while they may appear to offer useful explanations to a user, sometimes these explanations are actually incorrect [1, 23, 65]. These limitations raise questions about their usefulness, as recent research has shown by using human-centered experiments to evaluate the utility of attribution [8, 26, 41, 49, 60].

In particular, in [8], a protocol is proposed to measure the usefulness of explanations, corresponding to how much they help users identify rules driving a model’s predictions (correct or incorrect) that transfer to unseen data – using the concept of meta-predictor (also called simulatability) [11, 18, 39]. The main idea is to train users to predict the output of the system using a small set of images along with associated model predictions and corresponding explanations. A method that performs well on this this benchmark is said useful, as it help users better predict the output of the model by providing meaningful information about the internal functioning of the model. This framework being agnostic to the type of explainability method, we have chosen to use it in Section 4 in order to compare CRAFT with attribution methods.

Concepts-based methods

Kim et al. [40] introduced a method aimed at providing explanations that go beyond attribution-based approaches by measuring the impact of pre-selected concepts on a model’s outputs. Although this method appears more interpretable to human users than standard attribution techniques, it requires a database of images describing the relevant concepts to be manually curated. Ghorbani et al. [24] further extended the approach to extract concepts without the need for human supervision. The approach, called ACE [24], uses a segmentation scheme on images, that belong to an image class of interest. The authors leveraged the intermediate activations of a neural network for specific image segments. These segments were resized to the appropriate input size and filled with a baseline value. The resulting activations were clustered to produce prototypes, which they referred to as “concepts”. However, some concepts contained background segments, leading to the inclusion of uninteresting and outlier concepts. To address this, the authors implemented a postprocessing cleanup step to remove these concepts, including those that were present in only one image of the class and were not representative. While this improved the interpretability of their explanations to human subjects, the use of a baseline value filled around the segments could introduce biases in the explanations [28, 31, 42, 70].

Zhang et al. [77] developed a solution to the unsupervised concept discovery problem by using matrix factorizations in the latent spaces of neural networks. However, one major drawback of this method is that it operates at the level of convolutional kernels, leading to the discovery of localized concepts. For example, the concept of “grass” at the bottom of the image is considered distinct from the concept of “grass” at the top of the image.

3. Overview of the method

In this section, we first describe our concept activations factorization method. Below we highlight the main differences with related work. We then proceed to introduce the three novel ingredients that make up CRAFT: (1) a method to recursively decompose concepts into sub-concepts, (2) a method to better estimate the importance of extracted concepts, and (3) a method to use any attribution method to create concept attribution maps, using implicit differentiation [4, 25, 44].

Notations

In this work, we consider a general supervised learning setting, where $\left({\mathit{x}}_1,\dots ,{\mathit{x}}_n\right)\in {𝒳}^n\subseteq {ℝ}^{n\times d}$ are $n$ inputs images and $\left(y_1,\dots ,y_n\right)\in {𝒴}^n$ their associated labels. We are given a (machine-learnt) black-box predictor $f:𝒳\to 𝒴$, which at some test input $\mathit{x}$ predicts the output $\mathit{f}\left(\mathit{x}\right)$. Without loss of generality, we establish that $\mathit{f}$ is a neural network that can be decomposed into two distinct components. The first component is a function $\mathit{g}$ that maps the input to an intermediate state, and the second component is $\mathit{h}$, which takes this intermediate state to the output, such that $\mathit{f}(\mathit{x})=(\mathit{h}\circ \mathit{g})(\mathit{x})$. In this context, $\mathit{g}(\mathit{x})\subseteq {ℝ}^p$ represents the intermediate activations of $\mathit{x}$ within the network. Further, we will assume non-negative activations: $\mathit{g}(\mathit{x})\ge 0$. In particular, this assumption is verified by any architecture that utilizes $ReLU$, but any non-negative activation function works.

3.1. Concept activation factorization.

We use Non-negative matrix factorization to identify a basis for concepts based on a network’s activations (Fig.4). Inspired by the approach taken in ACE [24], we will use image sub-regions to try to identify coherent concepts.

Figure 4. Overview of CRAFT.

Starting from a set of crops $\mathbf{X}$ containing a concept $𝒞$ (e.g., crops images of the class “parachute”), we compute activations $\mathit{g}(\mathbf{X})$ corresponding to an intermediate layer from a neural network for random image crops. We then factorize these activations into two lower-rank matrices, (U, W). W is what we call a “concept bank” and is a new basis used to express the activations, while U corresponds to the corresponding coefficients in this new basis. We then extend the method with 3 new ingredients: (1) recursivity – by proposing to re-decompose a concept (e.g., take a new set of images containing ${𝒞}_1$) at an earlier layer, (2) a better importance estimation using Sobol indices and (3) an approach to leverage implicit differentiation to generate concept attribution maps to localize concepts in an image.

The first step involves gathering a set of images that one wishes to explain, such as the dataset, in order to generate associated concepts. In our examples, to explain a specific class $y\in 𝒴$, we selected the set of points $𝒞$ from the dataset for which the model’s predictions matched a specific class $𝒞=\left\{{\mathit{x}}_i:\mathit{f}\left({\mathit{x}}_i\right)=y,1\le i\le n\right\}$. It is important to emphasize that this choice is significant. The goal is not to understand how humans labeled the data, but rather to comprehend the model itself. By only selecting correctly classified images, important biases and failure cases may be missed, preventing a complete understanding of our model.

Now that we have defined our set of images, we will proceed with selecting sub-regions of those images to identify specific concepts within a localized context. It has been observed that the implementation of segmentation masks suggested in ACE can lead to the introduction of artifacts due to the associated inpainting with a baseline value. In contrast, our proposed method takes advantage of the prevalent use of modern data augmentation techniques such as randaugment, mixup, and cutmix during the training of current models. These techniques involve the current practice of models being trained on image crops, which enables us to leverage a straightforward crop and resize function denoted by $\mathit{\pi }(\cdot )$ to create sub-regions (illustrated in Fig.4). By applying $\mathit{\pi }$ function to each image in the set $𝒞$, we obtain an auxiliary dataset $\mathbf{X}\in {ℝ}^{n\times d}$ such that each entries ${\mathbf{X}}_i=\mathit{\pi }\left({\mathit{x}}_i\right)$ is an image crop.

To discover the concept basis, we start by obtaining the activations for the random crops $\mathbf{A}=\mathit{g}(\mathbf{X})\in {ℝ}^{n\times p}$. In the case where $\mathit{f}$ is a convolutional neural network, a global average pooling is applied to the activations.

We are now ready to apply Non-negative Matrix Factorization (NMF) to decompose positive activations $\mathbf{A}$ into a product of non-negative, low-rank matrices $\mathbf{U}\in {ℝ}^{n\times r}$ and $\mathbf{W}\in {ℝ}^{p\times r}$ by solving:

$$(U,W)=\underset{U\ge 0,W\ge 0}{\text{arg min}}\frac{1}{2}{\Vert A-U{W}^{\top }\Vert }_F^2,$$ (1)

where $\parallel \cdot {\parallel }_F$ denotes the Frobenius norm.

This decomposition of our activations $\mathbf{A}$ yields two matrices: $\mathbf{W}$ containing our Concept Activation Vectors (CAVs) and $\mathbf{U}$ that redefines the data points in our dataset according to this new basis. Moreover, this decomposition in this new basis has some interesting properties that go beyond the simple low-rank factorization – since $r\ll \min (n,p)$. First, NMF can be understood as the joint learning of a dictionary of Concept Activation Vectors – called a “concept bank” in Fig. 4 – that maps a ${ℝ}^p$ basis onto ${ℝ}^r$, and $\mathbf{U}$ the coefficients of the vectors $\mathbf{A}$ expressed in this new basis. The minimization of the reconstruction error $\frac{1}{2}\parallel \mathbf{A}-\mathbf{U}\mathbf{W}{\parallel }_F^2$ ensures that the new basis contains (mostly) relevant concepts. Intuitively, the non-negativity constraints $\mathbf{U}\ge 0,\mathbf{W}\ge 0$ encourage (i) $\mathbf{W}$ to be sparse (useful for creating disentangled concepts), (ii) $\mathbf{U}$ to be sparse (convenient for selecting a minimal set of useful concepts) and (iii) missing data to be imputed [56], which corresponds to the sparsity pattern of post-ReLU activations $\mathbf{A}$.

It is worth noting that each input ${\mathit{x}}_i$ can be expressed as a linear combination of concepts denoted as ${\mathbf{A}}_i={\sum }_{j=1}^r{\mathbf{U}}_{(i,j)}{\mathbf{W}}_j^{\mathrm{\top }}$. This approach is advantageous because it allows us to interpret each input as a composition of the underlying concepts. Furthermore, the strict positivity of each term – NMF is working over the anti-negative semiring, – enhances the interpretability of the decomposition. Another interesting interpretation could be that each input is represented as a superposition of concepts [13].

While other methods in the literature solve a similar problem (such as low-rank factorization using SVD or ICA), the NMF is both fast and effective and is known to yield concepts that are meaningful to humans [20, 74, 77]. Finally, once the concept bank $\mathbf{W}$ has been precomputed, we can associate the concept coefficients $\mathit{u}$ to any new input $\mathit{x}$ (e.g., a full image) by solving the underlying Non-Negative Least Squares (NNLS) problem ${\min }_{\mathit{u}\ge 0}\frac{1}{2}{∥\mathit{g}(\mathit{x})-\mathit{u}{\mathbf{W}}^{\mathrm{\top }}∥}_F^2$, and therefore recover its decomposition in the concept basis.

In essence, the core of our method can be summarized as follows: using a set of images, the idea is to re-interpret their embedding at a given layer as a composition of concepts that humans can easily understand. In the next section, we show how one can recursively apply concept activation factorizations to preceding layer for an image containing a previously computed concept.

3.2. Ingredient 1: A pinch of recursivity

One of the most apparent issues in previous work [24, 77] is the need for choosing a priori a layer at which the activation maps are computed. This choice will critically affect the concepts that are identified because certain concepts get amalgamated [53] into one at different layers of the neural network, resulting in incoherent and indecipherable clusters, as illustrated in Fig. 3. We posit that this can be solved by iteratively applying our decomposition at different layer depths, and for the concepts that remain difficult to understand, by looking for their sub-concepts in earlier layers by isolating the images that contain them. This allows us to build hierarchies of concepts for each class.

We offer a simple solution consisting of reapplying our method to a concept by performing a second step of concept activation factorization on a set of images that contain the concept $𝒞$ in order to refine it and create sub-concepts (e.g., decompose $𝒞$ into $\left\{{𝒞}_1,{𝒞}_2,{𝒞}_3\right\}$) see Fig. 3 for an illustrative example. Note that we generalize current methods in the sense that taking images $\left({\mathit{x}}_1,\dots ,{\mathit{x}}_n\right)$ that are clustered in the logits layer (belonging to the same class) and decomposing them in a previous layer – as done in [24, 77] – is a valid recursive step. For a more general case, let us assume that a set of images that contain a common concept is obtained using the first step of concept activation factorization.

We will then take a subset of the auxiliary dataset points to refine any concept $j$. To do this, we select the subset of points that contain the concept ${𝒞}_j=\left\{{\mathbf{X}}_i:{\mathbf{U}}_{(i,j)}>|{\lambda }_j,1\le i\le n\right\}$, where ${\lambda }_j$ is the 90th percentile of the values of the concept ${\mathbf{U}}_{(1,j)},\dots ,{\mathbf{U}}_{(n,j)}$. In other words, the 10% of images that activate the concept $j$ the most are selected for further refinement into sub-concepts. Given this new set of points, we can then re-apply the Concept Matrix Factorization method to an earlier layer ${\mathit{g}}^{\prime }(\cdot )$ to obtain the sub-concepts decomposition from the initial concept – as illustrated in Fig.3.

3.3. Ingredient 2: A dash of sensitivity analysis

A major concern with concept extraction methods is that concepts that makes sense to humans are not necessarily the same as those being used by a model to classify images. In order to prevent such confirmation bias during our concept analysis phase, a faithful estimate the overall importance of the extracted concepts is crucial. Kim et al. [40] proposed an importance estimator based on directional derivatives: the partial derivative of the model output with respect to the vector of concepts. While this measure is theoretically grounded, it relies on the same principle as gradient-based methods, and thus, suffers from the same pitfalls: neural network models have noisy gradients [66, 71]. Hence, the farther the chosen layer is from the output, the noisier the directional derivative score will be.

Since we essentially want to know which concept has the greatest effect on the output of the model, it is natural to consider the field of sensitivity analysis [9, 33, 34, 67, 68]. In this section, we briefly recall the classic “total Sobol indices” and how to apply them to our problem. The complete derivation of the Sobol-Hoeffding decomposition is presented in Section D of the supplementary materials. Formally, a natural way to estimate the importance of a concept $i$ is to measure the fluctuations of the model’s output $\mathit{h}\left(\mathbf{U}{\mathbf{W}}^{\mathrm{\top }}\right)$ in response to meaningful perturbations of the concept coefficient ${\mathbf{U}}_{(1,i)},\dots ,{\mathbf{U}}_{(n,i)}$. Concretely, we will use perturbation masks $\mathbf{M}=\left(M_1,\dots ,M_r\right)\sim 𝒰\left([\mathrm{0,1}{]}^r\right)$, here an i.i.d sequence of real-valued random variables, we introduce a concept fluctuation to reconstruct a perturbed activation $\tilde{\mathbf{A}}=(\mathbf{U}\odot \mathbf{M}){\mathbf{W}}^{\mathrm{\top }}$ where $\odot$ denote the Hadamard product (e.g., the masks can be used to remove a concept by setting its value to zero). We can then propagate this perturbed activation to the model output $\mathbf{Y}=\mathit{h}(\tilde{\mathbf{A}})$. Simply put, removing or applying perturbation of an important concept will result in a substantial variation in the output, whereas an unused concept will have minimal effect on the output.

Finally, we can capture the importance that a concept might have as a main effect – along with its interactions with other concepts – on the model’s output by calculating the expected variance that would remain if all the concepts except the $i$ were to be fixed. This yields the general definition of the total Sobol indices.

Definition 3.1 (Total Sobol indices). The total Sobol index ${𝒮}_i^T$, which measures the contribution of a concept i as well as its interactions of any order with any other concepts to the model output variance, is given by:

$${𝒮}_i^T=\frac{{𝔼}_{M_{\sim i}}({𝕍}_{M_i}(Y\mid M_{\sim i}))}{𝕍(Y)}$$ (2)

$$=\frac{{𝔼}_{M_{\sim i}}({𝕍}_{M_i}(\mathit{h}((U\odot M)W^{\top })\mid M_{\sim i}))}{𝕍(\mathit{h}((U\odot M)W^{\top }))}.$$ (3)

In practice, this index can be calculated very efficiently [36, 48, 52, 58, 72], more details on the Quasi-Monte Carlo sampling and the estimator used are left in appendix D.

3.4. Ingredient 3: A smidgen of implicit differentiation

Attribution methods are useful for determining the regions deemed important by a model for its decision, but they lack information about what exactly triggered it. We have seen that we can already extract this information from the matrices $\mathbf{U}$ and $\mathbf{W}$, but as it is, we do not know in what part of an image a given concept is represented. In this section, we will show how we can leverage attribution methods (forward and backward modes) to find where a concept is located in the input image (see Fig. 2). Forward attribution methods do not rely on any gradient computation as they only use inference processes, whereas backward methods require back-propagating through a network’s layers. By application of the chain rule, computing $\partial \mathbf{U}/\partial \mathbf{X}$ requires access to $\partial \mathbf{U}/\partial \mathbf{A}$.

To do so, one could be tempted to solve the linear system ${\mathbf{U}\mathbf{W}}^{\mathrm{\top }}=\mathbf{A}$. However, this problem is ill-posed since ${\mathbf{W}}^{\mathrm{\top }}$ is low rank. A standard approach is to calculate the Moore-Penrose pseudo-inverse ${\left({\mathbf{W}}^{\mathrm{\top }}\right)}^{†}$, which solves rank deficient systems by looking at the minimum norm solution [2]. In practice, ${\left({\mathbf{W}}^{\mathrm{\top }}\right)}^{†}$ is computed with the Singular Value Decomposition (SVD) of ${\mathbf{W}}^{\mathrm{\top }}$. Unfortunately, SVD is also the solution to the unstructured minimization of $\frac{1}{2}{∥\mathbf{A}-{\mathbf{U}\mathbf{W}}^{\mathrm{\top }}∥}_F^2$ by the Eckart-Young-Mirsky theorem [12]. Hence, the non-negativity constraints of the NMF are ignored, which prevents such approaches from succeeding. Other issues stem from the fact that the $\mathbf{U},\mathbf{W}$ decomposition is generally not unique.

Our third contribution consists of tackling this problem to allow the use of attribution methods, i.e., concept attribution maps, by proposing a strategy to differentiate through the NMF block.

Implicit differentiation of NMF block

The NMF problem 1 is NP-hard [73], and it is not convex with respect to the input pair $(\mathbf{U},\mathbf{W})$. However, fixing the value of one of the two factors and optimizing the other turns the NMF formulation into a pair of Non-Negative Least Squares (NNLS) problems, which are convex. This ensures that alternating minimization (a standard approach for NMF) of $(\mathbf{U},\mathbf{W})$ factors will eventually reach a local minimum. Each of this alternating NNLS problems fulfills the Karush-–Kuhn-–Tucker (KKT) conditions [38, 45], which can be encoded in the so-called optimality function $\mathbf{F}$ from [4], see Eq. 10 Appendix C.2. The implicit function theorem [25] allows us to use implicit differentiation [3, 25, 44] to efficiently compute the Jacobians $\partial \mathbf{U}/\partial \mathbf{A}$ and $\partial \mathbf{W}/\partial \mathbf{A}$ without requiring to back-propagate through each of the iterations of the NMF solver:

$$\frac{\partial (U,W,\overline{U},\overline{W})}{\partial A}=-{({\partial }_{1}F)}^{-1}{\partial }_{2}F.$$ (4)

However, this requires the dual variables $\overline{\mathbf{U}}$ and $\overline{\mathbf{W}}$, which are not computed in scikit-learn’s [54] popular implementation. Consequently, we leverage the work of [32] and we re-implement our own solver with Jaxopt [4] based on ADMM [5], a GPU friendly algorithm (see Appendix C.2).

Concretely, given our concepts bank $\mathbf{W}$, the concept attribution maps of a new input $\mathit{x}$ are calculated by solving the NNLS problem ${\min }_{\mathbf{U}\ge 0}\frac{1}{2}{∥\mathit{g}(\mathit{x})-\mathbf{U}{\mathbf{W}}^{\mathrm{\top }}∥}_F^2$. The implicit differentiation of the NMF block $\partial \mathbf{U}/\partial \mathbf{A}$ is integrated into the classic back-propagation to obtain $\partial \mathbf{U}/\partial \mathit{x}$. Most interestingly, this technical advance enables the use of all white-box explainability methods [59, 62, 66, 69, 71, 78] to generate concept-wise attribution maps and trace the part of an image that triggered the detection of the concept by the network. Additionally, it is even possible to employ blackbox methods [14, 47, 55, 57] since it only amounts to solving an NNLS problem.

4. Experimental evaluation

In order to evaluate the interest and the benefits brought by CRAFT, we start in Section 4.1 by assessing the practical utility of the method on a human-centered benchmark composed of 3 XAI scenarios.

After demonstrating the usefulness of the method using these human experiments, we independently validate the 3 proposed ingredients. First, we provide evidence that recursivity allows refining concepts, making them more meaningful to humans using two additional human experiments in Section 4.2. Next, we evaluate our new Sobol estimator and show quantitatively that it provides a more faithful assessment of concept importance in Section 4.3. Finally, we run an ablation experiment that measures the interest of local explanations based on concept attribution maps coupled with global explanations. Additional experiments, including a sanity check and an example of deep dreams applied on the concept bank, as well as many other examples of local explanations for randomly picked images from ILSVRC2012, are included in Section B of the supplementary materials. We leave the discussion on the limitations of this method and on the broader impact in appendix A.

4.1. Utility Evaluation

As emphasized by Doshi-Velez et al. [11], the goal of XAI should be to develop methods that help a user better understand the behavior of deep neural network models. An instantiation of this idea was recently introduced by Colin & Fel et al. [8] who described an experimental framework to quantitatively measure the practical usefulness of explainability methods in real-world scenarios. For their initial setup, these authors recruited $n=\mathrm{1,150}$ online participants (evaluated over 8 unique conditions and 3 AI scenarios) – making it the largest benchmark to date in XAI. Here, we follow their framework rigorously to allow for the robust evaluation of the utility of our proposed CRAFT method and the related ACE. The 3 representative real-world scenarios are: (1) identifying bias in an AI system (using Husky vs Wolf dataset from [57]), (2) characterizing the visual strategy that are too difficult for an untrained non-expert human observer (using the Paleobotanical dataset from [75]), (3) understanding complex failure cases (using ImageNet “Red fox” vs “Kit fox” binary classification). Using this benchmark, we evaluate CRAFT, ACE, as well as CRAFT with only the global concepts (CRAFTCO) to allow for a fair comparison with ACE. To the best of our knowledge, we are the first to systematically evaluate concept-based methods against attribution methods.

Results are shown in Table 1 and demonstrate the benefit of CRAFT, which achieves higher scores than all of the attribution methods tested as well as ACE in the first two scenarios. To date, no method appears to exceed the baseline on the third scenario suggesting that additional work is required. We also note that, in the first two scenarios, CRAFTCO is one of the best-performing methods and it always outperforms ACE – meaning that even without the local explanation of the concept attribution maps, CRAFT largely outperforms ACE. Examples of concepts produced by CRAFT are shown in the Appendix E.1.

Table 1. Utility scores on 3 datasets from [8].

Their Utility benchmark evaluates how well explanations help users identify general rules driving classifications that readily transfer to unseen instances. At training time, users are asked to infer rules driving the decisions of the model given a set of images, and their associated predictions and explanations. At test time, the Utility metric measures the accuracy of users at predicting the model decision on novel images averaged over 3 sessions, and normalized by the baseline accuracy of users trained without explanations. The higher the Utility score, the more useful the explanation, and the more crucial the information provided is for understanding –and thus predicting the model’s output– on novel samples. CRAFTCO stands for “CRAFT Concept Only” and designates an experimental condition where only global concepts are given to users, without local explanations (i.e., the concept attribution maps). The first and second best results above the baseline are in bold and underlined, respectively.

		Husky vs. Wolf				Leaves				“Kit Fox” vs “Red Fox”
	Session n°	1	2	3	Utility	1	2	3	Utility	1	2	3	Utility
	Baseline	55.7	66.2	62.9		70.1	76.8	78.6		58.8	62.2	58.8
	Control	53.3	61.0	61.4	0.95	72.0	78.0	80.2	1.02	60.7	59.2	48.5	0.94
Attributions	Saliency [62]	53.9	69.6	73.3	1.06	83.2	88.7	82.4	1.13	61.7	60.2	58.2	1.00
	Integ.-Grad. [71]	67.4	72.8	73.2	1.15	82.5	82.5	85.3	1.11	59.4	58.3	58.3	0.98
	SmoothGrad [66]	68.7	75.3	78.0	1.20	83.0	85.7	86.3	1.13	50.3	55.0	61.4	0.93
	GradCAM [59]	77.6	85.7	84.1	1.34	81.9	83.5	82.4	1.10	54.4	52.5	54.1	0.90
	Occlusion [76]	71.0	75.7	78.1	1.22	78.8	86.1	82.9	1.10	51.0	60.2	55.1	0.92
	Grad.-Input [61]	65.8	63.3	67.9	1.06	76.5	82.9	79.5	1.05	50.0	57.6	62.6	0.95
Concepts	ACE [24]	68.8	71.4	72.7	1.15	79.8	73.8	82.1	1.05	48.4	46.5	46.1	0.78
	CRAFTCO (ours)	82.4	87.0	85.1	1.38	78.8	85.5	89.4	1.12	55.5	49.5	53.3	0.88
	CRAFT (ours)	90.6	97.3	95.5	1.53	86.2	86.6	85.5	1.15	56.5	50.6	49.4	0.87

4.2. Validation of Recursivity

To evaluate the meaningfulness of the extracted high-level concepts, we performed psychophysics experiments with human subjects, whom we asked to answer a survey in two phases. Furthermore, we distinguished two different audiences: on the one hand, experts in machine learning, and on the other hand, people with no particular knowledge of computer vision. Both groups of participants were volunteers and did not receive any monetary compensation. Some examples of the developed interface are available the appendix E. It is important to note that this experiment was carried out independently from the utility evaluation and thus it was setup differently.

Intruder detection experiment

First, we ask users to identify the intruder out of a series of five image crops belonging to a certain class, with the odd one being taken from a different concept but still from the same class. Then, we compare the results of this intruder detection with another intruder detection, this time, using a concept (e.g., ${𝒞}_1$) coming from a layer $l$ and one of its sub-concepts (e.g., ${𝒞}_{12}$ in Fig.3) extracted using our recursive method. If the concept (or sub-concept) is coherent, then it should be easy for the users to find the intruder. Table 2 summarizes our results, showing that indeed both concepts and sub-concepts are coherent, and that recursivity can lead to a slightly higher understanding of the generated concepts (significant for non-experts, but not for experts) and might suggest a way to make concepts more interpretable.

Table 2.

Results from the psychophysics experiments to validate the recursivity ingredient.

	Experts (n = 36)	Laymen (n = 37)
Intruder
Acc. Concept	70.19%	61.08%
Acc. Sub-Concept	74.81% (p = 0.18)	67.03% (p = 0.043)
Binary choice
Sub-Concept	76.1% (p < 0.001)	74.95% (p < 0.001)
Odds Ratios	3.53	2.99

Binary choice experiment

In order to test the improvement of coherence of the sub-concept generated by recursivity with respect to the larger parent concept, we showed participants an image crop belonging to both a subcluster and a parent cluster (e.g., $\mathit{\pi }(\mathit{x})\in {𝒞}_{11}\subset {𝒞}_1$) and asked them which of the two clusters (i.e., ${𝒞}_{11}$ or ${𝒞}_1$) seemed to accommodate the image the best. If our hypothesis is correct, then the concept refinement brought by recursivity should help form more coherent clusters. The results in Table 2 are satisfying since in both the expert and non-expert groups, the participants chose the sub-cluster more than 74% of the time. We measure the significance of our results by fitting a binomial logistic regression to our data, and we find that both groups are more likely to choose the sub-concept cluster (at a $p<0.001)$.

4.3. Fidelity analysis

We propose to simultaneously verify that identified concepts are faithful to the model and that the concept importance estimator performs better than that used in TCAV [40] by using the fidelity metrics introduced in [24, 77]. These metrics are similar to the ones used for attribution methods, which consist of studying the change of the logit score when removing/adding pixels considered important. Here, we do not introduce these perturbations in the pixel space but in the concept space: once $\mathbf{U}$ and $\mathbf{W}$ are computed, we reconstruct the matrix $\mathbf{A}\approx \mathbf{U}{\mathbf{W}}^{\mathrm{\top }}$ using only the most important concept (or removing the most important concept for deletion) and compute the resulting change in the output of the model. As can be seen from Fig. 6, ranking the extracted concepts using Sobol’s importance score results in steeper curves than when they are sorted by their TCAV scores. We confirm that these results generalize with other matrix factorization techniques (PCA, ICA, RCA) in Section F of the Appendix.

Figure 6.

(Left) Deletion curves (lower is better). (Right) Insertion curves (higher is better). For both the deletion or insertion metrics, Sobol indices lead to better estimates (calculated on $>100\mathrm{K}$ images) of important concepts.

5. Conclusion

In this paper, we introduced CRAFT, a method for automatically extracting human-interpretable concepts from deep networks. Our method aims to explain a pre-trained model’s decisions both on a per-class and per-image basis by highlighting both “what” the model saw and “where” it saw it – with complementary benefits. The approach relies on 3 novel ingredients: 1) a recursive formulation of concept discovery to identify the correct level of granularity for which individual concepts are understandable; 2) a novel method for measuring concept importance through Sobol indices to more accurately identify which concepts influence a model’s decision for a given class; and 3) the use of implicit differentiation methods to backpropagate through non-negative matrix factorization (NMF) blocks to allow the generation of concept-wise local explanations or concept attribution maps independently of the attribution method used. Using a recently introduced human-centered utility benchmark, we conducted psychophysics experiments to confirm the validity of the approach: and that the concepts identified by CRAFT are useful and meaningful to human experimenters. We hope that this work will guide further efforts in the search for concept-based explainability methods.

Figure 5. Qualitative Results:

CRAFT results on 6 classes of ILSVRC2012 [10] for a trained ResNet50V2. The results showcase the top 3 most important concepts for each class. This is done by displaying crop images that activate the concept the most (using $\mathbf{U}$) and also feature visualization [51] of the associated CAVs (using $\mathbf{W}$).