Efficient nonparametric statistical inference on population feature importance using Shapley values

Abstract

The true population-level importance of a variable in a prediction task provides useful knowledge about the underlying data-generating mechanism and can help in deciding which measurements to collect in subsequent experiments. Valid statistical inference on this importance is a key component in understanding the population of interest. We present a computationally efficient procedure for estimating and obtaining valid statistical inference on the Shapley Population Variable Importance Measure (SPVIM). Although the computational complexity of the true SPVIM scales exponentially with the number of variables, we propose an estimator based on randomly sampling only Θ(n) feature subsets given n observations. We prove that our estimator converges at an asymptotically optimal rate. Moreover, by deriving the asymptotic distribution of our estimator, we construct valid confidence intervals and hypothesis tests. Our procedure has good finite-sample performance in simulations, and for an in-hospital mortality prediction task produces similar variable importance estimates when different machine learning algorithms are applied.

1. Introduction

In many scientific applications, understanding the intrinsic predictive value of a variable can shed light on the internal mechanisms relating the variable to the outcome of interest, help build future models, and guide experimental design. For example, hospital administrators may want to know the important features to collect for predicting patient outcomes. Likewise, vaccine researchers may want to know the most important molecular phenotypes to measure that are most predictive of binding or vaccine efficacy (see, e.g., Dunning, 2006). Variable importance measures (VIMs) provide necessary information towards answering these questions.

Our interest here is in statistical inference on the population VIM. This VIM quantifies the predictive value of a variable within the oracle prediction model f₀ defined relative to an arbitrary predictiveness measure V. For many choices of V, f₀ is either the conditional mean outcome given covariates (e.g., if V = R²) or a simple functional of this conditional mean (e.g., if V = classification accuracy). We note that population VIMs are distinct from algorithmic VIMs, which describe the importance of a variable within a fitted model $\widehat{f}$ (see, e.g., Breiman, 2001; Garson, 1991; Murdoch et al., 2019). Although algorithmic VIMs have been used as a proxy for population VIMs out of convenience, differences between $\widehat{f}$ and f₀ can often lead to substantially different interpretations of the resulting VIMs. Whereas an algorithmic VIM necessarily varies across fitted models, a population VIM is independent of the specific procedure used to estimate f₀.

Existing population VIMs suffer from a number of issues. Traditionally, population VIMs have relied on restrictive parametric assumptions (e.g., R² in linear models; see, e.g., Grömping, 2007; Nathans et al., 2012), which can lead to misleading results if the parametric model does not hold. Recent work has focused on extending these definitions by removing the parametric assumptions (Feng et al., 2018; Williamson et al., 2020b); however, these definitions define importance of a variable with respect to the others and assign near-zero importance when features are highly correlated. Other VIMs require strong assumptions on the design to be valid (e.g., ANOVA), but again fail in simple cases with correlated variables. To address this, Owen and Prieur (2017) proposed using Shapley values to quantify the population VIM, where the value function is the variance explained; these VIMs inherit many desirable theoretical properties from the Shapley value. In fact, contemporary work has also defined the ideal estimand of algorithmic VIM estimation procedures to be the Shapley population VIM (SPVIM) (Covert et al., 2020).

Unfortunately, exact estimation of SPVIM is computationally intractable in general settings (Owen and Prieur, 2017): the SPVIM is defined as the sum of 2^p terms, where p is the number of features and each term depends on estimating the conditional mean function with respect to a unique feature subset. Previous approaches have either suggested sampling as many subsets as possible to estimate the Shapley value (see, e.g., Castro et al., 2009) or utilized special properties of tree estimators to reduce the number of subsets required (Lundberg et al., 2020). Notably, Štrumbelj and Kononenko (2014) analyzed the asymptotic distribution of a sampling-based estimator of Shapley algorithmic variable importance to derive confidence intervals.

In this paper, we combine the aforementioned developments and provide a nonparametric statistical inference procedure for SPVIM. We generalize previous definitions of SPVIM and use an arbitrary measure of predictiveness V. We tackle the computational complexity of the problem by randomly sampling feature subsets according to the Shapley value weights and then fitting corresponding models. We derive the asymptotic distribution of this sampling-based SPVIM estimator and show that the error from our proposed procedure can be decomposed into two components: the error from estimating the oracle prediction models and the error from omitting summands from the Shapley value estimand. Given n training observations, we find that our estimator only needs to sample m = Θ(n) subsets to converge at an asymptotically optimal rate. Moreover, since the subset sampling distribution is highly skewed, the number of unique feature subsets is much smaller than m in practice. We then use the asymptotic distribution to construct asymptotically unbiased point estimates, valid confidence intervals, and hypothesis tests with proper type I error control.

We demonstrate the validity of our approach in a simulation study and estimate the SPVIM of hospital measurements for predicting mortality in the intensive care unit (ICU). All numerical results can be replicated using code available on GitHub at bdwilliamson/spvim_supplementary; the proposed methods are also implemented in the Python package vimpy and the R package vimp.

2. Variable importance

2.1. Data structure and notation

Let $\mathcal{M}$ be a nonparametric class of joint distributions over covariates $X=\left(X_1,\dots ,X_p\right)\in \mathcal{X}\subseteq {ℝ}^p$ and response $Y\in \mathcal{Y}\subseteq ℝ$, where $\mathcal{X}$ and $\mathcal{Y}$ denote the sample spaces of X and Y, respectively. Suppose that each observation O consists of (X, Y). In this article, we consider observations O₁,…,O_n drawn independently according to a joint probability distribution $P_0\in \mathcal{M}$.

Next, we define the feature subsets and oracle prediction models of interest. We take $\mathcal{S}$ to be the power set of N := {1,…,p}. Let s_(j) for j = 1,…,2^p be an ordered sequence of the subsets in $\mathcal{S}$, where s₍₁₎ = ∅ and $s_{\left(2^p\right)}=N$. For any index set $s\in \mathcal{S}$, we denote by ${\mathcal{X}}_s$ and ${\mathcal{X}}_{-s}$ the sample spaces of X_s and X_−s, respectively. We denote by a_s and a_−s the elements of a vector a with indices in s and not in s, respectively. We also consider the binary vector $z(s)\in {ℝ}^{p+1}$ for each $s\in \mathcal{S}$, where z(s)₁ = 1 for all $s\in \mathcal{S}$ and z(s)_k+1 = I(k ∈ s) for k = 1,…,p. Finally, we consider a rich class $\mathcal{F}$ of functions from $\mathcal{X}$ to $\mathcal{Y}$ endowed with a norm $\Vert \cdot {\Vert }_{\mathcal{F}}$. For any $s\in \mathcal{S}$, we define the subset ${\mathcal{F}}_s:=\{f\in \mathcal{F}:f(u)=f(v)$ for all u, $v\in \mathcal{X}$ satisfying u_s = v_s} of functions in $\mathcal{F}$ whose evaluation ignores elements of the input x with index not in s. In all examples we consider, we take $\mathcal{F}$ to be a rich class of functions that is essentially unrestricted up to regularity conditions.

2.2. Oracle predictiveness

We define the importance of a variable at the population level in terms of its oracle predictiveness. This predictiveness is measured by a real-valued functional $V:\mathcal{F}\times \mathcal{M}\mapsto ℝ$. We assume that larger values of V (f, P) imply higher predictiveness. Examples of predictiveness measures — including R², deviance, area under the ROC curve, and classification accuracy — are provided in Williamson et al. (2020b).

The oracle predictiveness is the maximum achievable predictiveness over a class of prediction functions. More formally, we define the total oracle predictiveness $v_{0,N}\text{:=}{\max }_{f\in \mathcal{F}}V\left(f,P_0\right)$ and its associated oracle prediction function $f_{0,N}\in {\mathrm{argmax}}_{f\in \mathcal{F}}V\left(f,P_0\right)$. For many machine learning algorithms, f_0,N is the target of interest. We further define the oracle prediction function f_0,s that maximizes V(f, P₀) over all $f\in {\mathcal{F}}_s$; the marginal oracle predictiveness v_0,s := V (f_0,s, P₀) quantifies the prediction potential of features with index in s. The null oracle predictiveness $v_{0,\varnothing }:=V\left(f_{0,\varnothing },P_0\right)$ quantifies the prediction potential of a model that uses no covariate information. Finally, let $v_0:={\left[v_{0,\varnothing },v_{0,s_{(2)}},\dots ,v_{0,N}\right]}^{\top }$ denote the 2^p-dimensional vector of predictiveness measures for all subsets in $\mathcal{S}$. The predictiveness measure $v_{0,s_{(j)}}$ is defined relative to the population P₀, a joint distribution in the nonparametric statistical model $\mathcal{M}$; thus, its interpretation is tied to neither any particular estimation procedure nor any parametric assumptions.

2.3. The Shapley population variable importance measure

We now define a population VIM using the classical form of the Shapley value (see, e.g., Shapley, 1953; Charnes et al., 1988) with an arbitrary measure of predictiveness V. Specifically, the Shapley population variable importance measure (SPVIM) of the variable X_j is the average gain in oracle predictiveness from including feature X_j over all possible subsets:

$${\psi }_{0,0,j}:=\sum _{{}_{s\in N\backslash \left\{j\right\}}}\frac{1}{p}{\left(\begin{array}{c}p-1\\ \left|s\right|\end{array}\right)}^{-1}\left\{V\left(f_{0,s\cup j},P_0\right)-V\left(f_{0,s},P_0\right)\right\},$$ (1)

where the indices of ψ describe the number of subsets, the distribution P₀, and the feature of interest j, respectively. We use the index 0 to indicate that the SPVIM is computed using all subsets and the true distribution P₀. SPVIMs inherit the following properties from Shapley values (Shapley, 1953):

• Non-negativity: by construction, ${\psi }_{0,0,j}\ge 0$.

• Additivity¹: the sum of the SPVIMs across all variables is equal to the difference between the total and null oracle predictiveness,

  $$\sum _{j=1}^p{\psi }_{0,0,j}=v_{0,N}-v_{0,\varnothing }$$ (2)

• Symmetry: if X_i = X_j, then ${\psi }_{0,0,i}={\psi }_{0,0,j}$.

• Null feature: if X_j provides no added predictive value, i.e., $v_{0,s\cup j}=v_{0,s}$ for all $s\subseteq (N\backslash \{j\})$, then its SPVIM value is ${\psi }_{0,0,j}=0$.

• Linearity: if $\tilde{V}\equiv \alpha V$, then its associated SPVIM values are ${\tilde{\psi }}_{0,0,j}=\alpha {\psi }_{0,0,j}$ for all j ∈ {1,…,p}.

Because SPVIMs satisfy these properties, they clearly address the issue of correlated features: given collinear variables X_j and X_k that are each marginally predictive, previous nonparametric population VIMs (see, e.g., Williamson et al., 2020b) would assign zero importance to both variables whereas SPVIM would assign the same positive value to both variables.

In this paper, we take advantage of an alternate formulation of the Shapley value noted in previous work (see, e.g., Charnes et al., 1988; Lundberg and Lee, 2017). In particular, we can rewrite the weighted average in (1) as the solution of a weighted linear regression problem, where we treat the predictiveness of a feature subset v_0,s as the response and the subset membership z(s) as the covariates. Define a diagonal matrix of weights $W\in {ℝ}^{2^p\times 2^p}$ where W_1,1 = W₂p_,2p = 1, and for any $j\in 2,\dots ,2^p-1$, $W_{j,j}={\left(\begin{array}{c}p-2\\ \left|s_{(j)}\right|-1\end{array}\right)}^{-1}$. The matrix $Z\in {ℝ}^{2^p\times (p+1)}$ consists of the stacked z(s) vectors for each $s\in \mathcal{S}$. Setting ${\psi }_{0,0,\varnothing }:=v_{0,\varnothing }$, we denote by ψ_0,0 the (p+1)-dimensional vector of population Shapley values. Then (1) is equivalent to

$${\psi }_{0,0}:=\underset{\psi \in {ℝ}^{p+1}}{\mathrm{argmin}}{\Vert \sqrt{W}\left(Z\psi -v_0\right)\Vert }_2^2,$$ (3)

a result that we prove in the Supplement. If we define the distribution Q₀ over subsets $\mathcal{S}$ with probability mass function assigning weight ${\left(\begin{array}{c}p-2\\ |S|-1\end{array}\right)}^{-1}$ for $S\in \mathcal{S}\backslash \{\varnothing ,N\}$ and weight 1 for S ∈ {∅, N} (scaled so that the weights sum to one), then (3) is equivalent to a population average:

$${\psi }_{0,0}\equiv \underset{\psi \in {ℝ}^{p+1}}{\mathrm{argmin}}{E}_{Q_0}\left[{\left(z(S)\psi -v_{0,S}\right)}^2\right].$$

We will use this fact in our estimation procedure below.

3. Estimation and inference

3.1. Plug-in estimation

We now discuss how to estimate the SPVIM values for all p features using independent observations O₁,…,O_n drawn from P₀. Definition (3) suggests considering an estimator based on plugging in estimators of each individual component. We discuss each component in turn.

First, we estimate the predictiveness measure v_0,s = V(f_0,s, P₀) for a subset $s\in \mathcal{S}$ by plugging in estimates of the oracle function f_0,s and the distribution P₀. A simple approach is to partition the data into a training set and a validation set, construct an estimator f_n,s for f_0,s on the training data (using only the observed covariates in s), and estimate P₀ using the empirical distribution of the validation set P_V. Using this training-validation split, our estimate of predictiveness is then

$$v_{n,s}=V\left(f_{n,s},P_V\right).$$ (4)

An alternative approach is to perform K-fold cross-fitting, where we partition the data into K subsets of roughly equal size and, for each k ∈ {1,…,K}, construct an estimator f_k,n,s based on all the data except for the kth subset. Let P_k,n be the empirical distribution of the kth subset. Then we could estimate v_0,s using

$$v_{n,s}=\frac{1}{K}\sum _{k=1}^{K}V\left(f_{k,n,s},P_{k,n}\right).$$ (5)

If we had the entire estimated vector of predictiveness measures v_n, we could estimate ψ_0,0 using the plug-in estimator

$${\psi }_{0,n}:=\underset{\psi \in {ℝ}^{p+1}}{\mathrm{argmin}}{E}_{Q_0}[{(Z(S)\psi -v_{n,S})}^2].$$ (6)

Unfortunately, obtaining v_n requires training 2^p models, rendering this a computationally intractable task in general. Instead, we can replace Q₀ in (6) with an empirical distribution estimator Q_m obtained by sampling m subsets from $\mathcal{S}$ according to Q₀. This leads us to the SPVIM estimator ψ_m,n which solves the constrained least squares problem

$$\underset{\psi \in {ℝ}^{p+1}}{\min }{E}_{Q_m}[{\left(Z(S)\psi -v_{n,S}\right)}^2]\text{subject}\text{to}G\psi =c_n,$$ (7)

where $G:={[z{(\varnothing )}^{\top },z{(N)}^{\top }]}^{\top }\in {ℝ}^{2\times (p+1)}$ and $c_n\text{:=}{\left[v_{n,\varnothing },v_{n,N}\right]}^{\top }\in {ℝ}^2$. The constraint ensures that the estimated SPVIMs satisfy the additivity property (2) and that the estimated SPVIM for the null set is the estimated null predictiveness value.

This constrained least squares problem can be solved by forming a Lagrangian and inverting its Karush-Kuhn-Tucker (KKT) conditions (Boyd and Vandenberghe, 2004). More specifically, let s₁, …, s_ℓ be the unique subsets in Q_m. Let W_m be the ℓ × ℓ diagonal matrix where the kth diagonal element is the probability mass of s_k in Q_m. Let $v_{m,n}=\left(v_{n,s_1},\dots ,v_{n,s_{\ell }}\right)$ be the estimated predictiveness measures for the ℓ subsets. Let Z_m be the stack of vectors $z\left(s_1\right),\dots ,z\left(s_{\ell }\right)$. Then (7) can also be written as

$$\underset{\psi \in {ℝ}^{p+1}}{\min }{\Vert \sqrt{W_m}\left(Z_m\psi -v_{m,n}\right)\Vert }_2^2\text{subject}\text{to}G\psi =c_n.$$

Solving the KKT conditions with Lagrange multipliers denoted by λ, we obtain a closed-form SPVIM estimator:

$$\left[\begin{array}{c}{\psi }_{m,n}\\ \lambda \end{array}\right]={\left[\begin{array}{cc}2Z_m^{\top }{W}_m{Z}_m& G^{\top }\\ G& 0\end{array}\right]}^{-1}\left[\begin{array}{c}2\sqrt{W_m}{v}_{m,n}\\ c_n\end{array}\right].$$ (8)

To ensure that (7) has a unique solution, we select a sufficiently large value of m so that Q_m inclues at least p + 1 unique subsets. The full estimation procedure is given in Algorithm 1.

We now describe the properties listed in Section 2.3 that are satisfied by this sampling-based SPVIM estimator. It is easy to see that the additivity, symmetry, and linearity properties always hold. One possible concern is that the nonnegativity property can be violated. Nevertheless, in practice we find that negative SPVIM estimates are close to zero and the 95% confidence intervals cover zero. If nonnegativity is truly a concern, one can also add a nonnegative constraint to (7). Finally, the null feature property holds with respect to estimated predictiveness values and the sampled subsets. Note that this property is only relevant for discrete predictiveness measures like 0–1 classification accuracy, since the estimated predictiveness values are rarely exactly the same for continuous predictiveness measures like R².

The plug-in estimator ${\psi }_{m,n}$ is appealing due to its simplicity. In general, however, such an estimator may fail to be consistent at rate n^−1/2 if the population optimizers f_0,s are flexibly estimated. This phenomenon is due in large part to the optimal bias-variance tradeoff for estimating f_0,s differing in general from the optimal bias-variance tradeoff for estimating v_n,s. Plug-in estimators typically inherit much of the bias from estimating f_0,s, and this bias does not in general tend to zero sufficiently fast to allow n^−1/2-rate estimation of ${\psi }_{0,0}$ (Williamson et al., 2020a). In the next section, we extend the results of Williamson et al. (2020b) to describe conditions under which the estimator ${\psi }_{m,n}$ is asymptotically normal.

Algorithm 1.

Estimation of SPVIM

1: Input initial parameter γ ≥ 1.

2: Sample m = γn subsets from Q0, denoted s1, …, sm.

3: Estimate prediction functions fn,s for each s ∈ {s1, …, sm} ∪ {∅, N}.

4: Compute predictiveness estimates vn,s for s ∈ {s1, …, sm} ∪ {∅, N} using a training-validation split (see Equation (4)) or K-fold cross-fitting (see Equation (5)).

5: Solve for ψm,n using Equation (8).

3.2. Large-sample inferential properties

We now study the conditions under which ${\psi }_{m,n}$ is an asymptotically normal estimator of the SPVIM ${\psi }_{0,0}$. Using these conditions, we can design a procedure to construct confidence intervals and hypothesis tests. To do this, we decompose the error of our estimator ${\psi }_{m,n}$ into the following components:

$${\psi }_{m,n}-{\psi }_{0,0}=\left({\psi }_{0,n}-{\psi }_{0,0}\right)+\left({\psi }_{m,0}-{\psi }_{0,0}\right)+r_{m,n},$$ (9)

where ${\psi }_{m,0}$ is obtained by replacing v_n,S with v_0,S in (7) and $r_{m,n}:\text{=}({\psi }_{m,n}-{\psi }_{m,0})-({\psi }_{0,n}-{\psi }_{0,0})$. Each term on the right-hand side of (9) can then be studied separately to determine the large-sample behavior of ${\psi }_{m,n}$. The first term is the error of the estimator ${\psi }_{0,n}$ (6) constructed using prediction functions f_n,s estimated using n observations for all subsets s. The second term is the error of the estimator ${\psi }_{m,0}$ constructed using oracle prediction functions for sampled subsets in $Q_m$. In other words, the first term characterizes the error contribution from sampling training observations and the second term characterizes the error contribution from sampling subsets. The third term is a difference-in-differences remainder term that we prove to be negligible under some regularity conditions. Based on this decomposition, we will show that the asymptotic variance of $\sqrt{n}\left({\psi }_{m,n}-{\psi }_{0,0}\right)$ is simply the sum of the asymptotic variances of the first and second error terms.

Our result makes use of several conditions that require additional notation. These conditions were initially provided in Williamson et al. (2020b). We define the linear space $\mathcal{R}:=\left\{c\left(P_1-P_2\right):c\in ℝ,P_1,P_2\in \mathcal{M}\right\}$ of finite signed measures generated by $\mathcal{M}$. For any $R\in \mathcal{R}$, e.g., R = c(P₁ − P₂), we consider the supremum norm ∥R∥_∞ := |c|sup_o |F₁(o) − F₂(o)|, where F₁ and F₂ are the distribution functions corresponding to P₁ and P₂, respectively. Next, we define the following notation for each subset $s\in \mathcal{S}$. For distribution $P_{0,ϵ}\text{:=}{P}_0+ϵh$ with $ϵ\in ℝ$ and $h\in \mathcal{R}$, we define $f_{0,ϵ,s}=f_{P_{0,ϵ},s}$ to be its corresponding oracle prediction function with respect to subset s. Let $\dot{V}\left(f,P_0;h\right)$ denote the Gâteaux derivative of P ↦ V (f, P) at P₀ in the direction $h\in \mathcal{R}$, and define the random function $g_{n,s}:o\mapsto \dot{V}\left(f_{n,s},P_0;{\delta }_o-P_0\right)-\dot{V}\left(f_{0,s},P_0;{\delta }_o-P_0\right)$, where δ_o is the degenerate distribution on {o}. Consider the following set of deterministic [(A1)–(A4)] and stochastic [(B1)–(B2)] conditions for each subset $s\in \mathcal{S}$:

(A1) (optimality) there is some C > 0 such that for each sequence $f_1,f_2,\cdots \in {\mathcal{F}}_s$ with ${\Vert f_j-f_{0,s}\Vert }_{{\mathcal{F}}_s}\to 0$, there is a J such that for all $j>J,\mid V\left(f_j,P_0\right)-V\left(f_{0,s},P_0\right)\mid \le C{\Vert f_j-f_{0,s}\Vert }_{{\mathcal{F}}_s}^2$;

(A2) (differentiability) there is some δ > 0 such that for each sequence ${ϵ}_1,{ϵ}_2,\dots \in ℝ$ and $h,h_1,h_2,\dots \in \mathcal{R}$ satisfying that ϵ_j → 0 and ∥h_j − h∥_∞ → 0, it holds that

$$\begin{gathered} \underset{f\in {\mathcal{F}}_s:{\Vert f-f_{0,s}\Vert }_{{\mathcal{F}}_s}<\delta }{\sup }\mid \frac{V\left(f,P_0+{ϵ}_j{h}_j\right)-V\left(f,P_0\right)}{{ϵ}_j} \\ -\dot{V}\left(f,P_0;h_j\right)\mid \to 0; \end{gathered}$$

(A3) (optimizer continuity) ${\Vert f_{0,ϵ,s}-f_{0,s}\Vert }_{{\mathcal{F}}_s}=o(ϵ)$ for each $h\in \mathcal{R}$;

(A4) (derivative continuity) $f\mapsto \dot{V}\left(f,P_0;h\right)$ is continuous at f_0,s relative to ${\mathcal{F}}_s$ for each $h\in \mathcal{R}$;

(B1) (minimum rate of convergence) ${\Vert f_{n,s}-f_{0,s}\Vert }_{{\mathcal{F}}_s}=o_P\left(n^{-1/4}\right)$;

(B2) (weak consistency) $E_0[\int {\left\{g_{n,s}(o)\right\}}^{2}d{P}_0(o)]=o_P(1)$;

The Gâteaux derivative $\dot{V}$ is provided in Williamson et al. (2020b) for several common measures of predictiveness, including classification accuracy, AUC, and R². Assuming conditions (A1)–(A4) and (B1)–(B2) hold for every subset in $\mathcal{S}$, v_n is an asymptotically linear estimator of v₀ with influence function ${\dot{V}}_0:o\mapsto {\left[\dot{V}\left(f_{0,\varnothing },P_0;{\delta }_o-P_0\right),\dots ,\dot{V}\left(f_{0,N},P_0;{\delta }_o-P_0\right)\right]}^{\top }$ by Theorem 2 in Williamson et al. (2020b). Finally, we introduce a condition that specifies the number of subsets to sample:

(C1) (minimum number of subsets) For γ > 0 and sequence ${\gamma }_1,{\gamma }_2,\dots \in {ℝ}^{+}$ satisfying that |γ_j − γ| → 0, m = γ_nn.

For convenience, we define several objects that simplify the notation in our main result below. Set A := Z^⏉WZ, where Z is the stack of vectors z(s) for all $s\in \mathcal{S}$, and define C := A⁻¹G(G^⏉A⁻¹G)⁻¹. Let the QR decomposition of G^⏉ be

$$G^{\top }=\left[\begin{array}{ll}{U}_1\hfill & U_2\hfill \end{array}\right]\left[\begin{array}{l}R\hfill \\ 0\hfill \end{array}\right],$$

where R is an upper-triangular matrix. We define the functions

$$\begin{gathered} {\varphi }_{0,1}(O)=A^{-1}{Z}^{\top }\sqrt{W}{\dot{V}}_0(O)\text{and} \\ {\varphi }_{0,2}\left(S;v_0\right)=-U_2{V}^{-1}[z{(S)}^{\top }{\psi }_{0,0}-v_{0,S}]U_2^{\top }z(S), \end{gathered}$$

where V = U₂^⏉Z^⏉WZU₂. Assuming all of the aforementioned conditions hold, then ψ_m,n is a consistent and an asymptotically normal estimator of ψ_0,0.

Theorem 1.

If the collection of conditions implied by (A1)–(A4) and (B1)–(B2) hold for every subset in $\mathcal{S}$ and condition (C1) holds, then ψ_m,n has the asymptotic distribution

$$\begin{gathered} \sqrt{n}({\psi }_{m,n}-{\psi }_{0,0}){\to }_{d}N(0,{\mathrm{\Sigma }}_0), \\ \mathit{where}{\mathrm{\Sigma }}_0:={\mathrm{Cov}}_{P_0}\left({\varphi }_{0,1}(O)\right)+{\gamma }^{-1}{\mathrm{Cov}}_{Q_0}\left({\varphi }_{0,2}\left(S;v_0\right)\right). \end{gathered}$$

To construct Wald-based confidence intervals (CIs) for ψ_0,0, we estimate the asymptotic covariance Σ₀ by plugging in consistent estimators of each component. That is, we use consistent estimators A_m, Z_m, and W_m of A, Z, and W, respectively. Note that the estimators and CIs may be constructed using only the sampled subsets. If ψ_0,0,j = 0 for any j, then the contribution from sampling observations to the asymptotic covariance term corresponding to index j will be zero, leading to some additional complications. We discuss this case further in the next section.

Conditions (A1)–(A4) are required to control the contribution from estimating f_0,s for each $s\in \mathcal{S}$. Williamson et al. (2020b) show that these conditions are satisfied for R², deviance, accuracy, and AUC. Conditions (B1)–(B2) place restrictions on the class of estimators of f_0,s that we may consider. While condition (B1) holds for many estimators (e.g., generalized additive models (Hastie and Tibshirani, 1990)), we show in Section 5 that this condition may only need to be approximately satisfied. Condition (B2) is implied by a form of consistency of f_n,s.

Finally, condition (C1) is necessary to control the contribution from having had to estimate Q₀. Because ψ_0,n is an asymptotically efficient estimator of ψ_0,0, this condition implies that sampling m = Θ(n) subsets is asymptotically optimal, up to a constant factor proportional to γ⁻¹. Intuitively, this is because there is an irremovable error contribution from having sampled n training observations. As such, we simply need to sample enough subsets for the second error term in (9) to be on the same order as the first term. Moreover, because the distribution Q₀ places the heaviest weight on subset sizes at the extremes (closest in size to the empty set and full set), we do not need to estimate a large number of unique prediction functions in practice. To our knowledge, this is the first result that delineates the number of feature subsets to sample for constructing an asymptotically normal estimator of Shapley values.

3.3. Testing the null SPVIM hypothesis

We now use Theorem 1 to construct a test for the null hypothesis that a variable is not important, i.e., ψ_0,0,j = 0 for some j. When a variable X_j has null importance, the true value ψ_0,0,j is at the boundary of the parameter space, and the contribution to the asymptotic variance from sampling observations in Theorem 1 is zero. This may cause difficulties in hypothesis testing: as the number of sampled subsets grows, the contribution to the asymptotic variance from sampling subsets tends to zero. Thus, in the limit, a hypothesis test based on the estimator of this asymptotic variance proposed in the previous section will fail to appropriately control the type I error.

Instead, we rely on sample-splitting to construct a valid test of the δ-null hypothesis of the jth SPVIM value, i.e., H_0,j : ψ_0,0,j ∈ [0, δ]. In our approach, we make use of the fact that ψ_0,0,∅ may be nonzero for some predictiveness measures (e.g., AUC). Based on one portion of the data, construct estimator ψ_m,n,j,+ := ψ_m,n,j + ψ_m,n,∅ of ψ_0,0,j + ψ_0,0,∅ and obtain an estimator ${\sigma }_{n,j}^2$ of the variance ${\sigma }_{0,j}^2:={\left({\mathrm{\Sigma }}_0\right)}_{jj}$. Based on the remaining data, obtain an estimator ψ_m,n,∅,1 of ψ_0,0,∅ with corresponding variance estimator ${\sigma }_{n,\varnothing }^2$. Then, we calculate a test statistic $T_n:=\frac{({\psi }_{m,n,j,+}-{\psi }_{m,n,\varnothing ,1})-\delta }{\sqrt{n_1^{-1}{\sigma }_{n,j}^2+2\ast n_2^{-1}{\sigma }_{n,\varnothing }^2}}$ and its corresponding p-value p_n := 1 − Φ(T_n), where n₁ and n₂ denote the respective sample sizes of the split dataset and Φ denotes the standard normal cumulative distribution function. We reject H₀ if and only if p_n < α for some pre-specified level α. Under conditions (A1)–(A4), (B1)–(B2), and (C1), for any α ∈ (0,1), the proposed test is consistent and has type I error equal to α.

4. Local and group variable importance

Until now, we have focused on a global measure of importance by integrating over the entire distribution P₀. For certain settings, we may be interested instead in a local version of variable importance. A simple extension of (1) or (3) allows us to define a local version of variable importance: for a subpopulation $A\subseteq \mathcal{X}$,

$$\begin{array}{l}{\psi }_{0,0,j}(A):=\frac{1}{p}\sum _{s\in \mathcal{S}}{\left(\begin{array}{c}p-1\\ \left|s\right|\end{array}\right)}^{-1}\{V\left(f_{0,s\cup j},P_{0\mid X\in A}\right)\\ -V\left(f_{0,s},P_{0\mid X\in A}\right)\},\end{array}$$

where we have simply plugged the conditional distribution P_0|X∈A into (1). Taken to the extreme, where the subpopulation A consists only of a single observation, this definition of local feature importance is equivalent to the SHAP values considered by Lundberg and Lee (2017), though here we use an arbitrary measure of predictiveness in place of the conditional expectation. Unfortunately, valid statistical inference on this individual-observation-level importance appears difficult, if not impossible.

In addition, if there is some scientifically meaningful partition of the features, we can extend SPVIM to these feature subgroups. For example, one may group together all measurements from the same medical device. Let the partition of features into groups be denoted $\mathcal{P}:=\left\{s_1,\dots ,s_k\right\}$ where $s_i\in \mathcal{S}$ and ${\cup }_{i=1}^k{s}_i=N$, and $s_i\cap s_j=\varnothing$ for every (i, j) pair. Then the Shapley-based population variable group importance measure may be determined as in (1), where the sum is taken over all subsets in $\mathcal{P}$.

5. Simulation study

In this section, we present simulation results validating our statistical inference procedure for SPVIM in finite samples. We consider 200 covariates X ∼ N₂₀₀(0, Σ). The variance-covariance matrix Σ has diagonal equal to 1 and several correlated features: Cov(X₁, X₁₁) = 0.7; Cov(X₃, X₁₂) = Cov(X₃, X₁₃) = 0.3; and Cov(X₅, X₁₄) = 0.05. The covariance of the remaining feature pairs is zero. Based on these covariates, we observe a continuous outcome Y | X = x ∼ N(f(x), 1), where

$$f(x)=\sum _{j\in \{1,3,5\}}{f}_j\left(x_j\right),$$

$$f_1(x)=\mathrm{sign}(x),$$

$$\begin{gathered} f_3(x)=(-6)I(x\le -4)+(-4)I(-4<x\le -2) \\ +(-2)I(0\le x<-2)+2I(2<x\le 4) \\ +4I(x>4),\text{and} \end{gathered}$$

$$\begin{gathered} f_5(x)=(-1)I(x\le -4\text{or}-2<x\le 0\text{or}2<x\le 4) \\ +I(-4<x\le -2\text{or}0<x\le 2\text{or}x>4). \end{gathered}$$

In this data-generating mechanism, the vector (X₁, X₃, X₅) is directly relevant to predicting the outcome, while the vector (X₁₁, …, X₁₄) is only related to the outcome through correlation with (X₁, X₃, X₅); the remaining 193 features are pure noise. We generated 1,000 random datasets of size n ∈ {500, 1000, 2000, 3000, 4000}. The true SPVIM values for predictiveness defined in terms of R² are approximately (0.19, 0.29, 0.23, 0.04, 0.01, 0.01, 0) for the non-noise features, respectively, and zero for the remaining features.

To obtain each f_n,s we fit boosted trees (Friedman, 2001) using the Python package xgboost (Chen and Guestrin, 2016) with maximum tree depth equal to one, learning rate equal to 10⁻², and ℓ₂-regularization parameter equal to zero. The number of trees varied among {50, 100, 250, 500, 1000, …, 3000} and the ℓ₁-regularization parameter varied among {10⁻³, 10⁻², 0.1, 1, 5, 10}; the combination of these parameters was tuned using five-fold cross-validation to minimize the mean squared error (MSE).

We computed the relevant SPVIM estimator using Algorithm 1, where we sampled m = 2n subsets and estimated predictiveness using five-fold cross-fitting. For comparison, we computed the mean absolute SHAP value (Lundberg and Lee, 2017), where the average was taken over all observations. This allows us to directly evaluate the accuracy of algorithmic VIMs for estimating the population VIMs. We then computed the empirical MSE scaled by n, the empirical coverage of nominal 95% CIs, and the empirical power of our proposed hypothesis test. Finally, we compare the accuracy of our SPVIM estimates and the mean SHAP values in terms of their correlation with the true SPVIM values. All analyses were performed on a computer cluster with 32-core CPU nodes with 64 GB RAM.

We display the results of this experiment in Figure 1. We see that as n increases, the scaled empirical MSE of our estimator decreases to a fixed level — namely, the scaled empirical variance — for each feature. This matches our expectations from Section 3.2: the scaled empirical bias of our proposed estimator should tend to zero with increasing sample size, while the scaled empirical variance tends to the asymptotic variance. We note here that while boosted trees are a popular estimation procedure, they do not necessarily satisfy condition (B1) (see, e.g., Zhang and Yu, 2005). Thus, the convergence observed here provides some empirical evidence that condition (B1) may only need to hold approximately in practice. We also find that the coverage of nominal 95% confidence intervals increases to the nominal level as the sample size increases. Our proposed hypothesis test controls the type I error rate and is consistent: the empirical type I error rate is at the nominal level for null feature X₆, while the empirical power is near one for each of the directly important features. Power tends to be small for the indirectly important features (X₁₁, …, X₁₄), especially at small sample sizes; this reflects the fact that the importance of these features is closer to the null hypothesis than the importance of the directly relevant features. Finally, we see that SPVIM estimates are more correlated with the true population importance than SHAP values. We provide the estimated SPVIM and mean absolute SHAP values in the Supplement.

Figure 1.

Performance of our statistical inference procedure for estimating the Shapley-based population variable importance (SPVIM) with respect to R² using n training observations and 2n sampled subsets. (A, E) Empirical MSE for the proposed plug-in estimator scaled by n for j ∈ {1, 3, 5, 6} and j ∈ {11, 12, 13, 14}, respectively; (B, F) Empirical coverage of nominal 95% confidence intervals; (C, G) Empirical power of the hypothesis testing procedure for null hypothesis that the jth variable has null importance; (D) Kendall’s tau between the true and estimated SPVIM values using our approach versus the mean absolute SHAP value.

6. Predicting mortality of patients in theintensive care unit

We now analyze data on patients’ stays in the ICU from the Multiparameter Intelligent Monitoring in Intensive Care II (MIMIC-II) database (Silva et al., 2012). We consider 4000 records on several features: five general descriptors collected upon admission to the ICU, and 15 features — including the Glasgow Coma Scale (GCS), blood urea nitrogen (BUN), and heart rate — measured over the course of the first 48 hours after admission to the ICU. The outcome of interest is in-hospital mortality. Rather than use the entire time series, we simplify the analysis by first computing the minimum, average, and maximum value of each of the time-series features used in the simplified acute physiology (SAPS) I or II scores. The SAPS scores are established measures for estimating the mortality risk of ICU patients. We then remove any features that are measured in fewer than 70% of the patients. When combined with the general descriptor variables, a total of 37 extracted features remain. We provide a full list of these extracted features in the Supplement.

We estimate the SPVIM for each variable using AUC to measure predictiveness. For comparison, we also provide the mean absolute SHAP value obtained from Tree SHAP (Lundberg et al., 2020) and Kernel SHAP (Lundberg and Lee, 2017); and the proportion of times a feature was selected across test instances using LIME (Ribeiro et al., 2016). We discuss conditions under which the mean absolute SHAP value is a suitable proxy for the SPVIM in Section 2.2 in the Supplement.

We obtained estimates of each f_0,s using two separate procedures. In the first analysis, we maximized the empirical log likelihood using boosted trees with maximum depth equal to four, learning rate equal to 10⁻³, and a number of estimators in {2000, 4000, …, 12000} selected using five-fold cross-validation. In the second analysis, we maximized the empirical log likelihood by fitting ensembles of five dense ReLU neural networks (NNs) with architectures chosen from {(37, 25, 25, 20, 10, 1), (37, 25, 20, 1), (37, 25, 20, 20, 1)} using 5-fold cross-validation. The NNs were trained using Adam (Kingma and Ba, 2014) with a maximum of 2000 iterations and with ℓ₂ regularization parameter equal to 0.1. We again used 5-fold cross-fitting to estimate the predictiveness measures for the sampled subsets. Using our procedure, we fit models for only 119 unique subsets and computed SPVIM estimates in two hours for each analysis. LIME had similar computation time (1.7 hours) in the case of NNs, but longer computation time (4 hours) in the case of trees. The computation time of both our procedure and LIME falls between the highly specialized Tree SHAP algorithm, which completed in a few minutes, and the general-purpose Kernel SHAP, which took approximately 20 hours.

In Figure 2, we display the estimates from each VIM and both estimation procedures. We first focus on the SPVIM estimates provided in Panel A. The GCS is estimated to be the most important feature using both trees and NNs, though different summaries of the GCS are most important across the two procedures (mean for trees and max for NNs). This result matches prior knowledge: GCS is used to assess the level of consciousness of patients and is the highest scoring item in the SAPS scores. We find that the confidence intervals for SPVIM are quite wide, which is important information for placing the results in context.

Figure 2.

We estimated importance of features for predicting in-hospital death in the ICU using our statistical inference procedure for SPVIM with respect to AUC (A), the mean absolute SHAP value (B), and LIME (C). Gray circles and blue triangles denote estimates from fitting boosted trees and neural networks, respectively. The features are ordered from top to bottom by their point estimate from the neural networks procedure. 95% confidence intervals only appear in (A) since there is no statistical inference procedure for mean absolute SHAP values or LIME.

Next, we compare the agreement between rankings calculated based on the fitted boosted trees and NNs for the SPVIM estimates, mean absolute SHAP values (Figure 2 panel B), and LIME (panel C). There is considerably more agreement between the two procedures for the SPVIM estimates than for the SHAP value estimates and LIME proportions. The estimated Kendall’s tau between procedures is 0.71 for our SPVIM estimator vs 0.37 for SHAP and 0.39 for LIME. Given the large discrepancies between the algorithmic VIMs, we conclude that they are poor proxies for our population VIM. Instead, one should use a procedure specifically designed to estimate SPVIM.

Finally, we find that the feature rankings within trees or NNs from our SPVIM estimator, SHAP, and LIME are substantively different. One noticeable difference is that SHAP and LIME values for several summary statistics derived from the same measurement (e.g., min, mean, and max GCS) differ widely; this should not occur, since these summary statistics are highly correlated. On the other hand, SPVIM estimates for summary statistics derived from the same measurement tend to be more similar.

7. Discussion

We have proposed a computationally tractable statistical inference procedure for the Shapley population variable importance measure (SPVIM). Methods for estimating SPVIM are complementary to those for estimating algorithmic variable importance. The former helps us understand the underlying data-generating mechanism and can help guide future experiments; the latter helps us interpret a particular fitted model. Here, we define SPVIM with respect to an arbitrary measure of predictiveness, allowing the data analyst to select the most appropriate measure for the task at hand. Since the SPVIM is also defined relative to the population, the target of inference is not affected by the choice of prediction algorithm. We have derived the asymptotic distribution of an SPVIM estimator based on randomly sampled feature subsets, and have used this distribution to construct asymptotically normal point estimates, valid confidence intervals, and hypothesis tests with the correct type I error control. Notably, we determined a minimum number of feature subsets to sample: we show that our estimator only needs to fit prediction models for m = Θ(n) sampled subsets for its error to be on the same order as an estimator that fits prediction models for all possible subsets.

In this manuscript, we have focused on quantifying the importance of a variable averaging across the entire population. Local importance measures can be obtained by restricting to smaller subpopulations. However, as the subpopulations decrease in size, the uncertainty of our estimates increases. Our asymptotic results do not apply to the most extreme case, the variable importance at the level of a single observation. Nevertheless, this value may be of interest in some tasks. Further work is necessary to define relevant importance measures at the single-observation-level and derive procedures with the desired performance.

Finally, we caution against interpreting SPVIM estimates in a causal manner. SPVIM reflects importance in the oracle prediction model rather than importance in the oracle causal model. In many scientific applications, the importance in the causal model is of ultimate interest. To get causal importance, one may need to employ techniques from causal inference. Recent developments relating prediction models and causal models may also be of use in these cases (Arjovsky et al., 2019).