Sensitivity analysis of unmeasured confounding in causal inference based on exponential tilting and super learner

Abstract

Causal inference under the potential outcome framework relies on the strongly ignorable treatment assumption. This assumption is usually questionable in observational studies, and the unmeasured confounding is one of the fundamental challenges in causal inference. To this end, we propose a new sensitivity analysis method to evaluate the impact of the unmeasured confounder by leveraging ideas of doubly robust estimators, the exponential tilt method, and the super learner algorithm. Compared to other existing methods of sensitivity analysis that parameterize the unmeasured confounder as a latent variable in the working models, the exponential tilting method does not impose any restrictions on the structure or models of the unmeasured confounders. In addition, in order to reduce the modeling bias of traditional parametric methods, we propose incorporating the super learner machine learning algorithm to perform nonparametric model estimation and the corresponding sensitivity analysis. Furthermore, most existing sensitivity analysis methods require multivariate sensitivity parameters, which make its choice difficult and subjective in practice. In comparison, the new method has a univariate sensitivity parameter with a nice and simple interpretation of log-odds ratios for binary outcomes, which makes its choice and the application of the new sensitivity analysis method very easy for practitioners.

1. Introduction

In order to explore causal inference, Neyman [19] proposed the potential outcome framework in the context of completely randomized experiments and later Rubin [29] extended it to both observational and experimental studies. Consider a binary treatment T with T = 1 indicating the treatment group and T = 0 indicating the control group. The potential outcome $Y(t)$ is defined as the outcome we would observe if a subject had been assigned the treatment T = t, t = 0, 1. The observed outcome will then be $Y=TY(1)+(1-T)Y(0)$. The causal effect, or average treatment effect (ATE), of the treatment is defined as $\tau ={\mu }_1-{\mu }_0$, where ${\mu }_t=E\{Y(t)\},t=0,1$. In observational studies, a standard assumption for causal inference is the assumption of strongly ignorable treatment assignment. Let X be a collection of measured baseline covariates which are considered possible confounders. The strong ignorability assumption states that the treatment assignment T is conditionally independent of the potential outcomes $Y(0)$ and $Y(1)$ given X, i.e.

$$Y(t)\perp T|X,t=0,1.$$ (1)

In other words, within each level of X, T is assumed to be independent from the potential outcomes, so we have a quasi-randomized experiment.

Researchers usually hope that sufficiently rich baseline information is collected so that the ignorability assumption (1) is reasonable. However, this assumption is usually questionable in observational studies, and the unmeasured confounding is one of the fundamental challenges in causal inference. If unmeasured confounders exist, the strong ignorability assumption is violated, which may result in a biased treatment effect estimation and undermine the validity and credibility of the corresponding causal inference. Unfortunately, the strong ignorability assumption cannot be validated with observed data for the estimation of causal effect, and there is frequently insufficient background knowledge to justify this assumption.

Without the strong ignorability assumption (1), the causal effect becomes unidentified, which has motivated alternative assumptions of various forms that aim to recover the identifiability. However, these alternative assumptions are also untestable with observed data like the strong ignorability assumption and may be as questionable as, or even more questionable than, the strong ignorability assumption. To address such uncertainty, it is important to conduct a sensitivity analysis that considers a variety of identifiability assumptions and compare results obtained under different assumptions.

The history of sensitivity analysis can be dated back to the work of Cornfield et al. [4] which explored a causal link between smoking and lung cancer. Rosenbaum and Rubin [24] proposed a sensitivity analysis method that explicitly adjusts for an unmeasured confounder as a latent variable. With U denoting the unmeasured confounder, their method assumes that X and U together satisfy assumption (1), i.e. $Y(t)\perp T|X,U$. In addition, their method depends on well-matched sets based on propensity scores by assuming that X is finitely discrete with both Y and U being binary. Later Imbens [12] extended Rosenbaum and Rubin [24] to allow for non-binary outcomes. As mentioned in Imbens [12], one of the key points is that a parametric model is postulated. More specifically, the observed data consists of $(Y,T,X)$ and a binary unmeasured confounder U is assumed for each individual. Suppose that the observed data and the unmeasured confounder were generated according to:

$$\begin{array}{rl}& U_i\sim \mathrm{B}\mathrm{e}\mathrm{r}\left(P_u\right)T_i|X_i,U_i\sim \mathrm{B}\mathrm{e}\mathrm{r}\left(\mathrm{e}\mathrm{x}\mathrm{p}\mathrm{i}\mathrm{t}\{{\gamma }_{x}X+{\gamma }_{u}U\}\right)\\ & Y_i|T_i,X_i,U_i\sim \mathrm{B}\mathrm{e}\mathrm{r}\left(\mathrm{e}\mathrm{x}\mathrm{p}\mathrm{i}\mathrm{t}\{{\mathrm{\text{β}}}_{t}T+{\mathrm{\text{β}}}_{x}X+{\mathrm{\text{β}}}_{u}U\}\right),\end{array}$$

where $\mathrm{Ber}(\pi )$ is a Bernoulli distribution with a success probability π, $\mathrm{expit}(s)=(1+e^{-s}{)}^{-1}=e^s/(1+e^s)$. If we could observe U, the parameters $(P_u,{\gamma }_x,{\gamma }_u,{\beta }_t,{\beta }_x,{\beta }_u)$ can be estimated via logistic regression models and report τ as the causal effect. The two coefficients ${\gamma }_u$ and ${\beta }_u$ measure the strength of the association between the unmeasured confounder and the treatment and the association between the unmeasured confounder and the outcome, respectively. However, U is not observed, thus $P_u$, ${\gamma }_u$ and ${\beta }_u$ are not identifiable using the observed data. What analysts can do is to specify plausible values of $(P_u,{\gamma }_u,{\beta }_u)$ based on their subjective judgments about these parameters, then the other parameters can be estimated based on $(P_u,{\gamma }_u,{\beta }_u)$. The final estimate of the causal effect can be expressed as $\hat{\tau }(P_u,{\gamma }_u,{\beta }_u)$. Veitch and Zaveri [36] pointed out that this approach has a major drawback: it relies on a parametric model for the full data generating process. Using the assumed model is equivalent to assuming that if U had been observed, it would have been appropriate to use logistic regression to model the treatment assignment and outcome. Also, the sensitivity analysis result depends on the distribution of U. Furthermore, the choice of the sensitive parameters $(P_u,{\gamma }_u,{\beta }_u)$ is subjective and challenging and could be even more difficult if U is not univariate. Lin et al. [15] proposed an approach to parameterize the unmeasured confounder as the bias of regression coefficients in an outcome regression model adjusting for both the measured and unmeasured confounders. There is also a line of work that formulated the sensitivity parameter as the bias of the outcome regression model caused by the unmeasured confounder, see, for example, Hogan et al. [11], Jung et al. [13], Roy et al. [28], etc. Ding and VanderWeele [6], VanderWeele and Ding [35] investigated how strong an unmeasured confounder has to be in order to qualitatively change the conclusion of a causal analysis. In recent years, another extension of the traditional sensitivity analysis was from a Bayesian perspective which used the average over the distribution of sensitivity parameters rather than varying the sensitivity parameters [7,17,18]. For sensitivity analysis methods that require specifying some parametric models for the confounding variable, their required assumptions may be difficult to hold in practice. In addition, they usually contain multivariate sensitivity parameters, which makes their choice difficult and subjective when performing sensitivity analysis in practice.

To this end, we propose a new sensitivity analysis method to evaluate the impact of the unmeasured confounder by leveraging ideas of doubly robust estimators [22], the exponential tilt method [30], and the super learner machine learning method [33]. Note that if there is any unmeasured confounder, $f(Y(t)|X,T=1)$ and $f(Y(t)|X,T=0)$ will be different, where f is a probability distribution function (either a density function for a continuous variable or a probability mass function for a discrete variable). Inspired by this, we propose to assess the sensitivity of the difference between the conditional distribution of the observed potential outcome given covariates and that of the counterfactual potential outcome via the exponential tilting method proposed by Scharfstein et al. [30]. Compared to most existing sensitivity analysis methods, the exponential tilting method does not directly impose any assumptions on the distribution of the unmeasured confounders. Therefore, the new method has the flexibility to allow the unmeasured confounder to be continuous, binary, or categorical, and be univariate or multivariate. In order to reduce the modeling bias of traditional parametric methods, we propose incorporating super learner machine learning algorithms to perform the nonparametric model estimation and the corresponding sensitivity analysis. Super learner algorithms aim to optimally combine many machine learning algorithms together to provide a better estimation than any individual candidate machine learning algorithm. Unlike most of the existing sensitivity analysis methods which usually contain multiple sensitivity parameters, the new method has a univariate sensitivity parameter, which directly measures the bias of the strong ignorability assumption (1) caused by the unmeasured confounding, and has a nice and simple interpretation of log-odds ratios for binary outcomes. The used univariate sensitivity parameter also makes its choice and the application of the new sensitivity analysis method very easy for practitioners.

The rest of this paper is structured as follows. In Section 2, we introduce our proposed estimation method in detail. In Sections 3 and 4, we present numerical examples based on both a simulation study and a real data application. A conclusion is given in Section 5.

2. Methodology

In this article, we mainly focus on the binary response variable due to its wide variety of applications in causal inference [8,18,24]. The proposed method can be easily extended to a continuous response variable. Note that for a binary response variable, the potential outcome mean ${\mu }_t=E\{Y(t)\}$ can be interpreted as the success rate among the target population if everyone had been assigned the treatment t. For simplicity of explanation, we introduce our new method by focusing on estimating ${\mu }_1=E\{Y(1)\}$, the mean of potential outcome if everyone in the target population had been treated. The ${\mu }_0=E\{Y(0)\}$ and the corresponding ATE $\tau ={\mu }_1-{\mu }_0$ can be similarly estimated.

When the subjects are in the treatment group, $Y(1)$ is the observed actual outcome, whereas when the subjects are in the control group, $Y(1)$ becomes the counterfactual outcome which cannot be observed. So the estimation of ${\mu }_1$ essentially boils down to the imputation of counterfactual outcomes. Let

$$m(x)=E(Y(1)|X=x),m_1(x)=E(Y(1)|T=1,X=x)$$

and

$$P(x,y)=P(T=1|X=x,Y(1)=y),q(x)=P(T=1|X=x).$$

Then our target parameter can be also written as ${\mu }_1=\mathrm{E}\{Y(1)\}=\mathrm{E}\{m(X)\}.$ Notice that when there is no unmeasured confounder,

$$m(x)=m_1(x)\mathrm{and}P(x,y)=q(x),$$ (2)

which can be estimated from the observed data.

2.1. Doubly robust estimator

Let $(X_i,Y_i,T_i{)}_{i=1}^n$ be an independent and identically distributed random sample from $X\in {\mathbb{R}}^d$, $Y\in \{0,1\}$ and $T\in \{0,1\}$, where X is a set of baseline covariates, and T is a binary treatment indicator. Under the strong ignorability assumption (i.e. there is no unmeasured confounder), three of the most commonly used estimators for ${\mu }_1=E\{Y(1)\}$ are

$$\begin{array}{rl}{\hat{\mu }}^{OR}& =\frac{1}{n}\sum _{i=1}^n{\hat{m}}_1(x_i),\end{array}$$ (3)

$$\begin{array}{rl}{\hat{\mu }}^{IPW}& =\frac{1}{n}\sum _{i=1}^n\frac{T_i{Y}_i}{\hat{q}(x_i)},\end{array}$$ (4)

and

$${\hat{\mu }}^{DR}=\frac{1}{n}\sum _{i=1}^n\{\frac{T_i{Y}_i}{\hat{q}(x_i)}-\frac{T_i-\hat{q}(x_i)}{\hat{q}(x_i)}{\hat{m}}_1(x_i)\},$$ (5)

where $\widehat{m_1}(x)$ and $\hat{q}(x)$ are the estimates of $m_1(x)$ and $q(x)$, respectively. Estimator (3) only involves an outcome regression (OR) model, $m_1(x)$, so it is referred to as an OR estimator [32]. Estimator (4) is called an IPW estimator [16,21,23], since it includes a model for the propensity score [25, PS], $q(x)=P(T=1|X=x)$, and uses the inverse probability weight (IPW). Estimator (5) is a doubly robust estimator of the potential outcome mean [16,22,26,27,31, DR]. It involves both OR and PS models. Notice that the DR estimator is an augmented form of the IPW estimator:

$${\hat{\mu }}^{DR}={\hat{\mu }}^{IPW}-\frac{1}{n}\sum _{i=1}^n\frac{T_i-\hat{q}(x_i)}{\hat{q}(x_i)}{\hat{m}}_1(x_i).$$

The consistency of the resulting estimators OR, IPW, and DR depends on the correct specification of the relevant models for $q(x)$ and $m_1(x)$. However, unless the parametric models for $q(x)$ and $m_1(x)$ are correctly specified, we cannot expect OR or IPW estimator to be consistent. It is worth noting that the validity of the DR estimator entails weaker conditions, since it only requires either $m_1(x)$ or $q(x)$ to be correctly specified. Because of this, we will adopt the DR method for the proposed sensitivity analysis.

If there exists any unmeasured confounder, then $m(x)\ne m_1(x)$ and $P(x,y)\ne q(x)$, and the traditional estimators of ${\mu }_1$ in (3)—(5) will be biased. To incorporate the unmeasured confounder to correct the estimation bias, we propose the following modified doubly robust estimator for ${\mu }_1$ using the estimators of $m(x)$ and $P(x,y)$ instead of $m_1(x)$ and $q(x)$, respectively,

$${\hat{\mu }}^{DR}=\frac{1}{n}\sum _{i=1}^n\{\frac{T_i{Y}_i}{\hat{P}(x_i,y_i)}-\frac{T_i-\hat{P}(x_i,y_i)}{\hat{P}(x_i,y_i)}\hat{m}(x_i)\}.$$ (6)

The difficulty of the estimation for $m(x)$ and $P(x,y)$ lies in the fact that the outcome $Y(1)$ is only partially observed.

2.2. Exponential tilt method for the unmeasured confounder

When there are unmeasured confounders, the conditional distribution of the observed $Y(1)$ is no longer the same as the conditional distribution of the unobserved $Y(1)$ given the covariates, and thus the relationship in equation (2) breaks down. Our goal is to restore the relationship between the conditional distributions of the observed and the unobserved outcome so that we can further estimate $m(x)$ and $P(x,y)$ in (6).

Let $g(y|x)$ be the conditional distribution of $Y(1)|X$ and $g_t(y|x)$ be the conditional distribution of $Y(1)|X,T=t,t=0,1$. Notice that when T = 1, $Y(1)$ is observed, and when T = 0, $Y(1)$ is missing. We propose leveraging the exponential tilt method from Scharfstein et al. [30] to build the connection between $g_0$ and $g_1$,

$$g_0(y|x)=\frac{g_1(y|x)e^{\alpha y}}{E\{e^{\alpha Y}|x,T=1\}},$$ (7)

which links the distribution of unobserved outcomes to the the distribution of observed outcomes. The denominator in (7) is a normalization constant to make $g_0$ a legitimate density. Since the exponential tilt method (7) does not require specifying parametric models for latent confounding variables, the unmeasured confounder could be continuous, binary, or categorical, and could be univariate or multivariate. The parameter α is a univariate sensitivity parameter and non-identifiable with the data and governs the departure of the truth from the strong ignorability assumption that ‘no unmeasured confounder exists ’. When $\alpha =0$, $g_0(y|x)=g_1(y|x)$, which indicates there is no unmeasured confounding. When $\alpha \ne 0$, $g_0(y|x)\ne g_1(y|x)$ and hence some unmeasured confounders exist. It can be derived from (7) that for a binary outcome, if

$$\begin{array}{rl}& [Y(1)|T=1,X=x]\sim \mathrm{Ber}(\pi (X)),\\ & \mathrm{then}[Y(1)|T=0,X=x]\sim \mathrm{Ber}\left(\frac{\pi (X)e^{\alpha }}{\pi (X)e^{\alpha }+1-\pi (X)}\right).\end{array}$$ (8)

Equation (8) also indicates that

$$\mathrm{logit}\{P(Y(1)=1|T=0,X=x)-\mathrm{logit}\{P(Y(1)=1|T=1,X=x)\}=\alpha ,$$

where $\mathrm{logit}(s)=\log \frac{s}{1-s}$. Therefore, for binary response variables, $e^{\alpha }$ is the conditional odds ratio (and hence α is the log odds ratio) of the unobserved potential outcome and the observed potential outcome being 1 after adjusting for the measured confounding variables X; the sensitivity parameter α measures the violation of the strong ignorability assumption (1) caused by the unmeasured confounding.

Figure 1 demonstrates how the conditional log-odds ratio relates to the underlying probabilities of the unobserved and observed potential outcome being 1 as the sensitivity parameter α varying from $-5$ to 5. When there is no unmeasured confounder, $P(Y(1)=1|T=0,X=x)=P(Y(1)=1|T=1,X=x)$ and thus the log odds ratio is $\alpha =0$. If $\alpha >0$, it implies that the probability of the unobserved potential outcome $Y(1)$ (given T = 0) being 1 is larger than that of the observed potential outcome (given T = 1) being 1 after adjusting for the measured confounding variables X. When $\alpha <0$, the relationship is reversed, i.e. the probability of the unobserved potential outcome $Y(1)$ (given T = 0) being 1 is smaller than that of the observed potential outcome (given T = 1) being 1 conditional on any X.

Figure 1.

The contour plot of log-odds ratio

For example, if Y = 1 indicates a certain disease being cured, then $e^{\alpha }$ is the conditional odds ratio of being cured between the control patients had they been treated and the treated patient after adjusting for the measured covariates. A value of $\alpha =1$ implies an odds ratio of $e^1=2.72$, meaning that the odds of being cured for the control patients had they been treated is almost 2.72 times as that of the treated patients after adjusting for measured covariates. The choice of α in practice relies on some subject-matter guidance, such as experts' experience and prior knowledge that is experiment-specific. In practice, usually, the α value from $-2$ to 2 (with the corresponding odds ratio from $e^{-2}=0.14$ to $e^2=7.39$) or even $-1$ to 1 (with the odds ratio from 0.37 to 2.72) is a reasonable choice for a sensitivity analysis.

As mentioned earlier, the exponential tilt method of (7) can be also applied to the continuous outcomes easily. For example, based on the assumption of (7), if $[Y(1)|T=1,X=x]\sim N(\eta ,{\sigma }^2)$, then $[Y(1)|T=0,X=x]\sim N(\eta +\alpha {\sigma }^2,{\sigma }^2)$ for the unobserved outcome variable. Note that σ can be estimated based on the observed outcome. If the exponential tilt method is applied to the standardized data (i.e. $\sigma =1$), then the sensitivity parameter α is exactly equal to the mean difference/shift between the unobserved potential outcome from the control arm and the observed potential outcome from the treatment arm.

To apply the proposed method (6), we need to estimate $m(x)$ and $P(x,y)$. Note that $g_1(y|x)$ and $q(x)$ can be estimated from observed data. We introduce a super learner machine learning estimation method for estimating $g_1(y|x)$ and $q(x)$ in Section 2.3. The $m(x)$ can then be estimated by $\hat{m}(x)=\hat{g}(1|x)$, where

$$\hat{g}(y|x)=\hat{q}(x){\hat{g}}_1(y|x)+[1-\hat{q}(x)]{\hat{g}}_0(y|x),$$

and $g_0(y|x)$ can be estimated based on $g_1(y|x)$ according to equation (7). To estimate $P(x,y)$, note that

$$\begin{array}{rl}P(x,y)& =\frac{g_1(y|x)q(x)}{g_1(y|x)q(x)+g_0(y|x)\{1-q(x)\}}\\ & =\frac{g_1(y|x)q(x)}{g_1(y|x)q(x)+\frac{g_1(y|x)e^{\alpha y}}{E(e^{\alpha Y}|x,T=1)}\{1-q(x)\}}\\ & =\mathrm{expit}[\mathrm{logit}\{q(x)\}-\alpha y+\log \{e^{\alpha }{g}_1(1|x)+g_1(0|x)\}],\end{array}$$ (9)

Then, $P(x,y)$ can be estimated by plugging the estimates to equation (9) as

$$\hat{P}(x,y)=\mathrm{expit}[\mathrm{logit}\{\hat{q}(x)\}-\alpha y+\log \{e^{\alpha }{\hat{g}}_1(1|x)+{\hat{g}}_1(0|x)\}],$$ (10)

and the difference between the modified DR estimator (6) and the standard DR estimator (5) is the bias caused by unmeasured confounding.

2.3. Super learner machine learning estimators

One way to estimate $g_1(x)$ and $q(x)$ is to specify some parametric models for them, for example, a linear or a logistic regression model for $g_1(x)$ for a continuous or binary response, respectively, and a logistic regression model for $q(x)$. However, if there are some violations of the parametric assumptions, which is usually the case in practice, it would aggravate the bias in the causal inference even for a doubly robust estimator.

To this end, we propose employing a super learner machine learning algorithm to nonparametrically estimate $g_1(x)$ and $q(x)$. Super learner is a general loss-based learning algorithm that was proposed and analyzed theoretically by Van der Laan et al. [33]. The algorithm optimally combines a library of machine learners by minimizing the cross-validation error and is aimed to estimate the regression function flexibly without over-fitting the data [34]. To illustrate this learning process, let the observed data be $(Y_i,X_i{)}_{i=1}^n,$ where Y is the outcome and X is a set of covariates with dimension d. The super learner algorithm aims to estimate $m(x)=E(Y|X=x)$ using a library of machine learners $m_1,\dots m_K$ weighted by ${\lambda }_1,\dots ,{\lambda }_K$, such that $\hat{m}(x)=\sum _{k=1}^K{\lambda }_k{\hat{m}}_k(X)$, where ${\hat{m}}_k$ is the estimator of $m(x)$ based on $k^{th}$ machine learner. The selection of a library of machine learners will be discussed in Section 3. The weight vector $\lambda =({\lambda }_1,\dots ,{\lambda }_K)$ can be chosen by the following cross-validation procedure:

Randomly split the sample $(Y_i,X_i{)}_{i=1}^n$ into J equally sized subsets. For each $j\in (1,\dots ,J)$, the $j^{th}$ subset, denoted by $S_j$, is used as a validation set and the other subsets are used as the training sets. Let ${\hat{m}}_k^{(-j)}$ be the estimator of $m(x)$ using the $k^{th}$ machine learner based on the training data without the $j^{th}$ subset $S_j$. Then we can find the weight vector by

$$({\hat{\lambda }}_1,\dots ,{\hat{\lambda }}_K)=\underset{({\lambda }_1,\dots ,{\lambda }_K)}{\arg min}\sum _{j=1}^J\sum _{(X_i,Y_i)\in S_j}{\{Y_i-\sum _{k=1}^K{\lambda }_k{\hat{m}}_k^{(-j)}(X_i)\}}^2.$$

Polley et al. [20] suggested bounding ${\lambda }_k$ and using the constraints $\sum _{k=1}^K{\lambda }_k=1,{\lambda }_k\ge 0,\mathrm{\forall }k$. A non-negative binomial likelihood maximization or maximizing AUC (Area Under The Curve) of ROC (Receiver Operating Characteristics) curve can also be used as a cross-validation criteria for binary outcomes.

2.4. New sensitivity analysis method

Below we summarize our proposed sensitivity analysis by combining the ideas of the doubly robust estimator, the exponential tilt method, and the super learner machine learning method introduced in Sections 2.1 to 2.3, respectively.

• Step 1:

  Train the super learner method for $q(x)$ based on the data $(T_i,X_i)$ using T as the response variable and X as the independent variable to obtain the estimate $\hat{q}(x)$.

• Step 2:

  Train the super learner method for $g_1(x)$ based on the subset of the data $\{(X_i,Y_i),T_i=1,i=1,\dots ,n\}$ using Y as the response variable and X as the independent variable to obtain the estimate ${\hat{g}}_1(x)$.

• Step 3:
  For a given value of the sensitivity parameter α, calculate:

  $$\begin{array}{rl}\widehat{g_0}(y|x)& =\frac{e^{\alpha y}{\hat{g}}_1(y|x)}{e^{\alpha }{\hat{g}}_1(1|x)+{\hat{g}}_1(0|x)},\\ \hat{g}(y|x)& =\hat{q}(x){\hat{g}}_1(y|x)+\{1-\hat{q}(x)\}{\hat{g}}_0(y|x),\\ \hat{m}(x)& =\hat{g}(1|x),\end{array}$$

  and

  $$\hat{P}(x,y)=\mathrm{expit}[\mathrm{logit}\{\hat{q}(x)\}-\alpha y+\log \{e^{\alpha }{\hat{g}}_1(1|x)+{\hat{g}}_1(0|x)\}].$$

  Then we can estimate μ using the proposed doubly robust estimator (6).
• Step 4:

  Repeat steps 1–3 for a set of the sensitivity parameter α and compare results across different α values, say from $-2$ to 2.

Note that the new sensitivity analysis method only has one sensitive parameter α, which also enjoys a nice interpretation based on the odds ratio. Therefore, unlike most existing sensitivity analysis methods containing multiple sensitivity parameters, the new method is much easier to implement and choose the sensitivity parameter.

3. Examples

In this section, we conduct two sets of simulation studies. Similar to Lin et al. [15], we use the first simulation study to demonstrate the effectiveness of the proposed super learner based doubly robust estimation method (6) for adjusting unmeasured confounders in the estimation of ${\mu }_1$ for any given sensitivity parameter α. The second simulation study is conducted to illustrate the performance of the proposed sensitivity analysis method in Section 2.4 by varying the sensitivity parameter. We compare our proposed nonparametric doubly robust estimator using the super learner algorithm (DR_np) with two parametric estimators: OR and IPW as shown in equations (3) and (4), respectively. We use parametric logistic regression to estimate the outcome and the propensity score models for OR and IPW estimators. For the proposed DR-np method, the super learner is based on the library of learners of generalized linear models (GLM), generalized additive models [9, GAM] and recursive partitioning and regression trees [2, rpart].

3.1. Estimating ${\mu }_1$

Given three independent baseline covariates $X=(X_1,X_2,X_3)$ generated from a standard normal distribution, we generate the treatment assignment T and the observable binary outcome $Y(1)$ in the treatment group (T = 1) from the following two logistic model:

$$\begin{array}{rl}& \mathrm{logit}\{P(T=1|X)\}=-2X_1+X_2+X_3+2\{X_1^2+\sin (X_2)+(X_1+0.5{)}^3{e}^{X_3}\},\\ & \mathrm{logit}\{P(Y(1)=1|X,T=1)\}=3X_1-2X_2-X_3+2\{X_1^2{X}_2-\sin (X_2)+(X_1+1{)}^3{e}^{X_3}\}.\end{array}$$

Note that the models used to generate both the treatment and the outcome include non-linear terms in X. When unmeasured confounding exists, the strong ignorability assumption (1) is violated and hence $\mathrm{logit}\{P(Y(1)=1|X,T=1)\}$ will be different from $\mathrm{logit}\{P(Y(1)=1|X,T=0)\}$ with the difference modeled by the exponential tilt relationship (8). The sensitivity parameter α in the exponential model can measure the departure of the truth from the assumption (1) regardless of the distribution and dimension of the unmeasured confounder. We examine $\alpha =(\pm 2,\pm 1.5,\pm 1,\pm 0.5)$ for estimating ${\mu }_1$ (the corresponding odds ratio is from 0.14 to 7.39). The simulation was performed for the sample size n = 1000 with 1000 replicates.

Figure 2 displays squared errors of estimates for ${\mu }_1$ based on the exponential tilt model (8) for each $\alpha =(\pm 2,\pm 1.5,\pm 1,\pm 0.5)$. It can be seen that our proposed method DR_np can estimate ${\mu }_1$ well after adjusting the confounding effect using the exponential tilt model and result in estimates with the smallest median squared error and the smallest variation, with the OR method a close second. In addition, both DR_np and OR have much better performance than IPW.

Figure 2.

Squared errors of estimates for ${\mu }_1$ in Simulation 1.

3.2. Sensitivity analysis

In the previous section, we have demonstrated that the proposed method can successfully adjust the unmeasured confounder for any given sensitivity parameter α. In practice, however, the α is unknown. Next we demonstrate the proposed sensitivity analysis method by checking how the estimate changes when varying the sensitivity parameter α for the exponential tilt model.

To illustrate how our new sensitivity analysis method can be applied to unmeasured confounding settings used by existing sensitivity analysis methods, we incorporate unmeasured confounding by explicitly including a latent variable $U\sim N(1,1)$ as one of the covariates in the models to generate the treatment T and the outcome $Y(t)$ as the following:

$$\begin{array}{rl}\mathrm{logit}\{P(T=1|X,U)\}& =-2X_1+X_2+X_3+{\beta }_u^{t}U\\ & +2\{X_1^2+\sin (X_2)+(X_1+0.5{)}^3{e}^{X_3}\},\end{array}$$

$$\begin{array}{rl}\mathrm{logit}\{P(Y(t)=1|X,U)\}& =3X_1-2X_2-X_3+2t+{\beta }_u^{y}U\\ & +2\{X_1^2{X}_2-\sin (X_2)+(X_1+1{)}^3{e}^{X_3}\},t=0,1.\end{array}$$

The observed outcome is $Y=TY(1)+(1-T)Y(0)$, and the data set used for analysis is $\{Y,T,X\}$. We assume ${\beta }_u^t={\beta }_u^y={\beta }_u$ and consider four cases of ${\beta }_u=(0,1,2,3)$ to represent different strengths of unmeasured confounding in the simulation. For each fixed ${\beta }_u=(0,1,2,3)$, we perform a sensitivity analysis by checking how the estimate of ${\mu }_1$ changes when α varies. The simulation is performed for the sample size n = 1000 with 1000 replicates.

Notice that case 1 with ${\beta }_u=0$ indicates ‘there is no unmeasured confounder’, which corresponds to $\alpha =0$. However, for other nonzero ${\beta }_u$, it is hard to derive the explicit relationship between ${\beta }_u$ and α. As mentioned earlier, a better specified range of α usually depend on subject-matter knowledge. In this simulation, to better understand the impact of choices of the sensitivity parameter, we calculate the estimated log-odds of the control group and compare it to the estimated log-odds of the treatment group. We use this comparison to assess the plausibility of the specified α values, and then we span the range of α based on this difference for each case. One should notice that this estimated difference between the log-odds is not equal to the true value of α; it is only be used as a guidance to specify a plausible range of α for the sensitivity analysis. Based on our simulation, this difference for the three cases where ${\beta }_u=(1,2,3)$ is approximately 0.57, $-0.5$ and $-1.59$, respectively. Thus for case 1 and 2 ( ${\beta }_u=0\mathrm{and}1$), we examine α values of the sequence from $-2$ to 2 with an increment of 0.5. For case 3 and 4 ( ${\beta }_u=2$ and 3), the sensitivity analysis are based on $\alpha =(-2.5,-2,-1.5,-1-0.5,0,1,1.5)$ and $\alpha =(-3,-2.5,-2,-1.5,-1,-0.5,0,0.5,1)$, respectively. The sensitivity parameter α used in our proposed method can measure the difference between the conditional log-odds of the unobserved outcome being 1 and the log-odds of the observed outcome being 1. In other words, α directly governs the difference between $\mathrm{logit}\{P(Y(t)=1|X,T=0)\}$ and $\mathrm{logit}\{P(Y(t)=1|X,T=1)\}$ caused by the unmeasured confounding U with sensitivity parameters $({\beta }_u^t,{\beta }_u^y)$ and the distribution for U. Therefore, compared to the sensitivity parameters $({\beta }_u^t,{\beta }_u^y)$ and the distribution for U used by existing sensitivity analysis methods, the univariate sensitivity parameter α used by our new method directly accounts for all possible unmeasured confounding and is also much easier to choose in practice.

Figures 3 display box plots of the estimates for ${\mu }_1$ obtained by OR, IPW and the proposed method DR_np after adjusting the unmeasured confounding using the exponential tilt method with varying sensitivity parameter α for ${\beta }_u=0,1,2,$ and 3, respectively. The dashed line indicates the true ${\mu }_1$. It can be seen that, at most specified α values, the proposed DR_np provides best estimates for ${\mu }_1$. For case 2, ${\beta }_u=1$ represents a mild unmeasured confounding, while the best estimate appears at $\alpha =0.5$, the estimate at $\alpha =0$ is also quite close to true ${\mu }_1$. This makes sense, since a mild unmeasured confounding would not drastically change the causal estimates. In another word, if we ignore a mild unmeasured confounding, it would not result in a big bias in the estimate. However, as the magnitude of ${\beta }_u$ increases, the departure between the estimate and the true ${\mu }_1$ also tends to increase. But the departure of DR_np remains the smallest at most α values in each case, which demonstrates that our proposed estimator is more resistant to unmeasured confounders.

Figure 3.

Simulation results of the estimates for ${\mu }_1$ for Case 1–4 with ${\beta }_u=0,1,2,3$, respectively.

4. Application

4.1. Effect of low EF on heart failure death

In this section, we discuss an application of the proposed sensitivity analysis to evaluate the causal relationship between heart failure death rate and low ejection fraction ( $\mathrm{EF}\le 30$). The ejection fraction (EF) measures how much blood the left ventricle pumps out with each contraction, which is usually represented as a percentage with a normal range between 50% and 75%. As stated by the World Health Organization Cardiovascular diseases (CVDs) fact sheet, CVD causes 31% of all global deaths and is the number one cause of death. We use the data set which was originally analyzed by Ahmad et al. [1]. The main objective of their study was to estimate death rates due to heart failure and to investigate its link with some major risk factors in the city of Faisalabad (the third most populous city of Pakistan). The data set contains medical records of 299 heart failure patients in Faisalabad from April to December 2015. Ten confounding variables are addressed in this study and summarized in Table 1. Ahmad et al. [1] used traditional survival models such as Cox regression and Kaplan-Meier plots to predict the death rate and identify the risk factors. According to their conclusion, low EF was a significant risk factor associated with death caused by heart failure. They also stated that there was a major difference among the Kaplan-Meier curves for patients with EF $\le$30 and patients with $\mathrm{EF}>30$. Thus, 30 is used as a cutoff to divide the EF into two groups: $T=1,\mathrm{if}\mathrm{EF}\le 30$ and $T=0,\mathrm{if}\mathrm{EF}>30$ in our application. Our goal is to estimate the causal effect of low EF on heart failure deaths and to assess the sensitivity of the result to some unmeasured confounder. The outcome is defined based on whether heart failure death had occurred: $Y=1,\mathrm{if}\mathrm{the}\mathrm{patient}\mathrm{died}$ and $Y=0,\mathrm{otherwise}$. A summary of the death event variable Y is also provided in Table 1. Then, the causal effect of low EF is $\tau ={\mu }_1-{\mu }_0=P[Y(1)=1]-P[Y(0)=1]$. We use our proposed modified doubly robust estimator to estimate ${\mu }_1$ and ${\mu }_0$, and include GLM, GAM, rpart and random forest [10] algorithms in the super learner library.

Table 1.

Mean (and standard deviation) of Ahmad et al. [1] dataset.

Variable	$EF\le 30$ (n = 93)	EF>30 (n = 207)	Measurement
Age	59.63 (10.80)	61.38 (12.34)	Years
Anaemia	0.46 (0.50)	0.42 (0.49)	Binary
$\log$(creatinine_phosphokinase)	5.68 (1.09)	5.65 (1.16)	mcg/L (log)
Diabetes	0.41 (0.49)	0.42 (0.50)	Binary
High blood pressure	0.37 (0.48)	0.34 (0.48)	Binary
Platelets	257,836.45 (94,691.67)	265,850.78 (99,303.30)	platelets/mL
Serum creatinine	1.45 (0.77)	1.47 (1.14)	mg/dL
Serum sodium	135.61 (4.62)	137.08 (4.25)	mEa/L
Sex	0.70 (0.46)	0.63 (0.48)	Binary
Smoking	0.34 (0.48)	0.31 (0.46)	Binary
Death event	0.55 (0.50)	0.22 (0.41)	Binary

With the adjustment of these 10 measured confounding variables, the estimated effect of low EF on heart failure death τ is 0.275. Since the factors that lead to heart failure deaths remain unclear, it is informative and worthwhile to evaluate the sensitivity of the result of this study. For the potential outcome if everyone in this study had $EF\le 30$, we assumed $\mathrm{logit}[P\{Y(1)|T=0,X\}]-\mathrm{logit}[P\{Y(1)|T=1,X\}]={\alpha }_1,$ where ${\alpha }_1$ is the difference between the conditional log-odds of heart failure death for patients with normal $EF(EF>30)$ if they had become $EF\le 30$ and the conditional log-odds of heart failure death for patients with $EF\le 30$. For the potential outcome if everyone in this study had EF>30, we assume $\mathrm{logit}[P\{Y(0)|T=1,X\}]-\mathrm{logit}[P\{Y(0)|T=0,X\}]={\alpha }_2$. The interpretation of ${\alpha }_2$ is similar to ${\alpha }_1$. For simplicity, let ${\alpha }_1={\alpha }_2=\alpha$, with the corresponding odds ratio being $e^{\alpha }$. Table 2 displays the point estimates for the low EF effect on heart failure death rate τ with the adjustment of the unmeasured confounder represented by α.

Table 2.

Estimates of the low EF effect on heart failure death rate.

α:	$-4$	$-3$	$-2$	$-1$
Odds ratio:	0.018	0.05	0.14	0.37
τ:	0.080	0.098	0.135	0.196
α:	1	2	3	4
Odds ratio:	2.72	7.39	20.09	54.60
τ :	0.343	0.372	0.369	0.360

The point estimates of τ change from approximately 0.08 to 0.37, while α being varied from $-4$ to 4. None of the α values makes the point estimate of τ null value zero (which would indicate that low EF would have no effect on heart failure death) or negative (which would indicate that low EF would decrease the heart failure death compared to higher EF). The results of this sensitivity study demonstrate that the potential unmeasured confounder will not likely change the sign of τ (except for extreme α values). Therefore, we can safely conclude that the low EF increases the heart failure death rate when compared to higher EF and such conclusion is robust against the possible unmeasured confounders.

4.2. NSW on post-intervention annual income

The National Supported Work (NSW) was a labor training program conducted in the 1970s. This program provided work experience to selected participants. The study measured 10 baseline covariates summarized in Table 3. The continuous variable, 1978 earnings, is the post-intervention annual income. The effect of the NSW program on post-intervention annual income levels was originally studied by LaLonde [14] and has been analyzed by various studies ever since, such as Imbens [12], Dehejia and Wahba [5], Carnegie et al. [3]. The data analyzed here is the same as in Dehejia and Wahba [5]. In this article, we focus on the the rate of participants who had an increase in the annual earning after the program in 1978. So we define our outcome variable y to indicate if there is an increase in the annual income by comparing 1978 earning to 1975 earnings. In another word, y = 1 indicating the participant's 1978 earnings is more than his/her 1975 earnings and y = 0 otherwise.

Table 3.

Mean (and standard deviation) of LaLonde [14] dataset.

Variable	Treatment (n = 185)	Control (n = 260)	Measurement
Age	25.82 (7.16)	25.05 (7.06)	Years
Education	10.35 (2.01)	10.09 (1.61)	Years of schooling
Black	0.84 (0.36)	0.83 (0.38)	Binary
Hispanic	0.06 (0.23)	0.11 (0.31)	Binary
Married	0.19 (0.39)	0.15 (0.36)	Binary
High School Diploma	0.71 (0.46)	0.83 (0.37)	Binary
1974 Earnings	2.10 (4.89)	2.11 (5.69)	1000 US dollar
1975 Earnings	1.53 (3.22)	1.27 (3.10)	1000 US dollar
1974 Unemployment	0.71 ( 0.46)	0.75 (0.43)	Binary
1975 Unemployment	0.60 (0.49)	0.68(0.47)	Binary
1978 Earnings	6.35 (7.87)	4.55 (5.48)	1000 US dollar
Whether 1978 Earnings increased	0.67 (0.47)	0.48 (0.49)	Binary

We use this data set to demonstrate how our proposed sensitivity procedure can address the impact of unmeasured confounding on the effect of NSW on the proportion of participants who had an increase in 1978 earnings based on our proposed modified DR estimator. Let T = 1 indicate enrollment in NSW and T = 0 otherwise. Let X be the vector of all measured confounders described in Table 3. Based on the assumption of no unmeasured confounding, the estimated causal effect of NSW (τ) is 0.092, which implies that if everyone has participated in NSW, the NSW program increases the proportion of participants with increased post-intervention income by 9.2%. However, as mentioned in Imbens [12], strong motivation to enroll in a job-training program may lead to more favorable outcomes. Thus motivation to join the program can definitely be considered unmeasured confounding and would be worthwhile to address using the sensitivity analysis. For the potential outcomes if everyone had enrolled in NSW, we use the exponential tilt method of (7) to assume $\mathrm{logit}\{Y(1)=1|T=0,X\}-\mathrm{logit}\{Y(1)=1|T=1,X\}={\alpha }_1$ and ${\alpha }_1$ represents the difference between the conditional log-odds of that 1978 earnings increased for nonparticipants of NSW if they had been enrolled in NSW and that of those who were actually enrolled in the NSW program. Similarly, for the potential outcome whether there is an increase in post-intervention outcome if everyone were not enrolled, we assume $\mathrm{logit}\{Y(0)=1|T=1,x\}-\mathrm{logit}\{Y(0)=1|T=0,x\}={\alpha }_0$, where ${\alpha }_0$ can be interpreted similarly to ${\alpha }_1$. For simplicity, we assume ${\alpha }_1={\alpha }_0=\alpha$. Table 4 displays the point estimates for the NSW effect on the ratio of participants whose post-intervention income has increased (τ) with the adjustment of potential unmeasured confounding represented by α. The point estimates of τ change from approximately $-0.02$ to 0.11. When $\alpha =-4$, the sign of the point estimate reversed. While unmeasured confounders represented by $\alpha =-4$ could reverse the sign of our causal estimates, it should be pointed out that $\alpha =-4$ indicates extremely strong unmeasured confounding with a corresponding odds ratio 0.0018. If the investigators have made efforts to collect the measured confounders, it might not be very likely to have unmeasured confounders that result in an $|\alpha |>2$ or even $|\alpha |>1$, although those values were examined in this sensitivity analysis. Based on the above results, we could conclude that such unmeasured confounder would only mildly affect the cause effect of the NSW program on the ratio of participants whose post-intervention annual earning increased.

Table 4.

Estimates of the NSW program effect on post-intervention income increase rate.

α:	$-4$	$-3$	$-2$	$-1$
Odds ratio:	0.018	0.05	0.14	0.37
τ:	$-0.022$	0.006	0.043	0.073
α:	1	2	3	4
Odds ratio:	2.72	7.39	20.09	54.60
τ :	0.101	0.106	0.108	0.109

5. Conclusion

We have proposed a new sensitivity analysis method for causal inference to adjust for unmeasured confounding in the estimation of the mean outcome by combining the ideas of the doubly robust estimator, the exponential tilting method, and the super learner algorithm. In causal inference, when unmeasured confounders exist, the conditional distribution of the observed outcome is different from that of the unobserved outcome given the covariates. We use the exponential tilting assumption to link these two conditional distributions together directly with a univariate sensitivity parameter. This sensitivity parameter addresses the departure from the assumption of no unmeasured confounder. Compared to most of the existing sensitivity analysis in the literature, our method does not require modeling assumptions for the unmeasured confounders as latent variables and hence the unmeasured confounder could be continuous, binary, or categorical and could be univariate or multivariate. In addition, the sensitivity parameter can be interpreted as a log-odds ratio for a binary outcome, which makes the choice of its range relatively easy for practitioners. To reduce the bias of traditional parametric methods, we propose a nonparametric doubly robust estimator by incorporating super learner algorithms. The simulation studies demonstrate the effectiveness of the proposed method and its superiority to some other existing methods.

Funding Statement

Yao's research is supported by U.S. Department of Energy with the award DE-EE0007328 and UCR Academic Senate CoR Grants 2020–2021.

Disclosure statement

No potential conflict of interest was reported by the authors.