Sensitivity analysis using bias functions for studies extending inferences from a randomized trial to a target population

Abstract

Extending (generalizing or transporting) causal inferences from a randomized trial to a target population requires assumptions that randomized and non-randomized individuals are exchangeable conditional on baseline covariates. These assumptions are made on the basis of background knowledge, which is often uncertain or controversial, and need to be subjected to sensitivity analysis. We present simple methods for sensitivity analyses that directly parameterize violations of the assumptions using bias functions and do not require detailed background knowledge about specific unknown or unmeasured determinants of the outcome or modifiers of the treatment effect. We show how the methods can be applied to non-nested trial designs, where the trial data are combined with a separately obtained sample of non-randomized individuals, as well as to nested trial designs, where a trial is embedded within a cohort sampled from the target population.

1 |. BACKGROUND

The distribution of effect modifiers among participants in randomized trials is often different from that of individuals seen in clinical practice. Consequently, average treatment effects estimated in trials do not directly apply to target populations beyond the population represented by the randomized individuals [1]. Methods for extending – “generalizing” or “transporting” [2, 3] – causal inferences from a trial to a target population require “generalizability” or “exchangeability” assumptions, which state that randomized and non-randomized groups are exchangeable conditional on some set of covariates (typically measured at baseline) [4, 5, 6, 7]. These exchangeability assumptions are made on the basis of background knowledge and are often uncertain or controversial. Thus, investigators interested in extending inferences beyond the population of randomized individuals need to conduct sensitivity analyses to examine how potential violations of the assumptions would affect their findings.

The literature on sensitivity analysis for unmeasured confounding in observational studies or for the related problem of data missing not-at-random is very extensive (starting with [8] and expanded in various ways, e.g., [9, 10, 11]). In the context of analyses extending inferences from a trial to a target population, however, the only proposal for sensitivity analysis methods that we are aware of is the work of Nguyen et al. [12, 13]. Their approach can be useful when background knowledge is strong enough to suggest that a single covariate, which is measured among randomized individuals but not among non-randomized individuals, when combined with fully observed covariate is enough to render randomized and non-randomized groups exchangeable. The approach may be less useful, however, in the more typical case where violations of the exchangeability assumptions are due to high dimensional unknown or unmeasured variables.

In this paper, we propose methods for sensitivity analysis that do not require detailed background knowledge about or measurement of specific effect modifiers or determinants of the outcome. Instead, our methods parameterize violations of the exchangeability assumption using bias functions expressed in terms of differences between the potential (counterfactual) outcome means of randomized and non-randomized individuals conditional on covariates.

To empirically evaluate the methods, we take advantage of data from two centers participating in the HALT-C trial comparing peginterferon alfa-2a treatment versus no treatment for patients with chronic hepatitis C infection [14]. We treat one of the centers as the “index center” and the other as the “target center,” with the latter providing a sample of individuals from a population different from the index center (see [15] for a similar setup). Our goal is to transport causal inferences from the index center to the population represented by the target center under a mean exchangeability assumption, and to propose methods for sensitivity analysis when the assumption does not hold. Because both centers were actually participating in the same trial, and treatment and outcome data are available from both, we have the ability to compare the estimates from our analyses against the randomization-based analyses in the target center. We also apply the methods to conduct sensitivity analysis when transporting inferences from the AIDS Clinical Trials Group Study 175 (ACTG 175) trial [16, 17] to a population of trial-eligible women in the Women’s Interagency HIV Study (WIHS) [18]. The data structures in our empirical evaluation and the data application are compatible with those of the non-nested trial design, where a trial dataset is combined with a separately obtained sample from a population of non-randomized individuals [19]. In the Appendix, we discuss the sampling scheme non-nested trial designs. We also show how the methods can be modified for use with nested trial designs, in which the trial is nested within a cohort sampled from the target population [6, 19].

2 |. EXTENDING INFERENCES TO A TARGET POPULATION

Consider a non-nested trial design, consisting of a trial and a separately obtained simple random sample of non-randomized individuals from the target population [7, 19], an (infinite) superpopulation to which inferences are to be extended (generalized or transported). We use X to denote baseline covariates; A to denote the time-fixed (non-time-varying) treatment assignment; S to denote the trial participation indicator (1, for randomized individuals; 0, for non-randomized individuals); and Y to denote the outcome.

In non-nested trial designs, the data available from the trial are realizations of independent random tuples (X_i, S_i = 1, A_i, Y_i), i = 1, …, n_RCT; the data from the sample of the target population are realizations of independent random tuples (X_i, S_i = 0), i = 1, …, n_obs; and we define the total number of (sampled) observations that contribute data to the analysis as n = n_RCT + n_obs. The n observations are organized in a composite dataset that contains data from the trial and the sample of non-randomized individuals from the target population. We require that, as n → ∞, $\frac{n_{\text{RCT}}}{n}\to {\pi }_{\text{RCT}}>0$ and $\frac{n_{\text{obs}}}{n}\to {\pi }_{\text{obs}}>0$. Note that the data exhibit a special missingness pattern: for randomized individuals we have data on (X, S = 1, A, Y); for non-randomized individuals we only have data on (X, S = 0) [7].

The non-nested design can be understood as inducing a biased sampling model [20, 21]: data are collected from all randomized individuals and a simple random sample of non-randomized individuals (with unknown sampling fraction) [19, 7] from an actual (finite) population meeting the study eligibility criteria; the actual population can be viewed in turn as a sample from the (infinite) target population. Provided that the sampled non-randomized individuals are indeed representative of the non-randomized subset of the target population, the estimands of interest in analyses extending inferences discussed below are identifiable, with expectations taken with respect to the distribution induced by the study design [20, 21]. In the Appendix, we provide a more detailed discussion of the sampling scheme underlying non-nested trial designs.

Let Y^a be the potential (counterfactual) outcome under intervention to set treatment to a [22, 23]. We only consider binary treatments, such that a ∈ {0, 1}; extensions to multivalued treatments are straightforward. One causal contrast of interest is the average treatment effect among non-trial participants in the target population, E[Y¹ − Y⁰|S = 0], which is identifiable both in non-nested and nested trial designs. In general, this treatment effect is not equal to the treatment effect among randomized individuals, E[Y¹ − Y⁰|S = 0] ≠ E[Y¹ − Y⁰|S = 1]. In nested trial designs, we can also identify the average treatment effect in the overall target population, E[Y¹ − Y⁰] [6, 19] (i.e., not just the non-randomized sub-population). In the main text of this paper, we focus on identification, estimation, and sensitivity analysis methods for E[Y¹ − Y⁰|S = 0] because our empirical evaluation and applied example only allow identification of that causal estimand [7]; in the Appendix we propose sensitivity analysis methods for studies that aim to draw causal inferences about E[Y¹ − Y⁰] [6].

2.1 |. Identifiability conditions

We now discuss sufficient conditions for identifying the potential outcome means among non-randomized individuals, E[Y^a|S = 0]. These means are of inherent scientific interest and can also be used to identify the average treatment effect among non-trial participants in the target population.

(I). Consistency:

The observed outcome for any individual the observed outcome under treatment a equals that individual’s counterfactual outcome under the same treatment, that is, if A_i = a, then $Y_i=Y_i^a$, for every i. Implicit in this notation is the assumption that the invitation to participate in the trial and trial participation do not affect the outcome except through treatment assignment [3]. In effect, we are making an exclusion restriction assumption of no direct effect of trial participation on the outcome, such that potential outcomes need only be indexed by treatment a, not trial participation s [24].

(II). Conditional mean exchangeability in the trial (over A):

Among randomized individuals, the potential outcome mean under treatment a is independent of treatment, conditional on baseline covariates, that is, E[Y^a|X = x, S = 1, A = a] = E[Y^a|X = x, S = 1] for every a and every x with positive density f_X,_S (x, S = 1) > 0.

(III). Positivity of treatment assignment:

In the trial, the probability of being assigned to each treatment, conditional on the covariates needed for exchangeability, is positive: Pr[A = a |X = x, S = 1] > 0 for every a and every x with positive joint density f_X,_S (x, S = 1) > 0.

(IV). Conditional mean exchangeability over S (transportability):

The potential outcome mean is independent of trial participation, conditional on baseline covariates, E[Y^a|X = x, S = 1] = E[Y^a|X = x, S = 0] for every a and every x with positive density f_X,_S (x, S = 0).

(V). Positivity of trial participation:

The probability of participating in the trial, conditional on the covariates needed to ensure conditional mean exchangeability over S, is positive, Pr[S = 1|X = x] > 0 for every x with positive density f_X,_S (x, S = 0) > 0.

Note that we have used X generically to denote baseline covariates. It is possible, however, that strict subsets of X are adequate to satisfy each exchangeability condition. For example, in a marginally randomized trial, the mean exchangeability among trial participants holds unconditionally.

Consistency, conditional mean exchangeability over A, and positivity of treatment assignment are expected to hold in (marginally or conditionally randomized) trials of well-defined interventions. In order to focus our attention on issues related to selective trial participation, we assume complete adherence to treatment assignment and no loss to follow-up in the trial.

Conditional mean exchangeability over S and positivity of trial participation are the assumptions that allow us to extend causal inferences beyond the trial. Positivity of trial participation is, in principle, testable [25]. In contrast, the conditional mean exchangeability over S is not testable using the observed data; in many applications, it is a controversial or uncertain assumption.

2.2 |. Identification of potential outcome means

The conditions listed above are sufficient [7] to identify the conditional potential outcome mean in the population of non-randomized individuals, E[Y^a|S = 0], using the observed data functional

$$\varphi (a)\equiv \text{E}[\text{E}[Y\mid X,S=1,A=a]\mid S=0].$$ (1)

In other words, provided that identifiability conditions I through V hold, ϕ(a) can be interpreted as the potential outcome mean had non-randomized individuals received treatment a. This functional can be re-expressed using inverse odds (IO) of participation weighting [7],

$$\varphi (a)=\frac{1}{\mathrm{Pr}[S=0]}\text{E}\left[\frac{I(S=1,A=a)Y\mathrm{Pr}[S=0\mid X]}{\mathrm{Pr}[S=1\mid X]\mathrm{Pr}[A=a\mid X,S=1]}\right],$$ (2)

where I (S = 1, A = a) is the indicator function that takes value 1 when S = 1 and A = a; zero otherwise. Furthermore, under the positivity conditions,

$$\mathrm{Pr}[S=0]=\text{E}\left[\frac{I(S=1,A=a)\mathrm{Pr}[S=0\mid X]}{\mathrm{Pr}[S=1\mid X]\mathrm{Pr}[A=a\mid X,S=1]}\right],$$ (3)

which will be useful in deriving estimators for ϕ(a).

2.3 |. Estimation and inference

In this section, we briefly review estimators of potential outcome means and treatment effects in the population of non-randomized individuals [7].

Estimation

Potential outcome means:

We can estimate potential outcome means using estimators that rely on modeling the outcome mean, the probability of trial participation, or both.

Outcome model-based (g-formula) estimator:

We can use the sample analog estimator based on (1):

$${\widehat{\varphi }}_{\text{OR}}(a)={\left\{\sum _{i=1}^n\left(1-S_i\right)\right\}}^{-1}\sum _{i=1}^n\left(1-S_i\right){\widehat{g}}_a\left(X_i\right),$$ (4)

where ${\widehat{g}}_a(X)$ is an estimator for E[Y|X, S = 1, A = a]. In applied analyses, it is impossible to nonparametrically estimate this conditional mean because of the curse of dimensionality [26], and we need to make modeling assumptions. Typically, we posit a parametric model g_a (X; θ_a), with finite-dimensional parameter θ_a. When using such a model, the validity of the g-formula estimator depends on correct model specification, in addition to the identifiability conditions.

Inverse odds weighting:

Using (2), we can obtain an inverse odds weighting estimator,

$${\widehat{\varphi }}_{\text{IOW}1}(a)={\left\{\sum _{i=1}^n\left(1-S_i\right)\right\}}^{-1}\sum _{i=1}^n{\widehat{w}}_a\left(X_i,S_i,A_i\right)Y_i,$$ (5)

where

$${\widehat{w}}_a\left(X_i,S_i,A_i\right)=I\left(S_i=1,A_i=a\right)\frac{1-\widehat{p}\left(X_i\right)}{\widehat{p}\left(X_i\right){\widehat{e}}_a\left(X_i\right)},$$

$\widehat{p}(X)$ is an estimator for Pr[S = 1|X], and ${\widehat{e}}_a(X)$ is an estimator for Pr[A = a|X, S = 1]. Alternatively, combining (2) and (3), we can normalize the weights,

$${\widehat{\varphi }}_{\text{IOW2}}(a)={\left\{\sum _{i=1}^n{\widehat{w}}_a\left(X_i,S_i,A_i\right)\right\}}^{-1}\sum _{i=1}^n{\widehat{w}}_a\left(X_i,S_i,A_i\right)Y_i.$$ (6)

As for the outcome model-based estimator, in applications, it is impossible to nonparametrically estimate the conditional probability of trial participation Pr[S = 1|X] and we have to make modeling assumptions. Typically, we posit a parametric model p (X; β) for Pr[S = 1|X], with finite-dimensional parameter β. When using such a model, the validity of the inverse odds weighting estimators depends on correct model specification, in addition to the identifiability conditions. The conditional probability of treatment among randomized individuals Pr[A = a|X, S = 1] is known and does not have to be estimated; in the presence of baseline covariate imbalances between the randomized groups, however, it is useful to estimate it using a parametric model, say, e_a (X; γ), with finite-dimensional parameter γ [27, 28, 29, 30, 31].

Augmented inverse odds weighting:

To improve the efficiency of the inverse odds weighting estimator and gain robustness to misspecification of the models for the conditional outcome mean or the probability of trial participation, we can use the augmented (doubly robust) inverse odds weighting estimator

$${\widehat{\varphi }}_{\text{AIOW}1}(a)={\left\{\sum _{i=1}^n\left(1-S_i\right)\right\}}^{-1}\sum _{i=1}^n\left\{{\widehat{w}}_a\left(X_i,S_i,A_i\right)\left\{Y_i-{\widehat{g}}_a\left(X_i\right)\right\}+\left(1-S_i\right){\widehat{g}}_a\left(X_i\right)\right\}.$$ (7)

Again, we can normalize the weights,

$$\begin{gathered} {\widehat{\varphi }}_{\text{AIOW}2}(a)={\left\{\sum _{i=1}^n{\widehat{w}}_a\left(X_i,S_i,A_i\right)\right\}}^{-1}\sum _{i=1}^n{\widehat{w}}_a\left(X_i,S_i,A_i\right)\left\{Y_i-{\widehat{g}}_a\left(X_i\right)\right\} \\ +{\left\{\sum _{i=1}^n\left(1-S_i\right)\right\}}^{-1}\sum _{i=1}^n\left(1-S_i\right){\widehat{g}}_a\left(X_i\right). \end{gathered}$$ (8)

These two estimators are “doubly robust” in the sense that they produce valid results when either the model for the probability of trial participation or the model for the conditional outcome mean among randomized individuals is correctly specified.

Treatment effects:

We can use the potential outcome mean estimators to estimate the average treatment effect in the target population of non-randomized individuals, E[Y¹ − Y⁰|S = 0], by taking differences. For instance, we can estimate the average treatment effect using the augmented inverse odds weighting estimator with normalized weights as ${\widehat{\varphi }}_{\text{AIOW}2}(1)-{\widehat{\varphi }}_{\text{AIOW}2}(0)$.

Inference:

Confidence intervals for the estimated potential outcome means and mean differences obtained using the above estimators can be performed by M-estimation [32] or by bootstrap methods [33]. See the section below on “Inference for sensitivity analysis” for additional details.

3 |. SENSITIVITY ANALYSIS FOR VIOLATIONS OF CONDITIONAL EXCHANGEABILITY OVER S

3.1 |. Violations of conditional exchangeability over S

The validity of all the estimators in the previous section depends critically on the assumption of conditional exchangeability over S. In most applications, however, it is unlikely that we know or can measure enough baseline variables to ensure that the assumption holds. Thus, we need to consider the impact of assumption violations, when E[Y^a|X = x, S = 1] ≠ E[Y^a|X = x, S = 0] for some x with positive density in the target population. The magnitude of the violations, that is, the magnitude of the difference between the conditional potential outcome means E[Y^a|X, S = 1] and E[Y^a|X, S = 0], determines the amount of bias. Because this magnitude cannot be assessed using the data, we need to conduct sensitivity analyses to examine the impact of violations of the condition affect on our results [10].

3.2 |. Sensitivity analysis with bias functions

3.2.1 |. Bias functions

Following prior work on sensitivity analysis for marginal structural causal models [10, 34, 35], we parameterize violations of conditional exchangeability over S using a “bias function” for each treatment a, defined as

$$u(a,X)\equiv \text{E}\left[Y^a\mid X,S=1\right]-\text{E}\left[Y^a\mid X,S=0\right].$$ (9)

Intuitively, u (a, X) expresses violations of the exchangeability assumption for each treatment a as a function of the potential outcome means between randomized and non-randomized individuals, conditional on (within strata of) the measured covariates X.

Parameterizing the violations of the exchangeability assumption allows us to re-express the conditional potential outcome mean under treatment a among non-randomized individuals. First, by re-arranging the definition in (9), we obtain E[Y^a|X, S = 0] = E[Y^a|X, S = 1] − u (a, X). Next, under consistency (condition I), exchangeability of the treated and untreated groups in the trial (condition II), and positivity of treatment assignment in the trial (condition III), we have that E[Y^a|X, S = 1] = E[Y|X, S = 1, A = a]. Putting everything together,

$$\text{E}\left[Y^a\mid X,S=0\right]=\text{E}[Y\mid X,S=1,A=a]-u(a,X).$$ (10)

Using the law of iterated expectation and (10),

$$\begin{gathered} \text{E}\left[Y^a\mid S=0\right]=\text{E}\left[\text{E}\left[Y^a\mid X,S=0\right]\mid S=0\right] \\ =\text{E}[\text{E}[Y\mid X,S=1,A=a]-u(a,X)\mid S=0] \\ =\text{E}[\text{E}[Y\mid X,S=1,A=a]\mid S=0]-\text{E}[u(a,X)\mid S=0], \end{gathered}$$ (11)

where the first term in the last expression is identifiable from the data, regardless of whether conditional exchangeability over S holds, and the second term is also identifiable for each user-specified u (a, X) function.

3.2.2 |. Sensitivity analysis

The result in (11) suggests simple approaches for sensitivity analysis using the potential outcome mean estimators in the previous section.

Outcome model-based (g-formula) estimator:

We can modify the outcome model-based estimator to directly incorporate the bias correction,

$${\widehat{\varphi }}_{\text{OR}}^{\text{bc}}(a)={\left\{\sum _{i=1}^n\left(1-S_i\right)\right\}}^{-1}\sum _{i=1}^n\left(1-S_i\right)\left\{{\widehat{g}}_a\left(X_i\right)-u\left(a,X_i\right)\right\}.$$ (12)

Inverse odds weighting:

Similarly, we can modify the inverse odds weighting estimators as

$${\widehat{\varphi }}_{\text{Iow}1}^{\text{bc}}(a)={\left\{\sum _{i=1}^n\left(1-S_i\right)\right\}}^{-1}\sum _{i=1}^n\left\{{\widehat{w}}_a\left(X_i,S_i,A_i\right)Y_i-\left(1-S_i\right)u\left(a,X_i\right)\right\},$$ (13)

or, normalizing the weights,

$$\begin{gathered} {\widehat{\varphi }}_{\text{IOW}2}^{\text{bc}}(a)={\left\{\sum _{i=1}^n{\widehat{w}}_a\left(X_i,S_i,A_i\right)\right\}}^{-1}\sum _{i=1}^n{\widehat{w}}_a\left(X_i,S_i,A_i\right)Y_i \\ -{\left\{\sum _{i=1}^n\left(1-S_i\right)\right\}}^{-1}\sum _{i=1}^n\left(1-S_i\right)u\left(a,X_i\right). \end{gathered}$$ (14)

In the Appendix, we show how the bias correction for the inverse odds weighting estimators can be easily implemented using standard regression software, by simply re-coding the outcome values to directly incorporate the bias correction function u (a, X).

The validity of the estimators in equations (12) through (14) depends on the correct choice of the bias functions u (a, X) and, when relying on parametric models, the correct specification of the models for the mean outcome among randomized individuals, g_a (X; θ_a), or the probability of trial participation conditional on covariates, p (X; β). By correct specification we mean that the models g_a (X; θ_a) and p (X; β) should be good approximations of the corresponding conditional mean/probability functions. Correct model specification is distinct from conditional exchangeability over S: informally, correct model specification addresses differences between randomized and non-randomized individuals with respect to measured variables; the bias functions address residual differences due to unmeasured or unknown variables.

Augmented inverse odds weighting:

We can incorporate the bias correction functions in the augmented inverse odds weighting estimators,

$$\begin{gathered} {\widehat{\varphi }}_{\text{AIOW}1}^{\text{bc}}(a)={\left\{\sum _{i=1}^n\left(1-S_i\right)\right\}}^{-1}\sum _{i=1}^n\{{\widehat{w}}_a\left(X_i,S_i,A_i\right)\left\{Y_i-{\widehat{g}}_a\left(X_i\right)\right\} \\ +\left(1-S_i\right)\left\{{\widehat{g}}_a\left(X_i\right)-u(a,X)\right\}\}, \end{gathered}$$ (15)

or, normalizing the weights,

$$\begin{gathered} {\widehat{\varphi }}_{\text{AIOW2}}^{\text{bc}}(a)={\left\{\sum _{i=1}^n{\widehat{w}}_a\left(X_i,S_i,A_i\right)\right\}}^{-1}\sum _{i=1}^n{\widehat{w}}_a\left(X_i,S_i,A_i\right)\left\{Y_i-{\widehat{g}}_a\left(X_i\right)\right\} \\ +{\left\{\sum _{i=1}^n\left(1-S_i\right)\right\}}^{-1}\sum _{i=1}^n\left(1-S_i\right)\left\{{\widehat{g}}_a\left(X_i\right)-u\left(a,X_i\right)\right\}. \end{gathered}$$ (16)

These estimators retain the double robustness property of their non-bias corrected counterparts, in the sense that, when the bias correction function is correctly specified, they produce valid results when either the model of the outcome mean among randomized individuals or the model for the probability of trial participation is correctly specified.

Sensitivity analysis for treatment effects:

As when conditional exchangeability over S holds, we can perform sensitivity analysis for treatment effects by taking the difference between the appropriate bias-corrected estimators.

Inference for sensitivity analysis:

The general theory in [10] shows that valid inference for the estimated bias-corrected potential outcome means and mean differences can be carried out using M-estimation methods [32]. Specifically, we can obtain “sandwich” estimators of the sampling variance, analytically (see [30, 31, 36] for related specific examples using parametric models) or using numerical methods [37]. In the Appendix, we illustrate how to obtain sandwich variance estimators for fairly general parametric models. Bootstrap methods can also be used [33]. Under appropriate technical conditions, both M-estimation and bootstrap methods kinds of methods can be used when observations are independent, even if they are not identically distributed, as is the case in non-nested trial designs (see Chapters 5 and 7 in reference [32] and Chapter 1 in reference [38]). Furthermore, both kinds of methods can account for the uncertainty in estimating the parameters of the models for the probability of trial participation and the outcome.

In the Appendix, we discuss the identification, estimation, and sensitivity analysis for potential outcome means and average treatment effects in the overall target population (i.e., not just the non-randomized subset of the population).

3.3 |. Choosing bias functions in practice

We cannot identify the bias functions u (a, X) using the data. Instead, we can use different bias functions to conduct sensitivity analyses. To develop some intuition about the choice of functions, note that u (a, X) quantifies the degree of selection into the trial on the basis of the potential outcome Y^a. Suppose that higher outcome values are preferred (e.g., the outcome is a quality-of-life score with higher values indicating higher quality). If we believe that, conditional on some specific level of the measured covariates, X = x, randomized individuals have better outcomes than non-randomized individuals in the absence of treatment, that is, we believe that E[Y⁰|X = x, S = 1] > E[Y⁰|X = x, S = 0], then we should select u (0, x) > 0. Conversely, if we believe that randomized individuals have worse outcomes than non-randomized individuals in the absence of treatment, that is, we believe that E[Y⁰|X = x, S = 1] < E[Y⁰|X = x, S = 0], then we should select u (0, x) < 0.

Now, define the difference of the bias functions, δ (X),

$$\begin{gathered} \delta (X)\equiv u(1,X)-u(0,X) \\ =\left\{\text{E}\left[Y^1\mid X,S=1\right]-\text{E}\left[Y^1\mid X,S=0\right]\right\}-\left\{\text{E}\left[Y^0\mid X,S=1\right]-\text{E}\left[Y^0\mid X,S=0\right]\right\} \\ =\left\{\text{E}\left[Y^1\mid X,S=1\right]-\text{E}\left[Y^0\mid X,S=1\right]\right\}-\left\{\text{E}\left[Y^1\mid X,S=0\right]-\text{E}\left[Y^0\mid X,S=0\right]\right\} \\ =\text{E}\left[Y^1-Y^0\mid X,S=1\right]-\text{E}\left[Y^1-Y^0\mid X,S=0\right]. \end{gathered}$$

This calculation shows that the difference of the bias functions equals the difference of the conditional average treatment effects given X among trial participants and non-participants. In other words, the difference of the bias functions reflects the “residual,” unexplained by X, effect modification over the participation indicator. Clearly, the choice of bias functions, especially covariate-dependent ones, will require substantial epidemiological and domain-specific expertise in practical applications.

For instance, suppose, as above, that higher outcome values are preferred and that, for some specific level of the covariates X = x, we believe that δ (x) > 0, that is, u (1, x) > u (0, x). This corresponds to a belief that individuals who choose to participate in the trial benefit more (or are harmed less) from a = 1 than a = 0, compared to individuals who choose not to participate. A similar argument shows that, when higher values of the outcome are preferred, δ (x) < 0 means that individuals who choose to participate in the trial benefit less (or be harmed more) from a = 1 than a = 0, compared to individuals who choose not to participate. In the special case of δ (x) = 0 for all x, the conditional average treatment effect among randomized and non-randomized individuals is the same, meaning that trial participants are not selected based on the magnitude of the benefit (or harm) they might experience from treatment.

The interpretation of u (0, X) and δ (X) suggests a convenient way to perform sensitivity analysis. For simplicity, we might use functions that do not depend on covariates, such that u (0, X) ≡ u (0) and δ (X) ≡ δ. Each choice of a pair of values for u (0), δ implies a choice of u (1) = δ +u (0). The sensitivity analysis, then, examines the impact of a sufficiently diverse set of u (0), δ pairs on inferences about potential outcome means and average treatment effects, using the estimators in Section 3.2.2.

Arguably, allowing the bias functions to vary over baseline covariates is more realistic. When supported by background knowledge, the use of the covariate-dependent functions u (0, X) and δ (X), corresponding to a covariate-dependent u (1, X), should result in more informative sensitivity analyses. When background knowledge is not that sharp, however, the simple approach of examining pairs of u (0), δ over a sufficiently broad range of values, may be adequate to explore the impact of violations of conditional exchangeability over S on our causal inferences. Of note, the double robustness property holds regardless of the dependence of the bias functions on covariates.

4 |. EMPIRICAL ASSESSMENT: THE HALT-C MULTICENTER TRIAL

We now present an empirical assessment of the sensitivity analysis methods. In practical applications of the methods, there will be no way to estimate the treatment effect in the target population. In order to get some insight into the need for sensitivity analysis, in our empirical assessment we re-purposed data from a multicenter trial, treating an index center as the “trial” and another target center as a sample from the “target population.” This allows us to estimate the treatment effect in the population underlying the target center in two ways: (1) using transportability methods, which combine the index center data with covariate data from the target center, but ignore treatment and outcome data from the target center; and (2) using standard trial analysis methods, which use data from the target center, but ignore all data from the index center.

4.1 |. Data and methods

The HALT-C trial and transportability between centers

The HALT-C trial enrolled patients with chronic hepatitis C and advanced fibrosis who had not responded to previous therapy and randomized them to treatment with peginterferon alpha-2a (a = 1) versus no treatment (a = 0). Patients were enrolled in 10 research centers and followed up every 3 months after randomization. We used data on the secondary outcome of platelet count at 9 months of follow-up; we report all platelet measurements as platelets ×10³/ml. To simplify exposition, we only used data from 217 patients (210 with complete data for our analyses) seen in two different research centers who had complete baseline covariate and outcome data: the index center, S = 1, contributing 113 patients (110 with complete data); and the target center, S = 0, contributing 104 patients (100 with complete data). For purposes of illustration, we treated the target center data as a sample from a population of (trial-eligible) non-randomized individuals. Our goal was to transport causal inferences from the index center to the population represented by the target center. Because both centers were actually participating in the same trial, and because treatment and outcome data were available from both, we could compare estimates from transportability and sensitivity analyses against the randomization-based analyses in the target center. We purposely chose these two centers because they each enrolled more than 100 patients and estimated the unadjusted treatment effect on the platelet count to be substantially different. In view of how we created the dataset for this illustration, our analyses should not be clinically interpreted.

Sensitivity analysis methods

We used bias functions that where constant within levels of the baseline covariates. Specifically, we examined u (0) values of −40, 0, and +40 and varied δ from −40 to +40, in steps of 20, examining all possible u (0), δ pairs. We chose this range of values because, across all 10 centers participating in the HALT-C trial, the smallest center-specific unadjusted mean difference was approximately 24.0 and the largest was 71.8 (all but one of these mean differences were smaller than 50). Furthermore, the difference between the largest and smallest post-treatment mean was approximately 27.0 for patients assigned to a = 1 and 41.7 for patients assigned to a = 0. For comparison, the standard deviation of the pre-treatment platelet count across all centers participating in the trial was approximately 65.9; the standard deviation of the post-treatment count across all centers participating in the trial was approximately 55.1 for patients assigned to a = 1 and 73.5 for patients assigned to a = 0.

We used the bias functions with the estimators provided in the previous section to perform sensitivity analyses. We obtained confidence intervals for the potential outcome mean under each treatment using the “sandwich” variance [32], accounting for uncertainty in estimating all required models (for the probability of trial participation, the probability of treatment in the trial, or the outcome among trial participants). In the Appendix, we describe an additional example sensitivity analysis using covariate-dependent bias functions. We implemented all sensitivity analyses using the R [39] package geex [37].

Model specification

The sensitivity analysis methods require the specification of models for the outcome mean in each treatment group, the probability of trial participation, and the probability of treatment among randomized individuals. We specified logistic regression models for the probability of being in center S = 1 and the probability of being assigned to peginterferon alpha-2a (a = 1) among randomized individuals in that center. We also specified two linear regression models, one for each treatment group, for the mean of the outcome among randomized individuals. Baseline covariate information is summarized in the Appendix; all models used the baseline covariates listed in the table as main effects (baseline platelets, age, sex, treatment history, race/ethnicity, baseline white blood cell count, history of using needles or recreational drugs, transfusion history, body mass index, creatinine, and smoking). We built separate outcome models in each treatment group to allow for heterogeneity of the treatment effect over all baseline covariates included in the models.

4.2 |. Results

The “base case” analyses under conditional mean exchangeability over S (i.e., u (0) = 0 and δ = 0) produced similar results across different estimators (see Appendix for numerical results), suggesting that models were approximately correctly specified [40]. The unadjusted randomization-based analyses among patients with S = 0 produced fairly different results compared to the transportability analyses. The differences, however, were smaller after using an augmented inverse probability weighting regression estimator to analyze the S = 0 data (because of randomization among individuals in S = 0, the covariate adjustment is virtually assumption-free [41]). The fairly large change in estimates after covariate adjustment in the sample from S = 0 is a reminder that large baseline covariate imbalances can occur despite randomization when the sample size is modest.

Sensitivity analysis results are summarized in Figure 1 (see Appendix for detailed results). Overall, the results were only moderately sensitive to violations of conditional mean exchangeability over S; for example, regardless of the choice of u (0), values of δ smaller than −25 or −30 (depending on the estimator) would be needed for the estimated average treatment effect in S = 0 to be in opposite direction compared to the base-case analysis (i.e., the black lines in the graphs cross only for δ values lower than −25 or −30). Sensitivity analysis results were much more uncertain when using inverse odds weighting compared to the g-formula or augmented inverse odds weighting estimators.

FIGURE 1.

Sensitivity analysis results for transporting inferences between two research centers participating in the HALT-C trial.

AIOW2 = augmented inverse odds weighting estimator with normalized weights; IOW2 = inverse odds weighting with normalized weights; OM = outcome model-based estimator. Results are shown as point estimates (black lines) and corresponding 95% confidence intervals (gray lines) for potential outcome means under treatment a = 1 (solid lines) and a = 0 (dashed lines). Results for IOW1 and AIOW1 were similar to IOW2 and AIOW2, respectively, and are shown in the Appendix.

5 |. DATA APPLICATION: THE ACTG 175 TRIAL

5.1 |. Data and methods

The ACTG 175 trial and the WIHS cohort study

To provide an illustration of the methods in a practical example, we used data from the ACTG 175 trial [16, 17] and the WIHS cohort study [18]. ACTG 175 randomized 2139 HIV-positive individuals to four antiretroviral regimens: zidovudine monotherapy, zidovudine + didanosine, zidovudine + zalcitabine, and didanosine monotherapy. Individuals were followed from their enrollment date (between December 1991 - October 1992) through November 30, 1994. Following earlier work [42, 43, 41], we combined the three combination therapy groups (a = 1, 1607 individuals) and compared them against zidovudine monotherapy (a = 0, 532 individuals). We used the CD4 cell count at 20 +/− 5 weeks post-randomization as the outcome; we report all cell counts in the data application as cells/mm³. We used the WIHS data to create a sample of trial-eligible non-randomized individuals from the target population. WIHS followed 4129 women either living with HIV or at risk for HIV infection in the U.S. between October 1994 and December 2012. We only used baseline covariate information from WIHS; outcome information on CD4 cell count at 20 +/− 5 weeks (the outcome assessment time-point of the trial) is unavailable in WIHS and the treatment regimes are not necessarily the same as those assigned in ACTG 175.

To create the sample of individuals from the target population and conduct transportability analyses we had to take several steps to harmonize the ACTG 175 and WIHS data. We restricted the ACTG 175 data to women (368 individuals) because the target population only included women. To better align the ACTG 175 and WIHS data, we used the first wave of the WIHS population of 2323 women that enrolled in 1994–1995, which was closest to the ACTG 175 enrollment period. To the extent possible, we restricted the WIHS sample to women who met the trial eligibility criteria. ACTG 175 required a screening CD4 cell count of between 200 and 500 and reported pre-treatment CD4 cell counts that were the average of two pre-treatment measurements (excluding the screening measurement) [16]. Because CD4 values fluctuate, the pre-treatment CD4 cell counts in ACTG 175 were not necessarily between 200 and 500; the screening CD4 cell counts had to be between 200 and 500, but the actual screening measurements were not available to us. Furthermore, the screening process of ACTG 175 was not followed in WIHS. In the analyses presented here, we restricted the WIHS sample to women with a baseline CD4 cell count between 200 and 500 (reflecting the trial eligibility criteria) and did not use CD4 cell counts or other baseline lab values in analyses because the screening measurements were not available in the ACTG 175 data, and available pre-treatment CD4-counts may have been affected by selection during screening (because of the requirement of a screening CD4 cell count between 200 and 500).

Because the baseline CD4 cell count is highly predictive of the CD4 cell count after 20 +/− 5 weeks, and may modify the treatment effect, extending inferences from the ACTG 175 to the population underlying WIHS without using baseline CD4 cell count data necessitates the use of sensitivity analysis, using the methods proposed earlier in the manuscript. In other words, the analyses presented here only used data on age, weight, and race in all models. These covariates are unlikely to be adequate for rendering the randomized and non-randomized individuals exchangeable, necessitating the use of sensitivity analyses. In the Appendix, we present additional analyses in which we attempted to use the pre-treatment CD4 and CD8 cell count information from the ACTG 175 and the WIHS data. These additional analyses require assumptions about the stability of CD4 cell counts over time or the impact of selection based on the screening CD4 cell count in the the ACTG 175; they produced similar results as the ones presented in the main text. We restricted both datasets to observations with complete data on all baseline covariates, including pre-treatment (in ACTG-175) or baseline (in WIHS) CD4 and CD8 cell counts. The final composite dataset for the main analysis included 368 women from ACTG 175 and 841 trial-eligible women from WIHS. In the Appendix, we report comparisons of the baseline covariates between the trial and the target population sample.

Sensitivity analysis methods

We used bias functions that were constant within levels of the baseline covariates. Specifically, we examined u (0) values of −50 to 50, in steps of 10, and varied δ from −30 to 30, in steps of 10, examining all possible (u (0), δ) pairs. For comparison, the standard deviation of the pre-treatment CD4 cell count was 120.7 (across all individuals in the trial); the standard deviation of the post-treatment CD4 cell count was 124.99 for patients assigned to a = 1 and 118.42 for patients assigned to a = 0. We used these bias functions with the estimators provided in Section 3.2.2 to perform sensitivity analyses. We obtained confidence intervals for the potential outcome mean under each treatment using the sandwich variance, accounting for uncertainty in estimating all required models.

Model specification

The sensitivity analysis methods require the specification of models for the outcome mean in each treatment group, the probability of trial participation, and the probability of treatment among randomized individuals. We specified logistic regression models for the probability of being in the trial and the probability of being assigned to combination therapy among randomized individuals. We also specified two linear regression models, one for each treatment group, for the mean of the outcome among randomized individuals. All models used main effects for age, weight, race. We built separate outcome models in each treatment group to allow for heterogeneity of the treatment effect over all baseline covariates included in the models.

5.2 |. Results

The “base case” analyses under mean exchangeability over S (i.e., u (0) = 0 and δ = 0) produced similar results across different estimators (see Appendix for numerical results), suggesting that models were approximately correctly specified. Sensitivity analysis results are summarized in Figure 2 (additional results are presented in the Appendix). In the base case analyses, all estimators suggested improvement in the CD4 cell count at 20 +/− 5 weeks with combination therapy compared to zidovudine monotherapy in the target population. These results, however, were fairly imprecise and fairly sensitive to violations of the exchangeability assumption.

FIGURE 2.

Sensitivity analysis results for transporting inferences from the ACTG 175 trial to the target population of trial-eligible women in WIHS.

AIOW2 = augmented inverse odds weighting estimator with normalized weights; IOW2 = inverse odds weighting with normalized weights; OM = outcome model-based estimator. Results are shown as point estimates (black lines) and corresponding 95% confidence intervals (gray lines) for potential outcome means under treatment a = 1 (solid lines) and a = 0 (dashed lines). Results for IOW1 and AIOW1 were similar to IOW2 and AIOW2, respectively, and are shown in the Appendix.

6 |. DISCUSSION

We propose sensitivity analysis methods for violations of exchangeability assumptions in studies extending inferences from a trial to the population of non-randomized individuals (in the main text) or the overall target population (in the Appendix). The methods rely on specifying bias functions that directly parameterize violations of the required exchangeability assumptions. They can be applied to sensitivity analyses for outcome model-based (g-formula) estimators, probability of trial participation-based estimators (inverse probability or odds weighting), or augmented estimators that combine outcome and probability of participation models. Because of the additive structure of the bias correction, our methods are best-suited to continuous outcomes with unbounded support.

The augmented weighting estimators are appealing for applied work because of their increased robustness to model misspecification. Because all methods depend critically on the specification of bias functions, which will often be highly speculative, some investigators might consider the double robustness property to be less compelling in the context of sensitivity analyses than in settings in which conditional mean exchangeability over S holds. Even so, the augmented weighting estimators may be preferred compared with non-augmented weighting estimators because of improved efficiency; as illustrated in our analyses of data from the HALT-C trial (this pattern was somewhat less clear in the ACTG 175 trial, perhaps due to the presence of individuals with outlying outcome values and imbalanced treatment groups in the trial). Furthermore, sensitivity analyses can be repeated with different bias functions to examine whether the choice of function leads to different conclusions regarding sensitivity to assumption violations.

An attractive aspect of our sensitivity analysis approach is that it does not require detailed background knowledge about unknown or unmeasured variables. Instead, it only requires expert judgments about the magnitude of the aggregate bias that these variables could induce. These judgments can be informed by examining readily available data on the variation of treatment effects among subgroups defined in terms of observed variables in the data at hand or external sources, including observational studies; the variation of treatment effects across studies examining similar interventions and outcomes in different populations (e.g., as assessed in meta-analyses); or the variation of the mean outcome under each treatments across populations and population subgroups. The benefit of our approach becomes clear when compared against approaches that require the specification of models for the distribution of unmeasured variables and the associations between unmeasured and measured variables (e.g., [12, 13]). These alternative approaches have multiple sensitivity parameters and require detailed background knowledge about sources of effect heterogeneity; such knowledge is often unavailable because empirical studies typically do not allow the precise assessment of effect modification [44, 45].

Some readers might find our sensitivity analysis approach unsatisfactory because it does not provide a single point estimate, and instead produces a range of results and associated confidence limits under possible violations of the exchangeability assumption between randomized and non-randomized individuals [10]. We believe that this is a desirable feature of our approach: when conditional exchangeability over S does not hold, the data do not contain adequate information to identify the causal quantities of interest, and at best we can hope to examine how our conclusions would be affected by different violations of our assumptions. In a sense, we view sensitivity analysis as a way to encourage “inferential humility” when extending causal inferences from a trial to a target population: sensitivity analysis highlights that the range of results compatible with the data, when considering violations of assumptions, is much broader than it appears when solely considering uncertainty due to sampling variability.