Sensitivity to Unobserved Confounding in Studies with Factor-structured Outcomes

Abstract

In this work, we propose an approach for assessing sensitivity to unobserved confounding in studies with multiple outcomes. We demonstrate how prior knowledge unique to the multi-outcome setting can be leveraged to strengthen causal conclusions beyond what can be achieved from analyzing individual outcomes in isolation. We argue that it is often reasonable to make a shared confounding assumption, under which residual dependence amongst outcomes can be used to simplify and sharpen sensitivity analyses. We focus on a class of factor models for which we can bound the causal effects for all outcomes conditional on a single sensitivity parameter that represents the fraction of treatment variance explained by unobserved confounders. We characterize how causal ignorance regions shrink under additional prior assumptions about the presence of null control outcomes, and provide new approaches for quantifying the robustness of causal effect estimates. Finally, we illustrate our sensitivity analysis workflow in practice, in an analysis of both simulated data and a case study with data from the National Health and Nutrition Examination Survey (NHANES).

1. Introduction

Large observational datasets often include measurements of multiple outcomes of interest. For example, in high-throughput biology, the goal might be to understand the effect of a treatment on multiple biomarkers (Leek and Storey, 2007; Zhao et al., 2018), or in patient-centered epidemiologic studies, researchers might study the potential impact of health recommendations on multiple disease-related outcomes (Sánchez et al., 2005; Kennedy et al., 2019). While it is always possible to analyze each outcome separately, there has been a recent emphasis on the importance of techniques for simultaneously inferring the effect of an intervention on multiple outcomes (VanderWeele, 2017).

As in any observational study, the validity of multi-outcome causal inference rests on untestable and often implausible assumptions about unconfoundedness. As such, methods which explore the sensitivity and robustness of effects to assumptions about unconfoundedness are increasingly recognized as a crucial part of any rigorous analysis. In this paper, we demonstrate how to leverage prior knowledge to strengthen causal conclusions from observational datasets with multiple outcomes. We propose a sensitivity analysis unique to multi-outcome inference, which leads to characterizations of the robustness of causal effects that are both simpler and sharper than those that can be achieved by analyzing each outcome separately. We focus on linear factor models of the outcomes where we 1) establish bounds on the magnitude of unobserved confounding bias for all outcomes as a function of a single interpretable sensitivity parameter under an assumption about shared confounding, 2) provide novel theoretical results that demonstrate how assumptions about null control outcomes inform the sensitivity analysis and 3) provide practical guidance and a workflow for sensitivity analysis for multi-outcome studies. We demonstrate this workflow in simulation and an analysis of the effect of light drinking on multiple biomarkers for health.

After reviewing related literature in Section 1.1, we introduce the problem setting and the challenges in multi-outcome causal inference (Section 2). In Section 3, we establish the framework for our proposed sensitivity analysis with multivariate outcomes and introduce theoretical results about confounding bias for a model in which the expected outcomes are linear in the unobserved confounders (given observed covariates). We provide additional novel theories demonstrating how null control outcomes can be leveraged in conjunction with our sensitivity analysis and how to characterize the robustness of treatment effect estimates. We illustrate the theoretical insights with a Bayesian implementation of our sensitivity analysis on simulated data in Section 4. In Section 5, we discuss the interpretation and calibration of sensitivity parameters. In Section 6, we illustrate our approach on a real-world example of the effects of light drinking on health measures using data from the National Health and Nutrition Examination Study (NHANES).

1.1. Related Literature

There are a number of methods for causal inference with multiple outcomes, although the majority of these appear in the context of randomized control trials (Freemantle et al., 2003; Mattei et al., 2013). In the context of observational studies, Sammel et al. (1999) propose a multivariate linear mixed effects model for multi-outcome causal inference, Thurston et al. (2009) consider a Bayesian generalization of multiple outcomes which are nested across different domains, and Sánchez et al. (2005) review a variety of structural equation models for multiple outcomes with application to epidemiological problems. These works have the important caveat that they typically assume a version of the “no unobserved confounding” (NUC) assumption. More recently, Kennedy et al. (2019) develop a nonparametric doubly robust method for estimation and hypothesis testing of scaled treatment effects on multiple outcomes, but again assume NUC.

When unobserved confounding is expected to be an issue, certain assumptions about additional outcomes can sometimes be used to identify the effects of alternative primary outcomes. As one important example, additional outcomes called “proxy confounders” or “null control outcomes” can sometimes be used to identify causal effects for a set of primary outcomes (Shi et al., 2020; Tchetgen et al., 2020). Relatedly, Wang et al. (2017) establish identification assumptions in a linear Gaussian model with multiple outcomes when the non-null effects are assumed to be sparse. Although not explicitly framed in causal language, a closely related line of work explores approaches for de-biasing estimates in the presence of confounders, for example when there are batch effects in high-throughput biological datasets (e.g. Gagnon-Bartsch and Speed, 2012; Gagnon-Bartsch et al., 2013). These works all focus on various assumptions that can be made to point identify causal effects for each outcome.

In contrast, sensitivity analyses are useful for explicitly relaxing such identification assumptions. There are a variety of different approaches for assessing sensitivity to potential unmeasured confounders in single-treatment, single-outcome settings (Rosenbaum and Rubin, 1983; Tan, 2006; Franks et al., 2019; Cinelli and Hazlett, 2020; Veitch and Zaveri, 2020). In the multi-outcome setting, Fogarty and Small (2016) consider sensitivity analysis for multiple comparisons in matched observational studies using weighting estimators. They focus on the implications of the key fact that omitted variable biases across multiple outcomes are connected through the shared effect of the unmeasured confounders on the treatment. Under a similar matched pairs framework, Rosenbaum (2021) considers a sensitivity analysis for a single primary outcome and shows how a null control outcome can sometimes increase the evidence for the robustness of the primary outcome. One potential concern with their sensitivity analyses for weighting estimators is that the sensitivity analyses are implicitly based on the overly conservative assumption that unmeasured confounders explain nearly all the variation in the outcomes. We address this concern by developing a sensitivity analysis in which we explicitly account for the effect of unmeasured confounders on the outcomes.

Building on existing parametric models for multiple outcomes (e.g. Sammel et al., 1999; Sánchez et al., 2005; Thurston et al., 2009), we propose an outcome model with latent variables which account for the residual correlations between outcomes. Unlike these previous works, we expressly consider the possibility that these latent variables might correspond to potential confounders, in that they may also correlate with the treatment. To account for the potential dependence between latent variables and treatment, we use a so-called latent variable sensitivity analysis (Rosenbaum and Rubin, 1983). Typically, in a latent variable sensitivity analysis, sensitivity parameters govern the functional relationship between a latent variable and both the outcome and the treatment. For instance, Cinelli and Hazlett (2020) propose a particularly intuitive latent variable sensitivity analysis for single-treatment, single-outcome problems, with two sensitivity parameters corresponding to the fraction of outcome residual variance explained by latent confounders and the fraction of treatment residual variance explained by confounders. We generalize their approach to multi-outcome settings, paralleling a sensitivity analysis strategy developed in the context of causal inference with multiple treatments and a scalar outcome (Zheng et al., 2021b).

2. Setup

We let $Y$ denote a q-vector of outcomes, $T$ a scalar treatment variable, $U$ an $m$-vector of potential unobserved confounders, and $X$ any observed pre-treatment covariates. The goal of multi-outcome causal inference is to infer the effect of a scalar treatment on the q-dimensional outcomes. In this setting, we define a class of causal estimands as the population average treatment effect (PATE) for any linear combination of the outcomes, $a^{\prime }Y$, as

$${\mathrm{P}\mathrm{A}\mathrm{T}\mathrm{E}}_{a,t_1,t_2}:=E\left[a^{\prime }Y\mid do\left(t_1\right)\right]-E\left[a^{\prime }Y\mid do\left(t_2\right)\right],$$ (1)

where the $do$-operator indicates that we are intervening to set the level to $t$ rather than merely conditioning on $t$ (Pearl, 2009). Most commonly, we take $a=e_j$ for some $j$, so that $PAT{E}_{e_j,t_1,t_2}$ is simply the causal effect on measured outcome $j$ in the original coordinates. In some cases, other linear combinations may be of interest, for example when there is not enough power to detect differences in individual outcomes, but there are detectable and interesting differences for linear combinations of outcomes (e.g. see Cook et al., 2010). Relatedly, it is often desirable to define the PATE on standardized outcomes, so that each dimension of $Y$ has unit variance e.g. $a_j=e_j/sd\left(Y_j\right)$ (Kennedy et al., 2019).

In general, the PATEs cannot be identified from observational data without assumptions about the absence of unobserved confounders. If, in addition to $X$, the unmeasured confounders $U$ were observed, then the following three assumptions would be sufficient to identify the causal effects:

Assumption 1 (Latent unconfoundedness).

$U$ and $X$ block all backdoor paths between $T$ and $Y$ (Pearl, 2009).

Assumption 2 (Latent positivity).

$f(T=t\mid U=u,X=x)>0$ for all $u$ and $x$.

Assumption 3 (SUTVA).

There is no hidden version of the treatment and no interference between units (Rubin, 1980).

Since $U$ is not observed, we cannot identify causal effects without additional assumptions. Instead of making potentially implausible assumptions about NUC, we advocate for reasoning about the strength of potential unobserved confounding. Here, we argue that sensitivity analysis can be a useful tool for characterizing how robust our multiple causal conclusions are to such confounding.

3. Sensitivity Analysis with Multiple Outcomes

Let $f(y\mid do(t\left)\right)$ denote the distribution of $y$ if we were to intervene to set the level of treatment to $t$. As shown in D’Amour (2019) and Zheng et al. (2021b), $f(y\mid do(t\left)\right)$ can be written as $f_{\psi }(y\mid do(t\left)\right)={\int }_{𝒰} f_{{\psi }_Y}(y\mid t,u)\left[\int f_{{\psi }_T}(u\mid \tilde{t})f\left(\tilde{t}\right)d\tilde{t}\right]du$ where $f_{{\psi }_Y}(y\mid t,u)$ is the full conditional outcome density, $f_{{\psi }_T}(u\mid t)$ is a proposed distribution for unobserved confounders given the treatment and $f\left(t\right)$ is the marginal density of the treatment. In the multivariate outcome setting, ${\psi }_Y$ are sensitivity parameters which govern the relationship between the $q$-dimensional outcomes and the $m$-dimensional unobserved confounders, whereas ${\psi }_T$ are parameters which govern the relationship between the scalar treatment and unobserved confounders.

A potential difficulty with multi-outcome sensitivity analysis is that the dimension of ${\psi }_Y$ rscales with the number of outcomes. However, with multiple outcomes, there is also often additional prior knowledge that can be brought to bear on the problem. We consider two such assumptions that mitigate the challenges associated with having to reason about high-dimensional sensitivity parameters, and can, in some cases, tighten bounds beyond what we would obtain from analyzing outcomes one at a time. First, we explore the implications of assuming that confounding is shared across multiple outcomes. Second, we consider how the sensitivity analysis changes under the additional assumption that there is no causal effect for some specific outcomes (null controls).

To begin, in Section 3.1 we propose our model for multi-outcome causal inference and then in Section 3.2, we establish our sensitivity parameterization and derive worst-case bounds on the causal effects for all outcomes. For this model, under standard identifiability conditions for factor models, we show that the relevant dimensions of ${\psi }_Y$ are identified and the bound on the magnitude of the confounding bias for all outcomes depends on a single unknown scalar sensitivity parameter governing the strength of the confounder-treatment relationship. In Section 3.4, we characterize the joint relationship between the confounding biases across outcomes by establishing how assumptions about null control outcomes constrain the set of plausible causal conclusions. Finally, we discuss different robustness measures in Section 3.5.

3.1. A Multi-outcome Model with Factor Confounding

In this paper, we focus on models for observed outcomes which have factor-structured residuals. We seek to leverage the factor-structured residuals for assessing the sensitivity of causal conclusions to assumptions about unobserved confounders. We define the conditional mean and covariance of the observed outcome distribution to be

$$E[Y\mid T=t,X=x]:=\check{g}(t,x)\text{and}Cov(Y\mid T=t,X=x)={\mathrm{\Gamma }}_{t,x}{\mathrm{\Gamma }}_{t,x^{\prime }}+\mathrm{\Delta }\text{,}$$ (2)

where ${\mathrm{\Gamma }}_{t,x}$ are rank- $m$ factor loadings and $\mathrm{\Delta }$ is a constant diagonal matrix. For now, we assume the factor loadings can vary with both $t$ and $x$, but later make the stronger assumption that the factor loadings are constant.

There are several causal models consistent with the observed data moments in (2). Here, we propose a causal model which is explicitly parameterized in terms of latent factors, $U$, motivated by the idea that unmeasured confounders can induce residual correlation between outcomes. Throughout, we assume the following structural equation model:

$$U={ϵ}_U$$ (3)

$$T=f_{ϵ}(X,U)$$ (4)

$$Y=g\left(T,X\right)+{\mathrm{\Gamma }}_{t,x}{\mathrm{\Sigma }}_{u\backslash t,X}^{-1/2}U+{ϵ}_Y,$$ (5)

where $U$ is mean zero and has identity covariance without loss of generality and we define ${\mathrm{\Sigma }}_{u\mid t,x}:=\mathrm{C}\mathrm{o}\mathrm{v}(U\mid T=t,X=x)$. ${ϵ}_Y$ is mean zero with diagonal covariance $\mathrm{\Delta }$ independent of $T,X$ and $U$. The proposed structural model satisfies (2) and implies that $E[Y\mid do(T=t),X=x]:=g(t,x)=g(t,x)-{\mathrm{\Gamma }}_{t,x}{\mathrm{\Sigma }}_{u|t,x}^{-1/2}{\mu }_{u\mid t,x}$ is the intervention mean, where ${\mu }_{u\mid t,x}:=E[U\mid T=t,X=x]$. The intervention mean differs from the observed data mean by an unidentifiable bias, ${\mathrm{\Gamma }}_{t,x}{\mathrm{\Sigma }}_{u\mid t,x}^{-1/2}{\mu }_{u\mid t,x}$. We say that $U$ are potential confounders because they are upstream of the treatment and outcomes, but are only truly confounding if ${\mathrm{\Gamma }}_{t,x}{\mathrm{\Sigma }}_{u\mid t,x}^{-1/2}{\mu }_{u\mid t,x}$ is non-zero.

Definition 1 (Potential confounding).

$U$ are potential confounders in that they are possible causes of $T$ and $Y$. Further, $T$ and $Y$ are not causes for any function of $U$.

We cannot test whether latent variables $U$ from the outcome model are potential confounders without the structural equation assumption, since the observed moments in (2) are also consistent with a model in which the latent variables are mediators caused by the treatment¹. However, even if $U$ are not potential confounders, our sensitivity analysis will yield conservative bounds on the treatment effects (see Appendix B).

When model (3)–(5) holds, the sensitivity analysis is well-defined in that the PATEs are identified given the parameters ${\psi }_T=\{{\mu }_{u|t,x},{\mathrm{\Sigma }}_{u|t,x}:t\in 𝒯,x\in 𝒳\}$, which govern the relationship between confounders and the treatment, and ${\psi }_Y=\left\{{\mathrm{\Gamma }}_{t,x}:t\in 𝒯,x\in 𝒳\right\}$, which are the factor loading matrices governing the influence of the unmeasured confounders on each outcome.

3.2. Establishing Sensitivity Bounds

To complete our model specification, we establish an interpretable sensitivity parameterization for ${\psi }_T=\{{\mu }_{u\mid t,x},{\mathrm{\Sigma }}_{u\mid t,x}:t\in 𝒯,x\in 𝒳\}$. Following Cinelli and Hazlett (2020), we propose a parametrization which imposes no restrictions on the observed data and reflects the strength of linear dependence between $T$ and $U$ via the partial correlation, generalized to account for the possibility that each dimension of $U$ can have a different confounding effect on each outcome.

Assumption 4 (Conditional Moments of Potential Confounders).

$U$ is linear in $T$, has constant variance, and is uncorrelated with observed covariates. Further, let $\rho$ be the $m$-dimensional sensitivity vector corresponding to the partial correlation vector between $T$ and $U$ after regressing out $X,\mathrm{C}\mathrm{o}\mathrm{r}\left(T^{\perp X},U^{\perp X}\right)$, where $T^{\perp X}:=T-E[T\mid X]$ and $U^{\perp X}:=U-E[U\mid X]=U$. Then, the conditional mean and covariance of $U$ is

$${\mu }_{u\mid t,x}=\rho \left(t-{\mu }_{t\mid x}\right)/\sigma ,$$ (6)

$$\mathrm{\Sigma }:={\mathrm{\Sigma }}_{u\mid t,x}=I_m-\rho {\rho }^{\prime }$$ (7)

for all $t\in 𝒯$, $x\in 𝒳$, where we define ${\mu }_{t\mid x}:=E[T\mid X=x]$, and ${\sigma }^2:=Var\left(T^{\perp X}\right)=E_X[Var(T\mid X)]$.

Equation (7) is implied by Equation (6) and the constraint that $Cov(U)=I$. Since $\mathrm{\Sigma }$ is positive definite, we have that $\parallel \rho {\parallel }_2^2<1$. To maintain consistency with related single-outcome sensitivity analyses (Cinelli and Hazlett, 2020), we denote $R_{T\sim U\mid X}^2={\parallel \mathrm{C}\mathrm{o}\mathrm{r}\left(T^{\perp X},U^{\perp X}\right)\parallel }_2^2=\parallel \rho {\parallel }_2^2$ to be the squared norm of the partial correlation vector between $T$ and $U$ given $x$, which is identically the partial R-squared based on a linear fit to $T$.

Finally, for the remainder of the paper, we focus on outcome models for which there are no interactions between unobserved confounders and the covariates or the treatment. In the no-interaction model, the residual outcome covariance is constant in $t$ and $x$ and thus is equivalent to the following assumption.

Assumption 5 (Homoscedasticity).

$\mathrm{C}\mathrm{o}\mathrm{v}(Y\mid T=t,X=x)$ is invariant to $t$ and $x$.

Together, Assumptions (4) and (5) imply that the factor loadings ${\mathrm{\Gamma }}_{t,x}=\mathrm{\Gamma }$ are also invariant to $t$ and $x.$² With these additional assumptions, we have the following bound on the treatment effect for all outcomes:

Theorem 1.

Assume model (3)–(5) with ${\psi }_T$ defined by (6)–(7), and Assumptions 1–5, and let ${\sigma }_{a^{\prime }Y}^2:=Var\left(a^{\prime }Y\mid T=t,X=x\right)$. The partial fraction of outcome variance explained by confounders is $R_{a^{\prime }Y\sim U\mid T,X}^2:={∥a^{\prime }\mathrm{\Gamma }∥}_2^2/{\sigma }_{a^{\prime }Y}^2$. The confounding bias of ${\mathrm{P}\mathrm{A}\mathrm{T}\mathrm{E}}_{a,t_1,t_2},{\mathrm{B}\mathrm{i}\mathrm{a}\mathrm{s}}_{a,t_1,t_2}$ is equal to $\frac{a^{\prime }\mathrm{\Gamma }{\mathrm{\Sigma }}^{-1/2}\rho }{\sigma }\left(t_1-t_2\right)$ and it is bounded by

$${\mathrm{B}\mathrm{i}\mathrm{a}\mathrm{s}}_{a,t_1,t_2}^2\le \frac{{\left(t_1-t_2\right)}^2}{{\sigma }^2}\left(\frac{R_{T\sim U\mid X}^2}{1-R_{T\sim U\mid X}^2}\right){∥a^{\prime }\mathrm{\Gamma }∥}_2^2\le \frac{{\left(t_1-t_2\right)}^2}{{\sigma }^2}\left(\frac{R_{T\sim U\mid X}^2}{1-R_{T\sim U\mid X}^2}\right){\sigma }_{a^{\prime }Y.}^2$$ (8)

The first bound is achieved when $\rho$ is collinear with $a^{\prime }\mathrm{\Gamma }$.

We can immediately see that the second bound is achieved when $R_{a^{\prime }Y\sim U\mid T,X}^2=1$ by recognizing that the squared-bias can equivalently be written as $\frac{{\left(t_1-t_2\right)}^2}{{\sigma }^2}\left(\frac{R_{T\sim U\mid X}^2}{1-R_{T\sim U\mid X}^2}\right)R_{a^{\prime }Y\sim U\mid T,X}^2{\sigma }_{a^{\prime }Y}^2$. The theorem implies that the true treatment effect for contrast $a^{\prime }Y$ lies in the interval $a^{\prime }{E}_X\left[g\left(t_1,x\right)-g\left(t_2,x\right)\right]\pm \frac{t_1-t_2}{\sigma }\sqrt{\frac{R_{T\sim U\mid X}^2}{1-R_{T\sim U\mid X}^2}}{∥a^{\prime }\mathrm{\Gamma }∥}_2$ and as a consequence, for any $R_{T\sim U\mid X}^2$, the bias for outcome $j$ is proportional to the norm of the $j$th row of $\mathrm{\Gamma }$.

In the following corollary, we establish a global bound on the biases over all the outcome contrasts $a^{\prime }Y$ with $\parallel a{\parallel }_2=1$.

Corollary 1.1.

Let $d_1$ be the largest singular value of $\Gamma$. For all unit vectors $a$, the confounding bias, ${\mathrm{B}\mathrm{i}\mathrm{a}\mathrm{s}}_{a,t_1,t_2}$, is bounded by

$${\mathrm{B}\mathrm{i}\mathrm{a}\mathrm{s}}_{a,t_1,t_2}^2\le \frac{{\left(t_1-t_2\right)}^2}{{\sigma }^2}\frac{R_{T\sim U\mid X}^2}{1-R_{T\sim U\mid X}^2}{d}_1^2,$$ (9)

with equality when $a=u_1^{\mathrm{\Gamma }}$, the first left singular vector of $\Gamma$, and when $\rho$ is collinear with $v_1^{\mathrm{\Gamma }}$, the first right singular vector of $\Gamma$. There is no confounding bias for the causal effect estimates of outcome $a^{\prime }Y$ when $a\in Null\left({\mathrm{\Gamma }}^{\prime }\right)$.

For $a=u_1^{\mathrm{\Gamma }},a^{\prime }Y$ corresponds to the linear combination of outcomes that is most correlated with confounders. In contrast, when $a$ is in the null space of ${\mathrm{\Gamma }}^{\prime }$, ${\mathrm{P}\mathrm{A}\mathrm{T}\mathrm{E}}_a$ is identified because $a^{\prime }Y$ is uncorrelated with $U$.

3.3. Factor Confounding

Under certain additional assumptions about the factor model, we can identify $\mathrm{\Gamma }$ up to rotation, so that the bounds in Theorem 1 and Corollary 1.1 only depend on the single unidentified sensitivity parameter, $R_{T\sim U\mid X}^2$.

Fact 1.

Assume outcome model (5) with homoscedastic residuals (Assumption 5). If $\Gamma$ is rank $m$ and there remain two disjoint matrices of rank $m$ after deleting any row of $\Gamma$, then $\Gamma$ is identifiable up to rotations from the right (Anderson and Rubin, 1956).

The bound in Theorem 1 depends on $\mathrm{\Gamma }$ only through ${∥a^{\prime }\mathrm{\Gamma }∥}_2$ which is invariant to rotations. Likewise, the bound in Corollary 1.1 depends on the first singular value of $\mathrm{\Gamma }$ which is rotation-invariant. In order for $\mathrm{\Gamma }$ to be identified up to rotation, we must have $(q-m{)}^2-q-m\ge 0$ and each confounder must influence at least three outcomes (Anderson and Rubin, 1956). We group the conditions under which the factor model (3)–(5) yields a sensitivity analysis entirely parameterized by ${\psi }_T=\rho$ into the following assumption.

Assumption 6 (Factor confounding).

The proposed causal model satisfies factor confounding. We say that a causal model satisfies factor confounding if the outcomes follow the model proposed in equation (5), $U$ are potential confounders (Definition 1), and $\mathrm{\Gamma }$ is identifiable up to rotations. We say that a model satisfies factor confounding for outcome $a^{\prime }Y$ if ${∥a^{\prime }\mathrm{\Gamma }∥}_2$ is identifiable. Factor confounding for outcome $a^{\prime }Y$ implies that the partial fraction of outcome variance explained by confounders, $R_{a^{\prime }Y\sim U\mid T,X}^2$, is identifiable.

There are some useful ways that practitioners can reason about the plausibility of factor confounding. In particular, since factor confounding is violated if there are confounders that influence fewer than three outcomes, practitioners should consider carefully whether there are important unmeasured confounders that might influence only one or two outcomes. While the bulk of our analysis is done under the factor confounding assumption, even when factor confounding is violated, we can still apply our sensitivity analysis, albeit with more conservative bounds on the causal effects. As such, we view factor confounding as a useful “reference assumption” that can help establish informative bounds on the causal effects. For now, we assume factor confounding, and explore additional relaxations of Assumption 6 in Appendix B.

3.4. Null Control Outcomes

In this section, we establish how additional assumptions about null control outcomes constrain the shared sensitivity vector, $\rho$, and thus reduce the size of the partial identification regions for the causal effects of the other outcomes. Under null control assumptions, we fix the biases for a set of so-called null control outcomes to match the observed effect under NUC, so that the causal effects for the null control outcomes are zero after accounting for confounding biases. Such assumptions add important context, in particular because Theorem 1 only establishes marginal bounds for any treatment contrast, but does not account for the dependence in the omitted variable bias across outcomes. Null control assumptions reveal the joint relationship between biases across outcomes. Our results complement those from Rosenbaum (2021), who explore sensitivity analysis for matching estimators and demonstrate that null control outcomes can make tests of significance on the primary outcome more robust to confounding.

For a fixed treatment contrast $t_1$ versus $t_2$, and with a slight abuse of notation, we let $T$ correspond to the $q$-vector of PATEs on each of the measured outcomes and let $\tau$ denote the $q$-vector of PATEs under NUC. Let $𝒞\subset \{1,\dots ,q\}$ be a set of indices for $c<q$ null control outcomes for which there is assumed to be no causal effect of the treatment on these $c$ measured outcomes, that is, ${\tau }_j=0$ for any $j\in 𝒞$. For these null control outcomes, the corresponding c-vector of treatment effects under NUC, ${\tau }_{𝒞}$, must equal the corresponding confounding biases. Since the bias is a function of the sensitivity vector $\rho$, we have that ${\tau }_{𝒞}=\frac{t_1-t_2}{\sigma \sqrt{1-R_{T\sim U\mid X}^2}}{\mathrm{\Gamma }}_{𝒞}\rho$ where ${\mathrm{\Gamma }}_{𝒞}$ is a $c\times m$ matrix equal to the $c$ rows of $\mathrm{\Gamma }$ corresponding to null control outcomes. This equation implies that ${\check{\tau }}_{𝒞}$ must be in the column space of ${\mathrm{\Gamma }}_{𝒞}$ and also implies a lower bound on the fraction of confounding variation in the treatment, $R_{T\sim U\mid X}^2$.

Proposition 1.

Assume model (3)–(5) with sensitivity parameterization (6)–(7) and Assumptions 1–5. Further, suppose there are $c$ null control outcomes, $Y_j$, such that ${\check{\tau }}_j=0$ for $j\in 𝒞$. Then, ${\tau }_{𝒞}$ must be in the column space of ${\mathrm{\Gamma }}_{𝒞}$. In addition, the fraction of variation in the treatment due to the confounding is lower bounded by

$$R_{T\sim U\mid X}^2\ge R_{\mathrm{m}\mathrm{i}\mathrm{n},𝒞}^2:=\frac{{\sigma }^2{\parallel {\mathrm{\Gamma }}_{𝒞}^{†}{\check{\tau }}_{𝒞}\parallel }_2^2}{{\left(t_1-t_2\right)}^2+{\sigma }^2{\parallel {\mathrm{\Gamma }}_{𝒞}^{†}{\check{\tau }}_{𝒞}\parallel }_2^2},$$ (10)

where ${\mathrm{\Gamma }}_{𝒞}^{†}$ denotes the pseudoinverse of ${\mathrm{\Gamma }}_{𝒞}.R_{\mathrm{m}\mathrm{i}\mathrm{n},𝒞}^2$ is identifiable under factor confounding (Assumption 6).

When the number of null control outcomes is smaller than the rank of $\mathrm{\Gamma }$, then ${\check{\tau }}_{𝒞}$ is automatically in the column space of $\mathrm{\Gamma }$. In order to correct for the biases of null control outcomes, confounding must explain at least $R_{min\text{,}𝒞}^2$ of the residual treatment variance. Moreover, for any assumed $R_{T\sim U\mid X}^2\ge R_{min,𝒞}^2$, the assumption of null controls constrains the space of possible effects for the non-null outcomes. We formalize this below.

Theorem 2.

Under the assumptions established in Proposition 1, for any value of $R_{T\sim U\mid X}^2\ge R_{min,𝒞}^2$, the confounding bias for the treatment effect on outcome $a^{\prime }Y$ is in the interval

$${\mathrm{B}\mathrm{i}\mathrm{a}\mathrm{s}}_{a,t_1,t_2}\in \left[a^{\prime }\mathrm{\Gamma }{\mathrm{\Gamma }}_{𝒞}^{†}{\check{\tau }}_{𝒞}\pm {∥a^{\prime }\mathrm{\Gamma }{P}_{{\mathrm{\Gamma }}_{𝒞}}^{\perp }∥}_2\sqrt{\frac{{\left(t_1-t_2\right)}^2}{{\sigma }^2}\left(\frac{R_{T\sim U\mid X}^2}{1-R_{T\sim U\mid X}^2}-\frac{R_{min,𝒞}^2}{1-R_{min,𝒞}^2}\right)}\right],$$ (11)

where $P_{{\mathrm{\Gamma }}_c}^{\perp }=I_m-{\mathrm{\Gamma }}_{𝒞}^{†}{\mathrm{\Gamma }}_{𝒞}$ is the $m\times m$ projection matrix onto the space orthogonal to the row space of ${\mathrm{\Gamma }}_{𝒞}$. Under Assumption 6, $R_{T\sim U\mid X}^2$ is the only unidentifiable parameter.

Note that the ignorance region is no longer centered at $a^{\prime }\check{\tau }$ but instead $a^{\prime }\check{\tau }-a^{\prime }\mathrm{\Gamma }{\mathrm{\Gamma }}_{𝒞}^{†}{\check{\tau }}_{𝒞}$, where $a^{\prime }\mathrm{\Gamma }{\mathrm{\Gamma }}_{𝒞}^{†}{\check{\tau }}_{𝒞}$ is the bias correction under the null controls assumption. Theorem 2 indicates that whenever ${\mathrm{\Gamma }}_{𝒞}$ is of rank $m$ or whenever we assume $R_{\mathrm{\Gamma }\sim U\mid X}^2=R_{min,𝒞}^2$, treatment effects for all outcomes are identifiable under factor confounding. A direct comparison of the ignorance regions from Theorem 1 and Theorem 2 indicates that after incorporating null control outcomes, for any fixed $R_{T\sim U\mid X}^2$ the width of the ignorance region is reduced by a multiplicative factor of

$$\sqrt{1-\left(\frac{R_{min,𝒞}^2}{1-R_{min,𝒞}^2}/\frac{R_{T\sim U\mid X}^2}{1-R_{T\sim U\mid X}^2}\right)}\frac{{\parallel a^{\prime }\mathrm{\Gamma }{P}_{{\mathrm{\Gamma }}_{𝒞}}^{\perp }\parallel }_2}{{∥a^{\prime }\mathrm{\Gamma }∥}_2}\le 1.$$ (12)

From Equation (12), it is evident that null controls reduce the width of the worst-case ignorance region in two ways. The first factor under the radical is due to the fact that only $R_{T\sim U\mid X}^2-R_{min,𝒞}^2$ of the treatment variance can be due to confounders which are uncorrelated with the null control outcomes. This factor reduces the width of the ignorance regions for all non-null outcomes by an equal proportion. As a special case, when all the unobserved confounders are correlated with the null control outcomes, then $R_{T\sim U\mid X}^2=R_{min,𝒞}^2$ and the treatment effects for all outcomes are identified. In contrast, the second factor depends on the specific outcome of interest, $a^{\prime }Y$. The ignorance region shrinks the most for outcomes that are mostly correlated with the same set of confounders as the null control outcomes. Mathematically, when $a^{\prime }\mathrm{\Gamma }$ is in the row space of ${\mathrm{\Gamma }}_{𝒞}$, the treatment effect of $a^{\prime }Y$ is identified under factor confounding. When $a^{\prime }\mathrm{\Gamma }$ is orthogonal to the row space of ${\mathrm{\Gamma }}_{𝒞},\frac{{\parallel a^{\prime }\mathrm{\Gamma }{P}_{{\mathrm{\Gamma }}_{𝒞}}^{\perp }\parallel }_2}{{∥a^{\prime }\mathrm{\Gamma }∥}_2}=1$, so that the confounders affecting the null control outcomes are independent of the confounders affecting $a^{\prime }Y$, and thus there is no further reduction of the ignorance region. In summary, the best null control outcomes are those which have large confounding biases (and hence large values of $R_{min,𝒞}^2$) and also have similar outcome-confounder associations with the other outcomes of interest. We illustrate these facts in a simulation study in Section 4.

3.5. Robustness

A common strategy for characterizing the robustness to confounding is to identify the “smallest” sensitivity parameter(s) which nullifies the causal effect. For example, for single-outcome inference Cinelli and Hazlett (2020) define the robustness value, $R{V}_a^1$, as the smallest value of $\mathrm{m}\mathrm{a}\mathrm{x}(R_{T\sim U\mid X}^2,R_{a^{\prime }Y\sim U\mid T,X}^2)$, needed to change the sign of the effect³ and define an “extreme robustness value” $(XR{V}_a)$ as the smallest value of $R_{T\sim U\mid X}^2$ needed to change the sign without any constraints on $R_{a^{\prime }Y\sim U\mid T,X}^2$ Cinelli and Hazlett, 2022). The $XR{V}_a$ is a more conservative measure of robustness than $R{V}_a^1$, since the smallest value of $R_{T\sim U\mid X}^2$ needed to change the sign of the causal effect is achieved when $R_{a^{\prime }\mathrm{Y}\sim U\mid T,X}^2=1$, that is, all the residual outcome variance is attributable to confounders.

Here, we define the factor confounding robustness value, $R{V}_a^{\mathrm{\Gamma }}$, for outcome $a^{\prime }Y$ as the smallest value of $R_{T\sim U\mid X}^2$ needed to make the causal effect zero under factor confounding. $R{V}_a^{\mathrm{\Gamma }}$ is a more accurate reflection of robustness than $R{V}_a^1$ or $XR{V}_a$ when factor confounding holds, because $R_{a^{\prime }Y\sim U\mid T,X}^2$ is identified under this assumption for all a (Assumption 6). $XR{V}_a\le R{V}_a^{\mathrm{\Gamma }}<1$ and $R{V}_a^{\mathrm{\Gamma }}$ can be either smaller or larger than $R{V}_a^1$, depending on how much variance is attributed to potential confounders from the factor model. When $R_{a^{\prime }Y\sim U\mid T,X}^2={∥a^{\prime }\mathrm{\Gamma }∥}_2^2/({∥a^{\prime }\mathrm{\Gamma }∥}_2^2+a^{\prime }\mathrm{\Delta }a)\ge R{V}_a^1$, then $R{V}_a^{\mathrm{\Gamma }}\le R{V}_a^1$. Conversely, when $R_{a^{\prime }Y\sim U|T,X}^2<R{V}_a^1$, then $R{V}_a^{\mathrm{\Gamma }}>R{V}_a^1.R{V}_a^{\mathrm{\Gamma }}=XR{V}_a$ if and only if $a^{\prime }\mathrm{\Delta }a=0$, that is, the latent factors explain all the residual outcome variance for $a^{\prime }Y$. We demonstrate the relationship between these robustness values in a simulation study in Section 4.

In addition, we can quantify how assumptions about null control outcomes influence $R{V}_a^{\mathrm{\Gamma }}$. First, note that when there is only a single null control, indexed by $c$, we have that $R_{min,𝒞}^2=R{V}_{e_c}^{\mathrm{\Gamma }}$, since $R{V}_{e_c}^{\mathrm{\Gamma }}$ is the smallest fraction of treatment variance needed to nullify outcome $c$. In other words, the total fraction of treatment variance explained by confounders is lower bounded by the robustness value for the null control. We define the “combined robustness value” with null controls, $𝒞$, as

$$R{V}_{a,𝒞}^{\text{Γ}}:=\underset{\rho }{min}\parallel \rho {\parallel }_2^2\text{s}.\text{t}.a^{\prime }\tau =0\text{and}{e}_c^{\prime }\tau =0,\forall c\in 𝒞,$$ (13)

where $e_c$ is the cth canonical basis vector. $R{V}_{a,𝒞}^{\mathrm{\Gamma }}$ corresponds to the minimum fraction of treatment variance explained by unobserved confounders that is required to make the causal effect on $a^{\prime }Y$ equal to zero and satisfy all null control assumptions. Naturally, the minimum fraction of treatment variance explained by confounders needed to satisfy the null control assumptions and nullify $a^{\prime }Y$ must be larger than the minimum fraction needed to nullify just $a^{\prime }Y$.

Theorem 3.

Let $\tau$ denote the vector of PATEs for all outcomes and ${\tau }_{𝒞}$ be the vector of PATEs for null control outcomes under NUC. Under the assumptions established in Proposition 1, the factor confounding robustness value for outcome $a^{\prime }Y$ is

$$R{V}_a^{\mathrm{\Gamma }}=\frac{\omega }{1+\omega },$$ (14)

where $\omega =\frac{{\sigma }^2}{{\left(t_1-t_2\right)}^2}\frac{{\left(a^{\prime }\check{\tau }\right)}^2}{{∥a^{\prime }\mathrm{\Gamma }∥}_2^2}$. The combined robustness value for outcome $a^{\prime }Y$ given null controls $𝒞$ is

$$R{V}_{a,𝒞}^{\mathrm{\Gamma }}=\frac{w_{𝒞}}{1+w_{𝒞}}\ge max(R_{min,𝒞}^2,R{V}_a^{\mathrm{\Gamma }}),$$ (15)

where $w_{𝒞}=\frac{{\sigma }^2}{{(t_1-t_2)}^2}\left[\frac{{(a^{\prime }(\check{\tau }-\mathrm{\Gamma }{\mathrm{\Gamma }}_{𝒞}^{†}{\check{\tau }}_{𝒞}))}^2}{{\Vert a^{\prime }\mathrm{\Gamma }{P}_{{\mathrm{\Gamma }}_{𝒞}}^{\perp }\Vert }_2^2}+{\Vert {\mathrm{\Gamma }}_{𝒞}^{†}{\check{\tau }}_{𝒞}\Vert }_2^2\right]$.

In the following examples, in addition to the combined robustness value $R{V}_{a,𝒞}^{\mathrm{\Gamma }}$, we report another useful summary, $R{V}_{a,𝒞}^{\mathrm{\Gamma }}-R_{min,𝒞}^2$ which corresponds to the additional fraction of variance explained by confounders that is needed to nullify outcome $a^{\prime }Y$ beyond the amount needed to nullify the null control outcomes, $𝒞$.

4. Simulation Study

In this section, we provide intuition for the theoretical results in a simple simulated example without covariates where both the treatment and outcome model are linear and Gaussian. We generate $n=1000$ observations from model (3)–(5) with $m=2$ latent variables, $q=10$ outcomes and fix $T=f_{ϵ}(U)={\beta }^{\prime }U+{ϵ}_T$ in (4) and $g(t)=\tau t$ in (5), where $\tau$ represents the $q-$ vector of causal effects for a unit change in $t$. We choose $\tau$ so that there is no causal effect on the first, second, and tenth outcomes, and a causal effect of one for the other seven outcomes.

The partial correlation vector between $T$ and $U,\rho =\frac{\beta }{\mathrm{V}\mathrm{a}\mathrm{r}(T)}$, is chosen uniformly on a sphere with $\parallel \rho {\parallel }_2^2=R_{T\sim U}^2=0.5$. We also choose $\mathrm{\Gamma }$ with a particularly simple structure, shown in Figure 1(a). After generating data, we infer $\mathrm{\Gamma }$ and $\tau$ using a Bayesian multivariate linear regression model with a factor model structure on the residual covariance using the probabilistic programming language Stan (Stan Development Team, 2022).

Fig. 1.

a) Heatmap of the $\mathrm{\Gamma }$. b) 95% posterior credible regions for the causal effects on each of the ten outcomes for $R_{T\sim U}^2\le 0.5$ based on $n=1000$ simulated observations without any null controls (blue) and with $Y_1$ as a null control (red). With the null control assumption, outcomes 2 and 3 are identifiable because the corresponding rows of $\mathrm{\Gamma }$ are collinear with row 1. Rows 4–6 of $\mathrm{\Gamma }$ are orthogonal to the first row of $\mathrm{\Gamma }$, so there is only a small reduction in the size of the identification region and no change in the midpoint of this region. Multiple robustness values are reported above the intervals, including the single-outcome robustness $(R{V}^1)$, extreme robustness $(XRV)$, and robustness under factor confounding $\left(R{V}^{\mathrm{\Gamma }}\right)$. The row labeled $R{V}_c^{\mathrm{\Gamma }}$ summarizes the robustness under factor confounding with null controls, where the denominator is the combined robustness value defined in (15) and the numerator is the fraction of outcome variance needed to nullify outcome $Y_i$ beyond that induced solely by the null control, $R{V}_{i,c}^{\mathrm{\Gamma }}-R_{min,\text{c}}^2$. The null control assumption significantly increases the robustness of all non-null effects, and is enough to effectively nullify outcomes 2 and 10.

In Figure 1(b), we plot the 95% posterior credible interval for the effect of one unit change in $t$ on each outcome assuming $R_{T\sim U}^2\le 0.5$ (blue) by computing the relevant posterior quantiles of the endpoints of the partial identification region determined by Theorem 1. The outcomes with the largest intervals are those for which the corresponding rows of $\mathrm{\Gamma }$ have the largest magnitudes (i.e. darker colors in Figure 1(a)). Only outcomes 4,7, and 8 are robustly different from zero at $R_{T\sim U}^2\le 0.5$.

We then trace the implications of a null control assumption through the sensitivity model to illustrate the results of Section 3.4. In Figure 1(b) we plot the posterior 95% credible regions under the additional assumption that the first outcome is a null control outcome (red). After incorporating the null control assumption, the posterior credible regions for outcomes 2–10 still include the true causal effects assuming factor confounding and $R_{T\sim U}^2\le 0.5$. Among the non-null outcomes, $Y_6$ is the only outcome with an ignorance region that still includes zero. Since the first three rows of $\mathrm{\Gamma }$ are mutually collinear, fixing the bias of $Y_1$ implies that, with infinite data, the effects for $Y_2$ and $Y_3$ are also identified since ${\mathrm{\Gamma }}_i{P}_{{\mathrm{\Gamma }}_c}^{\perp }=0$ for rows $i\in \{2,3\}$ (Equation (12)). Rows 4–10 are not mutually collinear with the first row of $\mathrm{\Gamma }$ and thus, even with infinite data, the effects remain unidentified. Rows 4–6 of $\mathrm{\Gamma }$ are orthogonal to row 1, which implies that ${\mathrm{\Gamma }}_i{P}_{{\mathrm{\Gamma }}_c}^{\perp }={\mathrm{\Gamma }}_i$ for $i\in \{4,5,6\}$ and thus the midpoints of the intervals remain unchanged for outcomes 4–6. The interval widths still shrink slightly, since only $R_{T\sim U}^2-R_{min,c}^2=0.5-0.37=0.13$ of the treatment variance can be explained by the confounders of $Y_4$ through $Y_6$ after accounting for the null control outcomes. The dot products of ${\mathrm{\Gamma }}_1$ with rows 7 through 10 are all nonzero, which means that the midpoints of the ignorance regions change for outcomes 7 through 10 after incorporating the null control assumption. Consistent with (11), the directional change in the midpoint of ignorance region for outcome $Y_i$ is determined by the sign of the dot product of row ${\mathrm{\Gamma }}_1$ and ${\mathrm{\Gamma }}_i$ (positive for $Y_7$ to $Y_9$ and negative for $Y_{10}$).

Finally, we compare four different measures of robustness to confounding in black above the corresponding intervals: single-outcome robustness $(R{V}^1)$ (Cinelli and Hazlett, 2020), extreme robustness $(XRV)$, robustness under factor confounding ($R{V}^{\mathrm{\Gamma }}$) and robustness under factor confounding with the first outcome as a null control $R{V}_{c=1}^{\mathrm{\Gamma }}$. By definition, $\mathrm{X}\mathrm{R}\mathrm{V}$ is the most conservative (smallest robustness values). In this simulation, $R{V}^1$ is smaller than $R{V}^{\mathrm{\Gamma }}$ for six of the outcomes and larger for the other four. $R{V}^{\mathrm{\Gamma }}$ is a more accurate reflection of the true robustness under factor confounding since we are able to infer the implied values of $R_{Y_i\sim U\mid T}^2$. After correctly incorporating the null control assumption, the robustness of all the outcomes with true non-zero effects increases significantly. The largest increase in robustness occurs for $Y_3$, for which the causal effect is identifiable under the null control assumption. For outcomes $Y_2$ and $Y_{10}$, the other outcomes with no true causal effect, $R{V}_{𝒞}^{\mathrm{\Gamma }}-R_{min,\text{c}}^2$ are close to zero, which means that the null control assumption alone is sufficient to nullify these effects as well.

5. Calibration

For a sensitivity analysis to be of practical value, it is essential to calibrate the magnitude of the sensitivity parameters against interpretable benchmarks. In this section, we briefly describe strategies for calibrating $R_{T\sim U\mid X}^2$, the sole sensitivity parameter of the worst-case bias under factor confounding. In Section 5.2, we propose an alternative sensitivity parameterization tailored to binary treatments which we apply to the analysis in Section 6.

5.1. Calibrating $R_{T\sim U\mid X}^2$

Recall that the magnitude of the sensitivity vector $\rho$ in (6) can be characterized by $R_{T\sim U\mid X}^2$. For linear models, $R_{T\sim U\mid X}^2$ can directly be interpreted as the partial fraction of treatment variance explained by confounders given $X,R_{T\sim U\mid X}^2=\frac{\mathrm{V}\mathrm{a}\mathrm{r}(T\mid X)-\mathrm{V}\mathrm{a}\mathrm{r}(T\mid U,X)}{\mathrm{V}\mathrm{a}\mathrm{r}(T\mid X)}=\parallel \rho {\parallel }_2^2$, and more generally, as the squared norm of the partial correlations between the treatment and confounders (see Section 3.2). Following prior work, we can calibrate $R_{T\sim U\mid X}^2$ in linear models by comparing it to an estimable benchmark. For a reference covariate (or set of covariates), $X_j$, and given all baseline covariates $X_{-j}$, we compute the partial $\mathrm{R}$-squared $R_{T\sim X_j\mid X_{-j}}^2:=\frac{R_{T\sim X_X}^2-R_{\mathrm{\Gamma }\sim X_{-j}}^2}{1-R_{T\sim X_{-j}}^2}$, which serves as a point of comparison for the unknown partial correlation $R_{T\sim U\mid X_{-j}}^2$. If we believed $U$ could be up to $\kappa$ times as informative about the treatment as $X_j$ was, meaning $R_{T\sim U\mid X_{-j}}^2\le \kappa R_{T\sim X_j\mid X_{-j}}^2$, then Cinelli and Hazlett (2020) show that this implies $R_{T\sim U\mid X}^2\le \kappa \frac{R_{T\sim X_j\mid X_{-j}}^2}{1-R_{T\sim X_j\mid X_{-j}}^2}$. Thus, following this work, we use $\kappa \frac{R_{T\sim X_j\mid X_{-j}}^2}{1-R_{T\sim X_j\mid X_{-j}}^2}$ as our benchmark statistic for calibrating $R_{T\sim U\mid X}^2,$ given an appropriately chosen value κ and covariate $X_j$. See Cinelli and Hazlett (2020) for additional details on this and related calibration strategies.

5.2. Calibration for Binary Treatments

For binary-valued treatments, we suggest an alternative sensitivity parameterization which is closely related to the parameterization in the marginal sensitivity model for inference with binary-valued treatments in single-outcome problems (Tan, 2006). Let $e(X)=P(T=1\mid X)$ be the “nominal” propensity score, $e_0(X,U)=P(T=1\mid X,U)$ be the “true” propensity score, and $\lambda (X,U)=\frac{e_0(X,U)/\left(1-e_0(X,U)\right)}{e(X)/(1-e(X))}$ be the multiplicative change in the odds of treatment after accounting for unmeasured confounders. The core assumption of the marginal sensitivity model is that the odds of treatment is bounded by some constant, that is, there exists a $\wedge$ such that ${\mathrm{\Lambda }}^{-1}\le \frac{e_0(X,U)/\left(1-e_0(X,U)\right)}{e(X)/(1-e(X))}\le \mathrm{\Lambda }$ holds with probability one. Under parametric assumptions about the conditional distribution of $U$, we can characterize the full distribution of $\lambda (X,U)$. Although not strictly necessary, in this work we focus our examples on settings in which $U$ is assumed to be Gaussian given $T$ and $X$.

Proposition 2.

Assume $U$ is conditionally Gaussian with mean $\mu U$ and covariance $\mathrm{\Sigma }$ as given in Equations (6) and (7). Further, denote $e(x)=P(T=1\mid X=x)$ and $e_0(x,U)=P(T=1\mid X=x,U)$. Then, for any $\rho$ we have $\lambda (X=x,U)=\frac{e_0(x,U)/\left(1-e_0(x,U)\right)}{e(x)/(1-e(x))}=\mathrm{e}\mathrm{x}\mathrm{p}\left(V_x\right)$, where $V_x=\left(2I_{\mathrm{x}}-1\right)Z,I_x\sim \mathrm{B}\mathrm{e}\mathrm{r}(e(x)),Z\sim N\left({\mu }_{\lambda },{\sigma }_{\lambda }^2\right)$ with ${\mu }_{\lambda }=\frac{1}{2{\sigma }^2}\frac{R_{T\sim U\mid X}^2}{1-R_{T\sim U\mid X}^2}$ and ${\sigma }_{\lambda }^2=\frac{1}{{\mathrm{\sigma }}^2}\frac{R_{T\sim U\mid X}^2}{1-R_{T\sim U\mid X}^2}$.

Proposition 2 states that the log odds ratio, $\mathrm{l}\mathrm{o}\mathrm{g}(\lambda (X=x,U))$, is a two-component mixture of normal distributions. Using this proposition, we can find ${\mathrm{\Lambda }}_{\alpha }$, such that $P\left({\mathrm{\Lambda }}_{\alpha }^{-1}\le \lambda (x,U)\le {\mathrm{\Lambda }}_{\alpha }\right)\ge 1-\alpha$. Unconditional on $x,\lambda (x,U)$ is a two-component mixture with means $\pm {\mu }_{\lambda }$, variance ${\sigma }_{\lambda }^2$ and mixture weights $P(T=1)$ and $P(T=0)$, which we use to compute ${\mathrm{\Lambda }}_{\alpha }$. Since ${\mu }_{\lambda }$ and ${\sigma }_{\lambda }^2$ only depend on $R_{T\sim U\mid X}^2$ and ${\sigma }^2$, we can also derive the robustness of outcome $a^{\prime }Y$ in the $\mathrm{\Lambda }$-parameterization by replacing $R_{T\sim U\mid X}^2$ with $R{V}_a^{\mathrm{\Gamma }}$ in the formulas for ${\mu }_{\lambda }$ and ${\sigma }_{\lambda }^2$. Note that by Proposition (2), there is a one-to-one mapping between the ${\mathrm{\Lambda }}_{\alpha }$ parameterization and the R-squared parameterization. Consequently, we can calibrate ${\mathrm{\Lambda }}_{\alpha }$ by first calibrating $R_{T\sim U\mid X}^2$ against $\kappa \frac{R_{T\sim X_j\mid X_{-j}}^2}{1-R_{T\sim X_j\mid X_{-j}}^2}$, as described in the previous section, and then converting the corresponding R-squared back to the ${\mathrm{\Lambda }}_{\alpha }$ scale. We demonstrate an analysis using this benchmarking strategy in the empirical example in the next section. Alternatively, we can use an informal benchmarkarking strategy for ${\mathrm{\Lambda }}_{\alpha }$ by directly computing how much the odds of treatment changes when adding a reference covariate into a propensity model which already includes some baseline covariates, e.g. by computing the 1-$\alpha \text{th}$ quantile of the odds ratio $\frac{e\left(X_{base},X_{ref}\right)/\left(1-e\left(X_{base},X_{ref}\right)\right)}{e\left(X_{base}\right)/\left(1-e\left(X_{base}\right)\right)}$ (see e.g. Kallus and Zhou, 2021; Dorn and Guo, 2022). We leave exploration of additional formal benchmarking strategies in this parameterization for future work.

6. Analyzing the Effects of Light Alcohol Consumption

We apply our proposed multi-outcome sensitivity analysis in an investigation of a long-standing question about the potential health benefits of light to moderate alcohol consumption on health outcomes. In particular, observational data indicates that light alcohol consumption is positively correlated with blood levels of HDL (“good cholesterol”) and negatively correlated with LDL (“bad cholesterol”) (e.g. Choudhury et al., 1994; Meister et al., 2000; O’Keefe et al., 2007). However, there are known to be many potential confounders related to diet and lifestyle which could explain these associations. We consider treated individuals $\left(n_T=114\right)$ as those who self-reported drinking between one and three alcoholic beverages per day, and untreated individuals $\left(n_C=1439\right)$ as those that averaged one drink per week or less. We make use of laboratory outcomes, $Y=\left(Y_1,\cdots ,Y_{10}\right)$, which consist of three measures of cholesterol (HDL, LDL, and triglycerides), as well as blood levels of potassium, iron, sodium, and glucose and levels of three environmental toxicants, methylmercury, cadmium and lead, all collected from 2017–2020 (pre-pandemic) as part of the National Health and Nutrition Examination Study (NHANES). We control for observed confounders which include age, gender, and an indicator for education beyond a high school degree.

In our analysis, we consider the layers of assumptions that one might make to reason about the set of causal effects which are consistent with the observed data. First, we report posterior intervals for causal effects under the NUC assumption, and then consider bounds on the causal effects under factor confounding (Assumption 6). We calibrate the bounds by benchmarking values of ${\mathrm{\Lambda }}_{0.95}$, the 95 th percentile of the multiplicative change in the odds of treatment after accounting for unobserved confounders (Section 5.2). Alternatively, in lieu of specifying ${\mathrm{\Lambda }}_{0.95}$ directly, we also explore the implications of a carefully chosen null control outcome on non-null outcomes. We report robustness values for all outcomes both with and without the null control assumption.

We start by estimating associations under NUC by fitting a multivariate linear regression model with factor-structured residuals, where the estimands reduce to the regression coefficients, $T_j$, for $j=1,\cdots ,10$. The logarithms of all outcomes are approximately unimodal, symmetric, and not heavy-tailed, and thus we regress the log outcomes on age, gender, education, and the alcohol consumption indicator to estimate ${\tau }_j$ under an assumption of no unobserved confounding. Here, we assume $g(t)=\tau t$ in (5) and fit a Bayesian multivariate linear regression with a rank- $m$ factor model structure on the Gaussian residuals using STAN (Stan Development Team, 2022). Note that factor confounding can only be satisfied if $m\le 6$ otherwise $(q-m{)}^2-q-m<0$. As such, we fit models of rank $m\le 6$ and use Pareto-Smoothed Importance Sampling estimates of the leave-one-out cross-validation loss to evaluate relative model fit (Vehtari et al., 2017). We compare differences in expected log predictive density (ELPD) for different ranks and find that models with ranks $m=5$ and $m=6$ have an ELPD within one standard deviation of the full rank model, and thus can be viewed as statistically indistinguishable from the model in which there are no constraints on the residual covariance of the outcomes (see Appendix C, Table 2). For the remainder of the analysis, we proceed with the rank- 5 model as the smallest model which can explain the correlations in the data (see Appendix Figure C.1 for a heatmap of the inferred $\mathrm{\Gamma }$).

Next, we use observed covariates to compute benchmark values for the sensitivity value in the $\mathrm{\Lambda }$-parameterization (Section 5.2). We find that age, gender, and education are significant predictors for almost every outcome (Appendix C, Table 1) and are also all significantly correlated with the propensity for light drinking. We then compute benchmark values of ${\mathrm{\Lambda }}_{\alpha }$ using the procedure proposed in Section 5.2, by assuming $R_{T\sim U\mid X_{-j}}^2=R_{T\sim X_j\mid X_{-j}}^2$ for different choices of covariate $j$, and converting the implied value of $R_{T\sim U\mid X}^2$ back to the $\mathrm{\Lambda }$-scale. (See Sections 5.1 and 5.2). We find that for an unmeasured Gaussian confounder as strong as age, we have ${\mathrm{\Lambda }}_{0.95}=3.6$, which means that for 95% of the observed units, the predicted odds of light drinking would change by a multiplicative factor of between 1/3.6 and 3.6 when adding the unmeasured covariate into the model which already controls for age, gender and education. When benchmarking against gender we find that ${\mathrm{\Lambda }}_{0.95}=2.1$ and for education we find that ${\mathrm{\Lambda }}_{0.95}=1.6$.

In Figure 2, we plot the 95% posterior intervals for the causal effects on each of the ten outcomes under the no unobserved confounding assumption as black lines. We find that HDL cholesterol, lead, methylmercury, and potassium are positively associated with light drinking and glucose is negatively associated with light drinking. The 95% credible intervals for all other outcomes include zero. We also include the worst-case bounds for each outcome assuming ${\mathrm{\Lambda }}_{0.95}=3.6$ (black rectangles), which matches the benchmark value computed using age, the observed confounder with the strongest relationship to the treatment. While each marginal interval except methylmercury includes zero under this assumption, it is not true that all of those outcomes could simultaneously be zero at ${\mathrm{\Lambda }}_{0.95}=3.6$, since the worst-case biases for each outcome are achieved with different values of the sensitivity vector, $\rho$.

Fig. 2.

95% posterior credible intervals for the causal effects of light drinking on each outcome under NUC (black) and under factor confounding with ${\mathrm{\Lambda }}_{0.95}\le 3.6$ (black box). A multiplicative change in the odds of light drinking of ${\mathrm{\Lambda }}_{0.95}=3.6$ is consistent with an unmeasured confounder which has a dependence with the treatment which is as strong as the dependence of the treatment on age (See Section 5). In red, we plot the 95% credible intervals when methylmercury is assumed to be a null control outcome and ${\mathrm{\Lambda }}_{0.95}={\mathrm{\Lambda }}_{min}$, the value needed to nullify methylmercury under factor confounding (red intervals). Numbers in black indicate different robustness values, converted to the ${\mathrm{\Lambda }}_{0.95}$-parameterization, including the single-outcome robustness $(R{V}^1)$, extreme robustness $(XRV)$, robustness under factor confounding $\left(R{V}^{\mathrm{\Gamma }}\right)$ and the combined robustness value with methylmercury as a null control, $R{V}_{NC}^{\mathrm{\Gamma }}\ge {\mathrm{\Lambda }}_{min,𝒞}$ (in red). Listed values are conservatively reported as the lower endpoint of the 95% posterior credible intervals of the robustness values.

We also include four different robustness values, converted to the ${\mathrm{\Lambda }}_{0.95}$ - parameterization, above each interval. We include the single-outcome robustness values ($R{V}^1$, Cinelli and Hazlett, 2020), extreme robustness values ($XRV$, Cinelli and Hazlett, 2022), robustness under factor confounding $(R{V}^{\mathrm{\Gamma }})$ and the combined robustness under factor confounding with methylmercury as a null control $(R{V}_{c=2}^{\mathrm{\Gamma }})$. Since there is estimation uncertainty for these quantities, we conservatively report the lower endpoint of the 95% posterior interval for each robustness value. We find that under the factor confounding assumption, methylmercury levels are the most robust to confounding $\left(R{V}^{\mathrm{\Gamma }}=4.1\right)$ followed by HDL $\left(R{V}^{\mathrm{\Gamma }}=3.4\right)$. The $RV$ for methlymercury exceeds the benchmark computed from age whereas the $RV$ for HDL is only slightly smaller than this benchmark, meaning an unmeasured confounder of comparable strength to age would be enough to nullify the apparent effect of light drinking on HDL. In this analysis, the single-outcome robustness values, $R{V}^1$, are all much larger than $R{V}^{\mathrm{\Gamma }}$, and likely overstate the robustness of the effects under NUC. In contrast, the inferred factor model implies that for many outcomes, a relatively large fraction of outcome variance may be due to unobserved confounding. As such, for biases large enough to change the sign of the causal effects, $R_{T\sim U\mid X}^2\ll R_{a^{\prime }Y\sim U\mid T,X}^2$ for most outcomes. $XRV$ are by definition the most conservative possible robustness values and thus always less than $R{V}^{\mathrm{\Gamma }}$.

Despite the apparent robustness of these effects, we suspect that methylmercury levels are primarily tied to the consumption of fish and note that mercury is not found in alcoholic beverages. Since there is no other known credible mechanism for light drinking directly influencing mercury levels, methylmercury makes an ideal null control outcome. We then evaluate whether correcting the inferred bias in effect estimates for mercury also explains away the apparent effects for the other outcomes. We plot 95% posterior credible intervals for all effects after incorporating methylmercury as a null control, assuming no additional confounding beyond the smallest amount required to explain away methylmercury’s association. Stated differently, these plots show the posterior distribution of $a^{\prime }\check{\tau }-a^{\prime }\mathrm{\Gamma }{\mathrm{\Gamma }}_{𝒞}^{†}{\check{\tau }}_{𝒞}$ which corresponds to the midpoint of the causal effect ignorance regions for outcome $a^{\prime }Y$ given null controls $𝒞$, or equivalently, the identifiable effect estimate under the assumption that ${\mathrm{\Lambda }}_{0.95}={\mathrm{\Lambda }}_{min,𝒞}$ (Theorem 2). If the posterior interval for $a^{\prime }\check{\tau }-a^{\prime }\mathrm{\Gamma }{\mathrm{\Gamma }}_{𝒞}^{†}{\check{\tau }}_{𝒞}$ includes 0, then apparently the null control assumption is enough to explain away the significance of $a^{\prime }Y$ without introducing additional unobserved confounding. If it does not include zero, then in red we report the combined robustness, $R{V}_{c=2}^{\mathrm{\Gamma }}$, corresponding to the magnitude of confounding needed to nullify the effect of light drinking on mercury and nullify the effect on the corresponding outcome (Theorem 3, right of slash). We separately report the additional amount needed to nullify the effect beyond what is needed to nullify methylmercury (left of slash).

When methylmercury is a null control, ${\mathrm{\Lambda }}_{min,\text{c}}=4.1$, mercury’s robustness value. The 95% posterior credible intervals of $a^{\prime }\check{\tau }-a^{\prime }\mathrm{\Gamma }{\mathrm{\Gamma }}_{𝒞}^{†}{\check{\tau }}_{𝒞}$ for HDL, lead, and glucose all include zero after incorporating the null control constraint, meaning that factor confounding and the null control assumption is enough to explain away the causal effects for these outcomes, assuming no additional confounding beyond the minimum needed to explain away the effect on mercury. In contrast, the 95% credible interval for the causal effect on potassium, which included zero under NUC, moves away from zero since potassium levels are negatively correlated with mercury levels. On top of the confounders needed to nullify methylmercury, we would need additional confounders that were associated with potassium to change the odds of treatment by as much as about 2.4 (left of slash). In total, we would need unobserved confounders to change the odds of treatment by a multiplicative factor of as much as 4.1 × 2.4 ≈ 9.9 to nullify both methylmercury and potassium (the combined robustness value, right of slash). When factor confounding holds, we conclude that there is evidence that light alcohol consumption has a positive causal effect on potassium levels, but no apparent effect on HDL, lead, or glucose levels. In summary, under factor confounding, a relatively strong association between confounders and treatment is needed to explain the bias in mercury, which further implies a large bias adjustment for potentially non-null outcomes.

Crucially, the factor confounding assumption (Assumption 6) can be violated if, for example, some unmeasured environmental confounders influenced both a single outcome and the propensity for light drinking, but not any other outcomes. In such a case, we could further explore relaxations of factor confounding by manually calibrating $R_{a^{\prime }Y\sim U\mid T,X}^2$ to values that are larger than the inferred values. We discuss such strategies and provide additional theory about generalizations in Appendix B. In Appendix C we provide several additional results under relaxations of the factor confounding assumption for this example, focused in particular on the potential for single outcome confounding for methylmercury. At one extreme, when we fix $R_{a^{\prime }Y\sim U\mid T,X}^2=1$ for methylmercury, we assume the strongest possible association between confounders and mercury levels. When this holds, the majority of unobserved confounding for methylmercury is uncorrelated with other outcomes, and thus the null control assumption has a much smaller impact on the conclusions for other outcomes (see Appendix Figure C.2). In full generality, we note that each of the $q\times m$ entries of $\mathrm{\Gamma }$ can always be specified manually by a practitioner, although in practice it is likely to be very difficult to rigorously justify any such choice without starting from higher level assumptions like those proposed in this work.

7. Discussion

In this paper, we propose a sensitivity analysis for characterizing the range of potential biases that can arise in observational analyses with multivariate outcomes. Unlike previous work on observational causal inference with multivariate outcomes, which typically require stronger assumptions for causal identification, we explore the range of causal effects that are compatible with the observed data under different untestable assumptions about the strength of unobserved confounding. We show precisely how the bias varies by outcome and depends on the inferred residual covariance in the outcome model. When appropriate, we show how assumptions about factor model identifiability can be used to provide stronger results about the robustness of effects. We then characterize how null control outcomes influence both the partial identification region and robustness of effects.

There are several extensions and generalizations that are worth considering. Importantly, in this work, we focus primarily on modeling under a factor model structure, although extensions for non-continuous outcomes and more complex structures could be developed, perhaps by leveraging the copula decomposition proposed by Zheng et al. (2021b) or by making use of generalized latent variable models (Skrondal and Rabe-Hesketh, 2004). Also, in the simulation and NHANES example, we used Bayesian inference, and there is room for a more in-depth exploration of the effects of different prior distributions on partially identified parameters (Gustafson, 2015; Zheng et al., 2021a).

Finally, this work builds on closely related work on sensitivity analyses for multi-treatment causal analyses (Zheng et al., 2021b). A natural follow-up would be to consider the identification implications for data that involve both multiple simultaneously applied treatments and multiple outcomes. This would further bridge our work and recent works in proximal causal inference (Tchetgen et al., 2020). There may be particular connections to the work of Miao et al. (2018) who discuss conditions under which the average treatment effects can be nonparametrically identified with a single null control treatment and a null control outcome, via a double null controls design (see also Miao et al., 2018; Shi et al., 2020).