Multiple-bias Sensitivity Analysis Using Bounds

Abstract

Confounding, selection bias, and measurement error are well-known sources of bias in epidemiologic research. Methods for assessing these biases have their own limitations. Many quantitative sensitivity analysis approaches consider each type of bias individually, although more complex approaches are harder to implement or require numerous assumptions. By failing to consider multiple biases at once, researchers can underestimate—or overestimate—their joint impact. We show that it is possible to bound the total composite bias owing to these three sources and to use that bound to assess the sensitivity of a risk ratio to any combination of these biases. We derive bounds for the total composite bias under a variety of scenarios, providing researchers with tools to assess their total potential impact. We apply this technique to a study where unmeasured confounding and selection bias are both concerns and to another study in which possible differential exposure misclassification and confounding are concerns. The approach we describe, though conservative, is easier to implement and makes simpler assumptions than quantitative bias analysis. We provide R functions to aid implementation.

Assessing evidence for causation is fundamental to plan and target interventions and improve public health. However, many causal claims in epidemiologic studies are met with suspicion by both researchers and the general public, owing to the fact that such studies are well known to be subject to various biases. Although faults in these studies can sometimes be addressed directly—for example, through better sampling schemes, blinded outcome ascertainment, more extensive covariate measurements, and so on—other times confounding, selection bias, and measurement error are unavoidable. In such situations, our next best option is to assess the extent to which a given study’s conclusions might be sensitive or robust to these biases and whether they threaten its conclusions. Often, however, this is limited to a few sentences in a discussion section qualitatively assessing the possibility of bias, sometimes appealing without quantitative justification to heuristics that may or may not hold true in a particular study.^1–3

The weak uptake of quantitative bias analysis in epidemiology belies its long history. Over a half century ago, Cornfield and then Bross argued that the extent of possible bias was quantifiable based on observed data and possibly hypothetical quantities.^4–7 Attempts to generalize these results, as well as consider other biases, sometimes simultaneously with confounding, followed.^7–12 More recently, probabilistic bias analysis methods have been developed, allowing researchers to propose distributions for various bias parameters across multiple biases and to explore how various combinations of those parameters would affect their results.^13–18 Despite the availability of these methods in textbook and software form,^16,19,20 the actual uptake of such quantitative bias analysis in empirical research has been limited,²¹ possibly because of the (at least perceived) computational complexity²² or difficulty in proposing plausible distributions.

Even more recently, simpler approaches to sensitivity analysis have hearkened back to the early days of bias assessment, with the development of bounds for various biases that require limited assumptions and at most basic algebra.^23–25 However, a one-at-a-time approach is not sufficient for many studies subject to multiple sources of bias. In this article, we extend the simple sensitivity analysis framework to multiple biases, describing a bound for the total composite bias from confounding, selection, and differential exposure or outcome misclassification.

THE PROBLEM OF MULTIPLE BIASES

We will describe a scenario in which all three types of bias are present, preventing the interpretation of an observed risk ratio as a causal risk ratio. Consider a binary exposure A, a binary outcome Y, and measured covariates C (C may also include unmeasured factors that are controlled by the study design). Let S be an indicator of the subset of the population for which data have been collected, and let A* and Y * denote misclassified versions of the exposure and outcome, respectively. We use potential outcome notation to describe causal quantities: Y_a is the outcome that would occur were exposure A set to value a. We assume consistency, meaning that Y_a = Y for observations for whom we observe A = a, and positivity, meaning that 0 < Pr(A =1|·) <1 within every stratum of the population.

We denote (conditional) independence between random variables with the symbol ∐, such that Y_a∐A|C implies conditional exchangeability; that is, potential outcomes are independent of exposure status conditional on C. However, when C does not capture all of the exposure–outcome confounding, it is not true that Y_a∐A|C. We assume in that case that additionally adjusting for some unmeasured factor(s) U_c would be sufficient to address confounding so that Y_a∐A|C, U_c. Here, U_c may be a single random variable or a vector of variables, which may be continuous or take on any number of discrete values, or some combination. Similarly, we allow for selection bias, which we define as a lack of the conditional independence Y_a∐A|C, U_c, S = 1 when it is otherwise true that Y_a∐A|C, U_c,. We likewise assume that the measurement of some variable(s) U_s, responsible for selection, would fully account for this bias, although the necessary conditions for it to do so will depend on whether we intend to make inferences about effects in the total population, or just the selected population. Finally, we allow for the possibility that the misclassification is differential, by which we mean that the sensitivity or specificity of the exposure measurement may differ depending on the value of the outcome, or that the sensitivity or specificity of the outcome measurement may depend on the exposure. In our notation, this means that it is not necessarily true that A*∐Y | A,C or that Y *∐A|Y,C. In this work, we consider only misclassification of the exposure or the outcome but not both at once.

MOTIVATING EXAMPLES

There is great interest in how exposures during pregnancy may affect offspring health. However important such questions are, they are difficult to answer with epidemiologic research. Ethics may limit inclusion of pregnant people in randomized trials, and many exposures of interest are not ethical or feasible to randomize to anyone. Case-control studies can efficiently capture rare childhood outcomes but recalling pregnancy exposures several years later can result in measurement error.²⁶ Prospective cohort studies can avoid this recall bias but are often subject to loss to follow-up when the duration between exposure and outcome assessment is long.²⁷ Observational studies of all types are threatened by uncontrolled confounding, and intergenerational confounders are particularly difficult to assess.²⁸ Importantly, studies like these are not affected by only one or another of these biases but may suffer from multiple threats to validity.

To demonstrate our sensitivity analysis approach, we will consider two questions about exposures during pregnancy and outcomes in children: whether HIV infection in utero causes wasting (low weight-for-length) and whether vitamin consumption during pregnancy protects against childhood leukemia.

Omoni et al²⁹ investigated the former hypothesis concerning HIV infection and wasting (among participants of a vitamin A supplementation trial in Zimbabwe) and found that, compared with children who were unexposed to HIV, those who had been infected with HIV in utero were more likely to be below a weight-for-length Z score of −2 as toddlers. The odds ratio comparing the two groups was 6.75 (95% CI, 2.79, 16.31) at 2 years. Although randomized trial data were used for the analysis, this was an observational study with respect to HIV infection, since infection is not randomized. The authors did not, however, adjust for any confounders. Furthermore, since enrollment occurred at delivery, after possible HIV exposure and transmission, the choice of whether to participate could have been affected by HIV status as well as other factors, leading to selection bias if those factors affect future child growth. We will consider the role that confounding and selection bias may play in this study.

As a second example, Ross et al³⁰ analyzed the relationship between vitamins and leukemia in a case-control study and found a decreased risk of acute lymphoblastic leukemia among children whose mothers consumed vitamin supplements during pregnancy. Their reported odds ratio, which, with a rare outcome, approximates a risk ratio of 0.51 (95% CI 0.30, 0.89), was conditional on maternal age, race, and a binary indicator of education. However, there may be other confounders that were not controlled, such as other indicators of a privileged or healthy lifestyle that are both associated with vitamin use and protection against leukemia. We also may be concerned about recall bias (differential exposure misclassification)—that mothers of children with a cancer diagnosis might be more likely to report not taking a vitamin even if they did so—so we consider how exposure misclassification and unmeasured confounding can be assessed simultaneously.

THE MULTIPLE-BIAS BOUND

Two overarching types of bias analysis have been described: one that explores how biases of a given magnitude affect an estimate, which Phillips labeled “bias-level sensitivity analysis” and another that reduces the analysis to a summary of how much bias would be necessary for an observation to be compatible with a truly null effect (or some other specified nonnull effect), which he called “target-adjusted sensitivity analysis.”² We focus on the former and address the latter in the eAppendix; http://links.lww.com/EDE/B824. Here, we present a multiple-bias bound, which allows researchers or consumers of research to explore the maximum factor by which unmeasured confounding, selection, and misclassification could bias a risk ratio.

We begin with outcome misclassification and then extend our results to exposure misclassification. We assume that the investigators have estimated ${\text{RR}}_{A{Y}^{*}}^{\text{obs}}=\frac{\mathrm{Pr}\left(Y^{*}=1\mid A=1,S=1,c\right)}{\mathrm{Pr}\left(Y^{*}=1\mid A=0,S=1,c\right)}$, the observed risk ratio conditional on some value c of the covariates but wish to make inference about the causal conditional risk ratio ${\text{RR}}_{AY}^{\text{true}}=\frac{\mathrm{Pr}\left(Y_1=1\mid c\right)}{\mathrm{Pr}\left(Y_0=1\mid c\right)}$. We will assess bias on the relative scale so that we define the bias as ${\text{RR}}_{A{Y}^{*}}^{\text{obs}}/{\text{RR}}_{AY}^{\text{true}}$.

Using bounds that have been previously described for misclassification, selection bias, and unmeasured confounding considered individually,^{23–25, 31} we can bound ${\text{RR}}_{A{Y}^{*}}^{\text{obs}}$ by factoring it into ${\text{RR}}_{AY}^{\text{true}}$ and components for each of the biases. The parameters that will be used to bound the biases are as follows:

$${\text{RR}}_{A{Y}^{*}\mid y,S=1}=\underset{y}{\max }\frac{\mathrm{Pr}\left(Y^{*}=1\mid Y=y,A=1,S=1,c\right)}{\mathrm{Pr}\left(Y^{*}=1\mid Y=y,A=0,S=1,c\right)}$$

$${\mathrm{RR}}_{U_{s}Y\mid A=a}=\frac{\underset{u}{\max }\mathrm{Pr}\left(Y=1\mid A=a,c,U_s=u\right)}{\underset{u}{\mathrm{mix}}\mathrm{Pr}\left(Y=1\mid A=a,c,U_s=u\right)}\text{for}a=0,1$$

$${\text{RR}}_{S{U}_s\mid A=a}=\underset{u}{\max }\frac{\mathrm{Pr}\left(U_s=u\mid A=a,S=a,c\right)}{\mathrm{Pr}\left(U_s=u\mid A=a,S=1-a,c\right)}\text{for}a=0,1$$

$${\mathrm{RR}}_{U_{c}Y}=\underset{a}{\max }\frac{\underset{u}{\max }\mathrm{Pr}\left(Y=1\mid A=a,c,U_c=u\right)}{\underset{u}{\mathrm{mix}}\mathrm{Pr}\left(Y=1\mid A=a,c,U_c=u\right)}$$

$${\text{RR}}_{A{U}_c}=\underset{u}{\max }\frac{\mathrm{Pr}\left(U_c=u\mid A=1,c\right)}{\mathrm{Pr}\left(U_c=u\mid A=0,c\right)}.$$

These bias parameters have been described elsewhere, although separately.^23–25 Briefly, the bias parameter defining the misclassification portion of the bound $\left({\text{RR}}_{A{Y}^{*}\mid y,S=1}\right)$ describes the maximum of the false-positive probability ratio or sensitivity ratio within the selected population. The selection bias parameters (${\mathrm{RR}}_{U_{s}Y\mid A=a}$ and ${\text{RR}}_{S{U}_s\mid A=a}$) describe the maximum factors by which the outcome risk differs by values of U_s, within strata of A, and the maximum factors by which some level of U_s differs between the selected and nonselected groups, within strata of A. Finally, the unmeasured confounding parameters (${\text{RR}}_{U_{c}Y}$ and ${\text{RR}}_{A{U}_c}$) describe the maximum factor by which U_c increases the outcome risk, conditional on A, and the maximum factor by which exposure is associated with some value of U_c. Each of the sensitivity parameters is conditional on the covariates adjusted for in the analysis and so describes the extent of bias above and beyond those factors.

To simplify notation, define the function $g(a,b)=\frac{a\times b}{a+b-1}$. Then, we have the following bound for the total composite bias.

Result 1:

If Y_a∐A|C, U_c and Y∐S|A, C, U_s then: ${\text{RR}}_{A{Y}^{*}}^{\text{obs}}/{\text{RR}}_{AY}^{\text{true}}\le {\text{BF}}_m\times {\text{BF}}_s\times {\text{BF}}_c$ where ${\text{BF}}_m={\text{RR}}_{A{Y}^{*}\mid y,S=1}$, ${\text{BF}}_s=g\left({\text{RR}}_{U_{s}Y\mid A=1},{\text{RR}}_{S{U}_s\mid A=1}\right)\times g\left({\text{RR}}_{U_{s}Y\mid A=0},{\text{RR}}_{S{U}_s\mid A=0}\right)$, and ${\text{BF}}_c=g\left({\text{RR}}_{A{U}_c},{\text{RR}}_{U_{c}Y}\right)$. The derivation of this and the results that follow are given in the eAppendix; http://links.lww.com/EDE/B824.

Result 1 can be used to quantify the maximum amount of bias that could be produced by parameters of a given value. Values for the sensitivity parameters may be taken from validation studies, previous literature, or expert knowledge or proposed as hypotheticals. Because the sensitivity parameters are maxima, they are always greater than or equal to 1, and the composite bound will thus be greater than or equal to 1. For an apparently causative observed exposure–outcome risk ratio (>1), one could divide the estimate and its confidence interval by the bound to obtain the maximum that the specified biases could shift the estimate and its confidence interval. For a preventive observed exposure–outcome risk ratio (<1), one could multiply the estimate and its confidence interval by the bound to obtain the maximum that the specified biases could shift the estimate and its confidence interval, or equivalently reverse the coding of the exposure to obtain a risk ratio >1. By applying the bound to the confidence interval closest to the null, we can make statements such as: “In 95% of repeated samples with the same sources of bias, adjusting the confidence interval in this way would result in a lower bound that is less than the true causal risk ratio, provided the proposed parameter values adequately bound (i.e., are as large as or larger than) the true parameter values.”

Although the bound allows for terms for all three biases, if any of them is judged not to threaten a given study or to bias toward the null, that factor can be omitted. Furthermore, the selection bias term can be simplified under certain assumptions;²⁴ we illustrate in the first example below.

Result 1 is summarized in the first row of Table 1. The assumptions required for the bound to hold are listed under the biases to which they pertain, and the bound itself is in the final column. Note that the factorization of the bound implies an ordering of the biases: the misclassification parameters are defined within the stratum S = 1. Intuitively, this corresponds to a study in which outcome measurement is done after people have been selected into the study and so requires considering the strength of differential misclassification only within that group. In general, we can think of biases as layers that we must peel off sequentially and the order in which we do so is the reverse of the order in which they occurred in the data.^32,33 Confounding is generally thought of as a property of nature within the population of interest, so occurs first (although if parameters describing the strength of confounding are derived based on misclassified exposure or outcome, that may not be the case³²), but the order in which selection and misclassification occur may depend on the study design. We could alternatively derive a bound that depends on a parameter describing the extent to which the outcome is misclassified in the total population, and on others describing how selection is associated with the misclassified outcome. These parameters may be more intuitive in a study with case-control sampling. Additionally, another ordering of the biases may be preferable if data exist to justify estimates of the alternative parameters. We define those alternative parameters and derive that bound in the eAppendix; http://links.lww.com/EDE/B824. The second row of Table 1 summarizes those results; there, the assumption for selection bias is an assumption about the misclassified outcome, and the bound in the final column is defined in terms of parameters that reflect that ordering.

TABLE.

Multiple Bias Bounds for Various Combinations of Biases

Biases and Associated Assumptions			Bound Under the Stated Assumptions
Bias 1	Bias 2	Bias 3
Unmeasured confounding Ya ⫫ A | C, Uc	General selection bias Y ⫫ S | A, C, Us	Outcome misclassification	$g\left({\mathrm{RR}}_{A{U}_c},{\text{RR}}_{U_{c}Y}\right)\times g\left({\mathrm{RR}}_{U_{s}Y\mid A=1},{\text{RR}}_{S{U}_s\mid A=1}\right)\times g\left({\mathrm{RR}}_{U_{s}Y\mid A=0}{\text{RR}}_{S{U}_s\mid A=0}\right)\times {\text{RR}}_{A{Y}^{*}\mid y,S=1}$
Unmeasured confounding Ya ⫫ A | C, Uc	Outcome misclassification	General selection bias Y* ⫫ S|A, C, Us	$g\left({\text{RR}}_{A{U}_c},{\text{RR}}_{U_{c}Y}\right)\times {\text{RR}}_{A{Y}^{*}\mid y}\times g\left({\mathrm{RR}}_{U_s{Y}^{*}\mid A=1},{\text{RR}}_{S{U}_s\mid A=1}\right)\times g\left({\text{RR}}_{U_s{Y}^{*}\mid A=0}{\text{RR}}_{S{U}_s\mid A=0}\right)$
Unmeasured confounding Ya ⫫ A | S = 1, C, Uc, Us	Selected population	Outcome misclassification	$g\left({\mathrm{RR}}_{A{U}_{sc}\mid S=1},{\mathrm{RR}}_{U_{sc}}Y\mid S=1\right)\times {\text{RR}}_{A{Y}^{*}\mid y}$
Unmeasured confounding Ya ⫫ A | C, Uc	General selection bias Y ⫫ S | A, C, Us	Exposure misclassification Pr(Y = 0 | a, c, S = 1) ≈ 1	$g\left({\text{RR}}_{A{U}_c},{\text{RR}}_{U_{c}Y}\right)\times g\left({\text{RR}}_{U_{s}Y\mid A=1},{\text{RR}}_{S{U}_s\mid A=1}\right)\times g\left({\text{RR}}_{U_{s}Y\mid A=0}{\text{RR}}_{S{U}_s\mid A=0}\right)\times {\text{OR}}_{Y{A}^{*}\mid a,S=1}$
Unmeasured confounding Ya ⫫ A | C, Uc	Exposure misclassification Pr(Y = 0 | a, c) ≈ 1	General selection bias Y ⫫ S | A*, C, Us	$g\left({\mathrm{RR}}_{A{U}_c},{\text{RR}}_{U_{c}Y}\right)\times {\text{OR}}_{Y{A}^{*}\mid a}\times g\left({\mathrm{RR}}_{U_{s}Y\mid A^{*}=1},{\text{RR}}_{S{U}_s\mid A^{*}=1}\right)\times g\left({\mathrm{RR}}_{U_{s}Y\mid A^{*}=0}{\text{RR}}_{S{U}_s\mid A^{*}=0}\right)$
Unmeasured confounding Ya ⫫ A | S = 1, C, Uc, Us	Selected population	Exposure misclassification Pr(Y = 0 | a, c, S = 1) ≈ 1	$g\left({\mathrm{RR}}_{A{U}_{sc}\mid S=1},{\mathrm{RR}}_{U_{sc}Y\mid S=1}\right)\times {\text{OR}}_{Y{A}^{*}\mid a}$
Unmeasured confounding Ya ⫫ A | C, Uc	General selection bias Y ⫫ S | A, C, Us	Exposure misclassification Pr(Y = 0 | a, c, S = 1) ≈ 1 Pr(A = 0 | y, c, S = 1) ≈ 1	$g\left({\text{RR}}_{A{U}_c},{\text{RR}}_{U_{c}Y}\right)\times g\left({\text{RR}}_{U_{s}Y\mid A=1},{\text{RR}}_{S{U}_s\mid A=1}\right)\times g\left({\text{RR}}_{U_{s}Y\mid A=0}{\text{RR}}_{S{U}_s\mid A=0}\right)\times {\text{RR}}_{Y{A}^{*}\mid a,S=1}$
Unmeasured confounding Ya ⫫ A | C, Uc	Exposure misclassification Pr(Y = 0 | a, c) ≈ 1 Pr(A = 0 | y, c) ≈ 1	General selection bias Y ⫫ S | A*, C, Us	$g\left({\mathrm{RR}}_{A{U}_c},{\mathrm{RR}}_{U_{c}Y}\right)\times {\text{RR}}_{Y{A}^{*}\mid a}\times g\left({\mathrm{RR}}_{U_{s}Y\mid A^{*}=1},{\text{RR}}_{S{U}_s\mid A^{*}=1}\right)\times g\left({\mathrm{RR}}_{U_{s}Y\mid A^{*}=0}{\text{RR}}_{S{U}_s\mid A^{*}=0}\right)$
Unmeasured confounding Ya ⫫ A | S = 1, C, Uc, Us	Selected population	Exposure misclassification Pr(Y = 0 | a, c, S = 1) ≈ 1	$g\left({\mathrm{RR}}_{A{U}_{sc}\mid S=1},{\text{RR}}_{U_{sc}Y\mid S=1}\right)\times {\mathrm{RR}}_{Y{A}^{*}\mid a}$
		Pr(A = 0 | y, c, S = 1) ≈ 1

The first three columns show different combinations and ordering of the biases, as well as the implied assumptions. The fourth column contains the expression that bounds the bias when those assumptions hold. The definitions of the parameters are given in the main text.

Example

We illustrate the use of the multiple-bias bound to assess possible bias in the study by Omoni and colleagues regarding the effect of HIV status on wasting.²⁹ Wasting is defined by weight-for-length Z score of −2 or below and is a rare outcome, so we can interpret the reported OR of 6.75 (95% CI, 2.79, 16.31) as an approximate risk ratio. Since we have no reason to believe that misclassification of wasting was differential by exposure status (i.e., child or mother HIV status) status, and nondifferential outcome misclassification would on average bias toward the null in this situation,⁹ we will focus on unmeasured confounding and selection bias in this example.

The choice of whether to participate in the trial, and therefore in the analysis in question, may have been influenced by prior maternal HIV status. For example, people with HIV infection may be hesitant to enroll owing to stigma regarding infection or fear of confirming their status. Other factors may affect enrollment as well: parents with food insecurity may be more likely to enroll in a vitamin-supplementation trial than those without if they think it will improve their children’s nutrition. This benefit could outweigh the hesitancy for some, resulting in selection bias: if a mother in the study is living with HIV, it is likely that her family is also food insecure, making her child more at risk of wasting. Participation in the trial is therefore a collider in a directed acyclic graph describing these relationships, as shown in Figure 1. Similarly, there are factors that are associated with HIV status that may also affect wasting; if these are not on the causal pathway, we may be worried about unmeasured confounding. The authors did not adjust for parity or marital status, although they report that primiparous women were less likely to have HIV, as were married women. We may be concerned that children in single-parent households and those with more siblings are at higher risk of wasting. To demonstrate interpretation of the bound, we will propose values for the parameters describing the strength of these relationships based on the data presented in the original article as well as our background knowledge.

FIGURE 1.

DAGs depicting the examples described in the text. A, This DAG depicts unmeasured confounding (due to U_c) and selection bias (due to U_s). In the graph, the assumptions Y_a ∐A |U_s, S = 1 and Y_a∐A| U_c are met. This corresponds to the first example in the text, where A indicates HIV infection, U_c family factors including parity and marital status, S participation in the trial, U_s food insecurity, and Y wasting. B, This DAG depicts unmeasured confounding (due to U_c) and differential misclassification of the exposure (due to the Y → A* edge). In the graph, the assumption Y_a∐A |C,U_c is met. This corresponds to the second example in the text, where A indicates vitamin consumption during pregnancy, A* reported vitamin consumption, U_c breastfeeding, C maternal age, race, and education, and Y child leukemia. DAGs indicates directed acyclic graphs.

Suppose that the most vulnerable in the population were more likely to participate in the trial, and thus that wasting is more likely in children of participants than of nonparticipants, both among those with HIV as well as those without. The assumption that the outcome is more likely in the selected population of both exposure groups allows us to simplify the selection bias component of the bound, so that the bounding factor only relies on two selection terms, as described by Smith and VanderWeele.²⁴ Suppose now that children of the most food-insecure mothers are 3 times as likely to have extremely low weight-for-length scores than the least likely group so that ${\text{RR}}_{U_{s}Y\mid A=1}=3$ and that the mothers with HIV infection in the study compared with those not in the study are twice as likely to be food insecure so that ${\text{RR}}_{S{U}_s\mid A=1}=2$.

Although the odds ratios from this study were not adjusted for parity and marital status, the authors reported proportions of these characteristics stratified by exposure,²⁹ which can aid in coming up with a reasonable value for ${\text{RR}}_{A{U}_c}$. For example, suppose we estimate that 3% of the women whose infants are infected with HIV are multiparous and unmarried, but that this is true of 7% of the women without HIV. If this is the family situation with the largest disparity between exposure groups, then we can specify ${\text{RR}}_{A{U}_c}=2.3$. Now suppose that children in these most precarious families have 2.5 times the risk of wasting than those in the least precarious, so that ${\text{RR}}_{U_{c}Y}=2.5$.

Then, we can calculate the bound as $\frac{3\times 2}{3+2-1}\times \frac{2.3\times 2.5}{2.3+2.5-1}=2.27$. If those are the only sources of selection bias and unmeasured confounding, and there is no measurement error, then this amount of bias cannot fully explain the approximate observed ${\text{RR}}_{AY}^{\text{obs}}$ of 6.75, since 6.75/2.27 = 2.97. Of course, this observed value is subject to statistical uncertainty, so we can also consider the lower limit of the confidence interval, 2.79. If the proposed parameter values hold, then even in the worst-case scenario, ${\text{RR}}_{AY}^{\text{true}}$ is still consistent with 2.79 / 2.27 = 1.23, an increase of about 23% in risk of wasting at 2 years of age owing to HIV infection. However, the parameter values, we used represent only a single reasonable choice. We might proceed by exploring a range of values, as we demonstrate below when we introduce software.

EXPOSURE MISCLASSIFICATION

When differential exposure misclassification is a concern, we can derive a similar bound under similar assumptions. However, unlike the bound for outcome misclassification, the bound for exposure misclassification that is used applies to the odds ratio, not the risk ratio, and the sensitivity parameters are also not themselves risk ratios.²⁵ We therefore cannot factor the observed risk ratio as in the previous section. However, for a sufficiently rare outcome, odds ratios approximate risk ratios, which allows for some progress.

In this section, ${\mathrm{RR}}_{A^{*}Y}^{\text{obs}}=\frac{\mathrm{Pr}\left(Y=1\mid A^{*}=1,S=1,c\right)}{\mathrm{Pr}\left(Y=1\mid A^{*}=0,S=1,c\right)}$ refers to the observed (approximate) risk ratio under exposure misclassification, when the outcome is rare in the selected population. Denote with ${\text{OR}}_{A^{*}Y\mid y,S=1}$ the largest out of the false-positive odds ratio $\left\{f_1^{\prime }/f_0^{\prime }\right\}/\left\{\left(1-f_1^{\prime }\right)/\left(1-f_0^{\prime }\right)\right\}$, the sensitivity odds ratio $\left\{s_1^{\prime }/s_0^{\prime }\right\}/\left\{\left(1-s_1^{\prime }\right)/\left(1-s_0^{\prime }\right)\right\}$, the correct classification ratio $\left\{s_1^{\prime }/s_0^{\prime }\right\}/\left\{\left(1-f_1^{\prime }\right)/\left(1-f_0^{\prime }\right)\right\}$, and incorrect classification ratio $\left\{f_1^{\prime }/f_0^{\prime }\right\}/\left\{\left(1-s_1^{\prime }\right)/\left(1-s_0^{\prime }\right)\right\}$, where $f_y^{\prime }=\mathrm{Pr}\left(A^{*}=1\mid Y=y,A=0,S=1,c\right)$ and $s_y^{\prime }=\mathrm{Pr}\left(A^{*}=1\mid Y=y,A=1,S=1,c\right)$. Then the following bound holds approximately; that is, to the extent that the odds ratio approximates the risk ratio.

Result 2:

If Pr(Y = 0|A* = a, S = 1, c) ≈ 1 and Pr(Y = 0|A = a, S = 1, c) ≈ 1 then:

$${\text{RR}}_{A^{*}Y}^{\text{obs}}/{\text{RR}}_{AY}^{\text{true}}\le {\text{BF}}_m\times {\text{BF}}_s\times {\text{BF}}_c$$

where ${\text{BF}}_m={\text{OR}}_{A^{*}Y\mid y,S=1}$ and BF_s and BF_c are as previously defined.

This result is summarized in the fourth row of Table 1, as are extensions involving exposure misclassification.

Example

We can jointly assess the magnitude of bias owing to differential recall of vitamin use and unmeasured confounding in the study of leukemia risk by Ross and colleagues, in which ${\mathrm{RR}}_{A^{*}Y}^{\text{obs}}=0.51$ (95% CI 0.30, 0.89), by proposing realistic values for the bias parameters. A probabilistic bias analysis for misclassification was previously done in relation to this study, in which Jurek et al conducted a literature search for validation studies of multivitamin use during the periconceptional period.³⁴ They found no pertinent articles and instead used expert knowledge and bounds from the data (e.g., by assuming correct classification is better than chance) to propose distributions for false negative and false-positive probabilities for the cases and controls, which we can use to inform our choice of parameters. Because we think the case-control differential in false negatives is stronger than that for false positives, we might choose that Pr(A* = 0 |Y = 1, A = 1 = 0.15) and Pr(A* = 0 |Y = 0, A = 1 = 0.1) to compute ${\text{BF}}_m^{\prime }$. Since we are dealing with a possibly protective factor, however, and the bound is greater than 1 by definition, we reverse the coding of the exposure to reflect that the original estimate of ${\mathrm{RR}}_{A^{*}Y}^{\text{obs}}=0.51$ represents a 1/ 0.51=1.96-fold increase in risk associated with not taking vitamins. Therefore, $f_1^{\prime }=0.15$ and $f_0^{\prime }=0.10$, and ${\text{BF}}_m^{\prime }=1.59$.

Jurek et al’s probabilistic bias analysis used the crude 2-by-2 table from the original article, so did not take into account even the few measured confounders.³⁴ However, even those measured confounders would likely not be sufficient to control for confounding by healthy lifestyle, as there is evidence that other healthy behaviors are associated with leukemia. For example, a recent meta-analysis found that not breastfeeding compared with breastfeeding for at least 6 months was associated with an increase in acute lymphoblastic leukemia risk by a factor of 1.22.³⁵ Using breastfeeding as a proxy for healthy lifestyle, for the unmeasured confounding parameters, we will take ${\text{RR}}_{U_{c}Y}=1.22$ and ${\text{RR}}_{A{U}_c}=2$, suggesting that children who were not breastfed are 1.22 times as likely to get leukemia, and that mothers who take multivitamins are twice as likely to breastfeed than those who do not. A directed acyclic graph depicting this example is shown in Figure 1.

Using these values, we find that $1.59\times \frac{1.22\times 2}{1.22+2-1}=1.75$ indicating that the observed risk ratio may be biased by a factor of 1.75 if the differential misclassification and unmeasured confounding were of the strengths we proposed. Since we are dealing with a possible protective factor, we multiply the observed estimate of 0.51 and its confidence interval (95% CI 0.30, 0.89) by the bound (or equivalently divide the reverse-coded estimate of 1.96 by the bound), resulting in a bias-adjusted estimate and confidence interval of 0.89 (95% CI 0.52, 1.56). Unlike the Jurek et al sensitivity analysis,³⁴ which found that results were largely unchanged by exposure misclassification, we have focused specifically on a situation in which misclassification is differential by outcome, and have additionally taken both measured and unmeasured confounding into account. Doing so indicates that the results may be sensitive to misclassification and uncontrolled confounding, as can be seen if the chosen parameter values are thought to be reasonable.

INFERENCE IN THE SELECTED POPULATION

Results 1 and 2 are derived with respect to the true causal effect in the total population, despite possible selection bias. In other situations, we may only be interested in the existence and magnitude of a causal effect in the selected population. In this case, our estimand of interest is ${\mathrm{RR}}_{AY\mid S=1}^{\text{true}}=\frac{\mathrm{Pr}\left(Y_1\mid S=1,c\right)}{\mathrm{Pr}\left(Y_0\mid S=1,c\right)}$ If only selection bias is present, one can derive a bound under the assumption that Y_a∐A|S = 1,c,U_s.²⁴ In the present context, we additionally accommodate unmeasured confounding and measurement error. Consider unmeasured confounding by U_c such that it is only the case that Y_a∐A|S = 1,c,U_s,U_c. Therefore, we must consider the vector of factors causing selection bias and unmeasured confounding U_sc = (U_s, U_c). Define the sensitivity parameters ${\text{RR}}_{U_{sc}Y}={\max }_a\frac{{\max }_u\mathrm{Pr}\left(Y=1\mid A=a,c,U_{sc}=u\right)}{{\min }_u\mathrm{Pr}\left(Y=1\mid A=a,c,U_{sc}=u\right)}$ and ${\text{RR}}_{A{U}_{sc}}={\max }_u\frac{\mathrm{Pr}\left(U_{sc}=u\mid A=1,c\right)}{\mathrm{Pr}\left(U_{sc}=u\mid A=0,c\right)}$. Then under outcome misclassification, we have the following bound.

Result 3:

If Y_a⨆A|S = 1, c, U_c, U_s then:

$${\text{RR}}_{A{Y}^{*}}^{\text{obs}}/{\text{RR}}_{AY\mid S=1}^{\text{true}}\le {\text{BF}}_m\times {\text{BF}}_{sc}$$

where BF_m is defined as in Result 1, and ${\text{BF}}_{sc}=g\left({\text{RR}}_{U_{sc}Y},{\text{RR}}_{A{U}_{sc}}\right)$. These latter parameters now refer to the maximum risk ratio for the outcome among the selected comparing any two levels of any of U_s and U_c, and the maximum ratio for any joint level of U_s and U_c comparing exposed to unexposed, among the selected. This bound holds under exposure misclassification with a rare outcome in the selected population as well, with ${\text{BF}}_m^{\prime }={\text{OR}}_{A^{*}Y\mid y,S=1}$

This result is summarized in the third row of Table 1.

SOFTWARE

The R package EValue³⁶ allows for easy calculation of the multiple-bias bounds for various combinations of biases and assumptions, including all those presented in Table 1, as well as the possible simplifications to the selection bias bound, as in the first example. The function multi_bias() creates a set of biases according to the user’s specifications. The user can then input this object along with a proposed set of parameter values to the multi_bound() function to calculate a bound.

For example, the biases in the HIV example can be set with HIV_biases <- multi_bias(confounding(), selection(“general”, “increased risk”)). The command to calculate the bound is then multi_bound(biases = HIV_biases, RRAUc = 2.3, RRUcY = 2.5, RRUsYA1 = 3, RRSUsA1 = 2). Similarly, for the vitamins-leukemia example, the biases are set with leuk_ biases <- multi_bias(confounding(), misclassification(“ex posure”, rare_outcome = TRUE, rare_exposure = FALSE)) and the bound command is multi_bound(biases = leuk_biases, RRAUc = 2, RRUcY = 1.22, ORYAa = 1.59).

These functions can be used to prepare a table or figure of bounded bias-adjusted estimates across a range of proposed parameter values. For example, Figure 2 shows the upper bound for the estimate of the protective effect of multivitamin use on leukemia across various values for ${\text{RR}}_{U_{c}Y}$ and ${\text{RR}}_{A{U}_c}$ (in the columns and rows), and for two values of the misclassification ratio ${\text{OR}}_{A^{*}Y\mid y}$ above and below the diagonal. We can use this table to describe multiple scenarios under which it would be possible for the true effect to be null. We can also see that even if, for example, the prevalence of the unmeasured confounder, or set of confounders, differs greatly between consumers and nonconsumers of multivitamins (e.g., ${\text{RR}}_{A{U}_c}=3$), a relatively small association between the unmeasured confounder and leukemia (e.g., ${\text{RR}}_{U_{c^Y}}=1.25$) and between vitamin use and misclassifcation (e.g., ${\text{OR}}_{A^{*}Y\mid y}=1.25$) would at most lead to a bias-adjusted estimate of 0.74.

FIGURE 2.

Corrected estimates for the effect of multivitamin use in pregnancy on childhood leukemia, taking into account unmeasured confounding and recall bias. The original estimate was 0.51. Corrected estimates are arranged in rows and columns by the parameters defining the unmeasured confounding, RR_AUc and RR_UcY. The two parameters are interchangeable with respect to the bound, so a table of estimates corrected only for unmeasured confounding would be symmetric. However, the estimates in the upper and lower triangles have been corrected by misclassification parameters of different magnitudes. Below the diagonal, the misclassification ratio is assumed to be 1.25; above the diagonal, it is assumed to be 1.5.

More examples are available in the eAppendix; http://links.lww.com/EDE/B824, and the package documentation is available online.

DISCUSSION

We have described an approach to sensitivity analysis that we hope can help bridge the gap between complex methods that require specifying many parameters and making restrictive assumptions, and simpler methods that allow for assessment of only one type of bias at a time. The multiple-bias bound can be used to simultaneously consider the possible effects of biases that are of different strengths. Researchers can propose values for the parameters based on background knowledge, validation studies, or simply hypothetical situations, and assess the minimum possible true risk ratio that would be compatible if the observed value were affected by biases of that magnitude. When planning for future research, the bound can be used to compare the effects of biases within a given situation and prioritize more extensive confounder assessment, a more valid sampling or inclusion scheme, and better measurement techniques if resource constraints or data collection options forced one to choose among them. It may also show that certain improvements to study design are futile; if the amount of an unavoidable bias greatly attenuates the anticipated risk ratio estimate, investing resources into reducing another type of bias may not be worth it.^37,38

There are a number of caveats and limitations to this approach. Although the calculations involved in our approach are simple, the entire process of assessing bias should not be. Importantly, it should be specific to the study design, the available data, and the research question; values for the sensitivity parameters are meaningless without a frame of reference. Indeed, critiques of the E value for unmeasured confounding have emphasized the importance of clearly specifying the confounder, or set of confounders, that have not been measured.^39–41 The same should be true for factors potentially causing selection bias or the reason behind possible differential misclassification. Unmeasured confounders could be anything from a single missing risk factor to the “ultimate covariate,”⁴² the variable encoding an individual’s causal type. Misclassification may be negligible or close to nondifferential, or as bad as chance in one or another group; it is up to researchers and readers to assess the plausibility of these situations with respect to a given study and what was conditioned on in the analysis and then assess how much bias they would create. Like any tool, the multiple-bias bound can be misused; we encourage researchers to not be careless, owing to its apparent simplicity, but rather to be thoughtful in its use.

Additionally, in avoiding certain assumptions, we have necessarily invoked others. In particular, the bounds we propose describe a “worst-case scenario” for the bias; in almost all settings, the actual bias will be smaller than the bound. For example, for the actual bias to obtain the bound assumes that the unmeasured confounder has the distribution that maximizes confounding, given the two parameters defining it.⁴³ The same is true of selection bias and misclassification; for example, the general selection bias bound implies that outcomes and exposures in the nonselected group are distributed to result in the most possible bias. This is of course necessary for a bound to be a bound, but many realistic conditions would not result in as much bias, and the bound should be interpreted as the bias that could result from parameters of a given magnitude not that necessarily would result. The few assumptions that are required for the bound to hold may not be reasonable in all settings; for example, the general selection bias assumption is unlikely to hold in case-control studies. In addition, the interpretation of the bias-adjusted confidence interval pertains to the application of the adjustment to repeated samples with the same sets of biases; if the biases were truly resolved in the design or analysis, the bounds would differ.

Finally, although we have suggested two possible orderings for factoring the bias, others that take into account, for example, misclassification that is also differential by an unmeasured confounder, are possible. We have presented results for risk ratios, which can in many cases be extended to odds ratios. However, our bound for exposure misclassification relies on a rare outcome assumption that limits its use and results in an approximate bound. Because we are never sure of the true values of the parameters that make up the bound, and the bound represents a worst-case scenario not likely to hold anyway, this approximation is not likely to meaningfully affect interpretation. Further work could be done to extend this approach to risk differences or mean differences, which may be especially challenging because the bounds are more frequently noninformative.⁴⁴ Other approaches exist to quantify as simply as possible unmeasured confounding in linear or probit models,^45–48 but to our knowledge, they have not yet been extended to multiple biases.

There is no single solution to the problem of bias in epidemiologic research. Some biases can be corrected at the design phase, others in the main analysis, but the assessment of what bias may remain should be a regular component of any study that attempts to quantify causal effects. The multiple-bias bound can make it simpler to do so, and we hope to encourage thoughtful consideration of multiple sources of bias in epidemiologic research.