Cornfield’s Pragmatic Contribution to Policy Discourse: Continued

Abstract

Cornfield et al.’s contributions included documenting epidemiological and policy debates regarding the effects of smoking on lung cancer. It was in the context of those debates that Cornfield et al. expressed sensitivity of the estimated effect of smoking to the dual components of confounding—relationship of a potential confounder (e.g., genes) to smoking (the focal predictor) and to lung cancer (the outcome). Current advancements have used various graphical and tabular representations to similarly express sensitivity to the dual components of confounding. As a representative of these advancements, we present Frank’s Impact Threshold for a Confounding (ITCV), which resolves the dual components of confounding into a single term based on the product of the confounder’s correlations with the focal predictor and with the outcome. We calculate the ITCV for an inference of an effect of kindergarten retention on achievement and provide benchmarks based on observed covariates to interpret the ITCV. Ultimately, any sensitivity analysis should only be applied after one has estimated the best possible model. The value of the sensitivity analysis is then to inform policy-oriented discourse among as wide a range of stakeholders as possible.

1. Introduction¹

To appreciate the full contribution of Cornfield, Haenszel, Hammond, Lilienfeld, Shimkin, and Wynder (1959) to discourse about policy related to smoking, it is critical to note that many questioned the effect of smoking on lung cancer in the context of lack of randomized experiments. For example, Fisher (1958) argued “both characteristics [smoking and lung cancer] might be largely influenced by a common cause [genotype]” (p. 108). Cornfield et al. (1959) famously countered that to reduce the estimated effect of smoking on lung cancer from observational studies to zero, an unobserved covariate “would need to be a near perfect predictor of lung cancer and about nine times more common among smokers than among nonsmokers” (as paraphrased by Rosenbaum, 2005, p. 1809).

Thus, Cornfield et al. (1959) not only established the conditions necessary to nullify the inference of an effect but they presented those conditions as a tool to inform debate about public policy. Note the subtitle to Cornfield et al. (1959): “Recent Evidence and a Discussion of Some Questions” (emphasis added), with the subtitle reflecting Cornfield’s open disposition towards debate and discussion as well as his pragmatic disposition to informing policy (Schlesselman, 2016). Ultimately, Cornfield was an acknowledged contributor to the U.S. Department of Health, Education, and Welfare (1964) report on smoking and lung cancer and Cornfield et al. (1959) was cited repeatedly (e.g., pp. 141, 183) as a basis of causal inference in the report. The report, in turn, affected public policy concerning the use, sale, and advertising of tobacco products (Alberg, Shopland, and Cummings, 2014, p. 407) in the United States (U.S. Public Health Service, 1989) and Britain (Berridge, 2006).

Cornfield et al. (1959) illustrated how sensitivity analysis dovetails with the desire to inform debate about policy or practice in at least two ways. First, policymakers must often make decisions in the absence of conclusive “proof” of efficacy or harm (Cornfield, 1976). This is particularly true in cases such as smoking and lung cancer, where the social costs of delaying the decision are especially large. By characterizing the conditions under which a scientific finding might be overturned, sensitivity analysis helps inform policy while acknowledging that not all alternative explanations have been accounted for.

Second, if scientific research on causal relationships is to inform policy and practice, it must be readily accessible to a broad range of stakeholders (Cornfield, 1976). The communication challenge is then to account for potential confounding in terms of multiple scenarios generated by the two components of a confounding variable—its relationship to the outcome (e.g., cancer) and its relationship to the focal predictor (e.g., smoking). Cornfield et al. addressed the communication challenge by allowing for the maximal relationship to the outcome (“a near perfect predictor of lung cancer”) and then focusing on the corresponding relationship to the focal predictor “nine times more common among smokers than among nonsmokers” necessary to nullify the inference. Taking the two components together, the statement established the extreme conditions necessary to nullify the inference of smoking’s effect on lung cancer and invited stakeholders to recognize how improbable such extreme conditions were.

In this paper, we briefly review how sensitivity analyses inspired by Cornfield et al. attempt to address the challenge of the dual components of confounding. We then illustrate the value of modern sensitivity analyses to policy discourse by applying one such approach—the Impact Threshold of a Confounding Variable (ITCV)—which aspires to be as accessible as Cornfield et al. while extending to a correlational framework, using an illustrative example concerning the potential effect of kindergarten retention on achievement.

2. The Dual Components of Confounding

Inspired by Cornfield et al., sensitivity analyses have proliferated in statistics and broadly in policy-oriented fields (e.g., Altonji, Elder, and Taber, 2005; Cinelli and Hazlett, 2020; Frank, 2000; Frank, Maroulis, Duong, and Kelcey, 2013; Frank et al., 2021; Gastwirth, Krieger, and Rosenbaum, 1998; Hong, Qin, and Yang, 2018; Hong, Yang, and Qin, 2021; Hosman, Hansen, and Holland, 2010; Hsu and Small, 2013; Ichino, Mealli, and Nannicini, 2008; Imbens, 2003; Knaeble and Dutter, 2017; Mauro, 1990; Park and Esterling, 2021; Robins, Rotnitzky, and Scharfstein, 2000; Rosenbaum, 1986; Rosenbaum and Rubin, 1983; VanderWeele and Ding, 2017; Veitch and Zaveri, 2020). These techniques take different approaches to expressing the dual relationships associated with an unobserved confounder. Some have expressed the dual relationships through tables for combinations of odds ratios or correlations associated with an unobserved covariate (e.g., Lin, Psaty, and Kronmal, 1998; Mauro, 1990; Rosenbaum, 1986, 2002), others through graphs such as contour plots (Cinelli and Hazlett, 2020; Hsu and Small, 2013; Imbens, 2003; Middleton, Scott, Diakow, and Hill 2016; Veitch and Zaveri, 2020), heat plots (Franks, D’Amour, and Feller, 2019) or through customized simulations (Carnegie, Harada, and Hill, 2016; Ichino et al., 2008). In widely cited papers, Altonji, Elder, and Taber (2000), Altonji et al. (2005) and Oster (2019) expressed the dual components of confounding in terms of commonly used two-stage econometric techniques—one stage to predict the outcome, Y, and a second to predict selection into the focal predictor, X. While each of these techniques faithfully represents the dual aspects of confounding they can be challenging for a broad set of stakeholders to utilize, requiring simultaneous consideration of multiple possible scenarios.

3. The Impact of a Confounding Variable²

A few techniques have attempted to simplify the expression of sensitivity by resolving the dual components of confounding into a single term, assuming the two components were equally strong (Cinelli and Hazlett, 2020; Frank, 2000; VanderWeele and Ding, 2017). Here, we present Frank’s (ITCV) as representative of these techniques, as well as those that extend Cornfield et al. to a correlational framework (see also Cinelli and Hazlett, 2020; Knaeble and Dutter, 2017; Mauro, 1990). Consider a model that was estimated:

$$\begin{array}{c}\mathit{Y}={\mathit{\beta }}_0+{\mathit{\beta }}_1\mathit{X}+{\mathit{\beta }}_2\mathit{Z}\mathrm{,}\end{array}$$ (1)

where Y is the outcome (or dependent variable), X is the focal predictor (or independent variable), and Z is an observed confounder. In the example we develop below, Y might be achievement, X kindergarten retention, and Z socioeconomic status, which is related to both kindergarten retention and achievement and is causally prior to both. Any observed association between kindergarten retention and achievement that does not account for the confounder of socioeconomic status may lead to an overestimate and invalid inference of the effect of kindergarten retention on achievement.

Ordinary Least Squares (OLS) estimates of models such as (1) adjust for confounding as a function of the correlation of the omitted variable with the focal predictor and with the outcome. Correspondingly, Frank (2000) expressed the dual associations of the confounder as the product of two correlations, defining the impact of a confounding variable:

$$\begin{array}{c}\mathit{\text{impact}}={\mathit{\rho }}_{\mathit{X}·\mathit{Z}}\times {\mathit{\rho }}_{\mathit{Y}·\mathit{Z}\mathrm{,}}\end{array}$$

where ${\mathit{\rho }}_{\mathit{X}·\mathit{Z}}$ is the population correlation between X and Z and ${\mathit{\rho }}_{\mathit{Y}·\mathit{Z}}$ is the population correlation between Y and Z. Examples of this type of product can be traced back to Cochran (1938), if not earlier (e.g., to Fisher, 1935), as well as to expressions for partial correlations (Cohen and Cohen, 1983) and omitted variable bias in econometrics (Wooldridge, 2010). Through the product, each component of confounding is important in proportion to the size of the other; relationships with the outcome are important for variables strongly related to the focal predictor, and vice versa.

The functional form of the product is also implied by other sensitivity analyses. The curvature in Imbens (2003) line plot implies that small increases in the partial ${\mathit{R}}^2$ between omitted variables and assignment (variance explained by an omitted variable conditional on observed covariates) reduce an estimate by the targeted amount for large values of the partial ${\mathit{R}}^2$ with the outcome, and vice versa (others such as Carnegie et al., 2016; Cinelli and Hazlett, 2020; Dorie, Harada, Carnegie, and Hill, 2016; Knaeble and Dutter, 2017 have extended this to contour plots and graphs). See similar implications for binary outcomes (Rosenbaum, 2002; Harding, 2003) and in a propensity score framework (Hirano and Imbens, 2001; Hong et al., 2021) or mediation (Imai, Keele, and Yamamoto, 2010).

3.1. Sensitivity: The Impact Threshold for a Confounding Variable (ITCV)

Frank (2000) leveraged the definition of impact to develop a sensitivity index quantifying how strong the impact of an unobserved confounder would have to be to nullify an inference—the ITCV. Consider that the estimate of ${\mathit{\beta }}_1$ in model (1) is biased because the model omitted a Confounding Variable ($\mathit{C}\mathit{V}$) that should be included in the true model:

$$\begin{array}{c}\mathit{Y}={\mathit{\beta }}_0+{\mathit{\beta }}_1\mathit{X}+{\mathit{\beta }}_2\mathit{Z}+{\mathit{\beta }}_3\mathit{C}\mathit{V}.\end{array}$$ (2)

The question for decision makers is then how strong the $\mathit{i}\mathit{m}\mathit{p}\mathit{a}\mathit{c}\mathit{t}$ of the $\mathit{C}\mathit{V}$ must be to nullify the inference of an effect of X on Y. A straightforward way (Frank et al., 2008) to calculate the ITCV leverages the fact that there is a one-to-one correspondence between a t-ratio for the inference for ${\mathit{\beta }}_1$ in equation (2) and a partial correlation. Specifically,

$$\begin{array}{c}{\mathit{t}}_{\mathit{X}·\mathit{Y}\mid \mathit{Z}\mathrm{,}\mathit{C}\mathit{V}}={\displaystyle \frac{{\hat{\mathit{\beta }}}_{1\mid \mathit{Z}\mathrm{,}\mathit{C}\mathit{V}}}{se\mathrm{(}{\hat{\mathit{\beta }}}_{1\mid \mathit{Z}\mathrm{,}\mathit{C}\mathit{V}}\mathrm{)}}}\mathrm{,}\end{array}$$ (3a)

where “∣” indicates partialling for, and

$$\begin{array}{c}{\mathit{r}}_{\mathit{X}·\mathit{Y}\mid \mathit{Z}\mathrm{,}\mathit{C}\mathit{V}}={\displaystyle \frac{{\mathit{t}}_{\mathit{X}·\mathit{Y}\mid \mathit{Z}\mathrm{,}\mathit{C}\mathit{V}}}{\sqrt{\mathit{d}\mathit{f}+{\mathit{t}}_{\mathit{X}·\mathit{Y}\mid \mathit{Z}\mathrm{,}\mathit{C}\mathit{V}}^2}}}\mathrm{,}\end{array}$$ (3b)

where $\mathit{d}\mathit{f}$ is the degrees of freedom used to estimate equation (2). In turn, the partial correlation can be expressed as a function of the dual components of confounding (Cohen and Cohen, 1983):

$$\begin{array}{c}{\mathit{r}}_{\mathit{X}·\mathit{Y}\mid \mathit{C}\mathit{V}\mathrm{,}\mathit{Z}}={\displaystyle \frac{{\mathit{r}}_{\mathit{X}·\mathit{Y}\mid \mathit{Z}}-{\mathit{r}}_{\mathit{X}·\mathit{C}\mathit{V}\mid \mathit{Z}}\times {\mathit{r}}_{\mathit{Y}·\mathit{C}\mathit{V}\mid \mathit{Z}}}{\sqrt{\left(1-{\mathit{r}}_{\mathit{X}·\mathit{C}\mathit{V}\mid \mathit{Z}}^2\right)\left(1-{\mathit{r}}_{\mathit{Y}·\mathit{C}\mathit{V}\mid \mathit{Z}}^2\right)}}}.\end{array}$$ (4)

Note that ${\mathit{r}}_{\mathit{X}·\mathit{C}\mathit{V}\mid \mathit{Z}}{\mathit{r}}_{\mathit{Y}·\mathit{C}\mathit{V}\mid \mathit{Z}}$ appears in the numerator of (4) and each component separately appears in the denominator.

Similar to Frank (2000), Xu, Frank, Maroulis, and Rosenberg (2019) show that the maximum of equation (4) occurs for ${\mathit{r}}_{\mathit{X}·\mathit{C}\mathit{V}\mid \mathit{Z}}={\mathit{r}}_{\mathit{Y}·\mathit{C}\mathit{V}\mid \mathit{Z}}$ (Franks et al., 2019, also observed that estimates changed most when ${\mathit{r}}_{\mathit{X}·\mathit{C}\mathit{V}\mid \mathit{Z}}={\mathit{r}}_{\mathit{Y}·\mathit{C}\mathit{V}\mid \mathit{Z}}$). That is, the smallest possible product ${\mathit{r}}_{\mathit{X}·\mathit{C}\mathit{V}\mid \mathit{Z}}{\mathit{r}}_{\mathit{Y}·\mathit{C}\mathit{V}\mid \mathit{Z}}$ that could reduce ${\mathit{r}}_{\mathit{X}·\mathit{Y}\mid \mathit{Z}\mathrm{,}\mathit{C}\mathit{V}}$ below a threshold occurs for ${\mathit{r}}_{\mathit{X}·\mathit{C}\mathit{V}\mid \mathit{Z}}={\mathit{r}}_{\mathit{Y}·\mathit{C}\mathit{V}\mid \mathit{Z}}$. Thus, making the assumption that ${\mathit{r}}_{\mathit{X}·\mathit{C}\mathit{V}\mid \mathit{Z}}={\mathit{r}}_{\mathit{Y}·\mathit{C}\mathit{V}\mid \mathit{Z}}$ favors the challenger of the inference, consistent with a conservative stance protecting a null hypothesis (Cinelli and Hazlett make a similar assumption without claiming maximization).

Assuming ${\mathit{r}}_{\mathit{X}·\mathit{C}\mathit{V}\mid \mathit{Z}}={\mathit{r}}_{\mathit{Y}·\mathit{C}\mathit{V}\mid \mathit{Z}}$, one can substitute $\mathit{i}\mathit{m}\mathit{p}\mathit{a}\mathit{c}\mathit{t}$ for ${\mathit{r}}_{\mathit{X}·\mathit{C}\mathit{V}\mid \mathit{Z}}\times {\mathit{r}}_{\mathit{Y}·\mathit{C}\mathit{V}\mid \mathit{Z}}$, ${\mathit{r}}_{\mathit{X}·\mathit{C}\mathit{V}\mid \mathit{Z}}^2$, and ${\mathit{r}}_{\mathit{Y}·\mathit{C}\mathit{V}\mid \mathit{Z}}^2$ in (4), yielding:

$$\begin{array}{c}{\mathit{r}}_{\mathit{X}·\mathit{Y}|\mathit{Z}\mathrm{,}\mathit{C}\mathit{V}}={\displaystyle \frac{{\mathit{r}}_{\mathit{X}·\mathit{Y}\mid \mathit{Z}}-\mathit{i}\mathit{m}\mathit{p}\mathit{a}\mathit{c}\mathit{t}}{1-|\mathit{i}\mathit{m}\mathit{p}\mathit{a}\mathit{c}\mathit{t}|}}.\end{array}$$ (5)

Then, setting ${\mathit{r}}_{\mathit{X}·\mathit{Y}|\mathit{Z}\mathrm{,}\mathit{C}\mathit{V}}$ equal to a threshold value for making an inference, ${\mathit{r}}^{\mathrm{\#}}$, and solving for $\mathit{i}\mathit{m}\mathit{p}\mathit{a}\mathit{c}\mathit{t}$ yields the ITCV:

$$\begin{array}{c}\mathit{I}\mathit{T}\mathit{C}\mathit{V}={\displaystyle \frac{{\mathit{r}}_{\mathit{X}·\mathit{Y}\mid \mathit{Z}}-{\mathit{r}}^{\mathrm{\#}}}{1-|{\mathit{r}}^{\mathrm{\#}}|}}.\end{array}$$ (6)

Thus, the partial correlation ${\mathit{r}}_{\mathit{X}·\mathit{Y}|\mathit{Z}\mathrm{,}\mathit{C}\mathit{V}}$ will fall below the threshold value of ${\mathit{r}}^{\mathrm{\#}}$ if the $\mathit{i}\mathit{m}\mathit{p}\mathit{a}\mathit{c}\mathit{t}$ (${\mathit{r}}_{\mathit{X}·\mathit{C}\mathit{V}\mid \mathit{Z}}{\mathit{r}}_{\mathit{Y}·\mathit{C}\mathit{V}\mid \mathit{Z}}$) of an omitted confounder is greater than $\frac{{\mathit{r}}_{\mathit{X}·\mathit{Y}\mid \mathit{Z}}-{\mathit{r}}^{\mathrm{\#}}}{1-|{\mathit{r}}^{\mathrm{\#}}|}$, defining the ITCV.

In the spirit of Cornfield et al.’s transparent and parsimonious expression of sensitivity, the closed-form expression in (6), derived from a partial correlation, supports intuition about sensitivity. Specifically, as in the numerator, sensitivity about an inference is based on the difference between the estimated effect and the threshold for inference (${\mathit{r}}_{\mathit{X}·\mathit{Y}\mid \mathit{Z}}-{\mathit{r}}^{\mathrm{\#}}$). This difference is then scaled relative to the threshold in the denominator ($1-{\mathit{r}}^{\mathrm{\#}}$)—an inference based on a given difference is less robust if the threshold for making the inference is small.

A key feature of expressing sensitivity in terms of the partial correlation ${\mathit{r}}_{\mathit{X}·\mathit{Y}|\mathit{Z}\mathrm{,}\mathit{C}\mathit{V}}$ is that the threshold for inference can directly account for sampling variability. Although ${\mathit{r}}^{\mathrm{\#}}$ can represent any specified threshold, the expression in (3b) can be adapted to define a threshold for statistical significance:

$$\begin{array}{c}{\mathit{r}}_{\mathit{s}\mathit{t}\mathit{a}\mathit{t}\mathit{s}\mathit{i}\mathit{g}}^{\mathrm{\#}}={\displaystyle \frac{{\mathit{t}}_{\mathit{c}\mathit{r}\mathit{i}\mathit{t}\mathit{i}\mathit{c}\mathit{a}\mathit{l}}}{\sqrt{\mathit{d}\mathit{f}+{\mathit{t}}_{\mathit{c}\mathit{r}\mathit{i}\mathit{t}\mathit{i}\mathit{c}\mathit{a}\mathit{l}}^2}}}\mathrm{,}\end{array}$$ (7)

where ${\mathit{t}}_{\mathit{c}\mathit{r}\mathit{i}\mathit{t}\mathit{i}\mathit{c}\mathit{a}\mathit{l}}$ is the critical value of the t-distribution for specified level of statistical significance (e.g., $\mathit{\alpha }=.05$) and $\mathit{d}\mathit{f}$. Because the inference based on ${\hat{\mathit{\beta }}}_1$ for (2) is identical to that for the partial correlation in (4), the expressions in (5) through (7) directly account for changes in the estimated effect and its standard error. Correspondingly, when the impact of an omitted variable is greater than the ITCV defined by ${\mathit{r}}_{\mathit{s}\mathit{t}\mathit{a}\mathit{t}\mathit{s}\mathit{i}\mathit{g}}^{\mathrm{\#}}$, ${\hat{\mathit{\beta }}}_1$ would not be statistically different from zero if the omitted variable were included in the model (see Frank, 2000 or Frank, Lin, Xu, Maroulis, and Mueller, 2023 for equivalent expressions of the ITCV directly based on estimated effects and their standard errors).

3.2. Policy Application: Kindergarten Retention

We apply the ITCV to quantify the robustness of an inferred negative effect of retention on achievement. Retention is a long-studied and large-scale phenomenon (Berne, Jacob, Weiland, and Strunk, 2026; Cummings and Turner, 2020; Mittelstet, 2025; for Education Statistics, 2019), with the U.S. Department of Health and Human Services (2000) estimating that 8% of second graders (more than five hundred thousand) were a year behind their expected grade level as a result of not being promoted, known as retention, in kindergarten or first grade (see also Alexander, Entwisle, and Dauber, 2003, p. 1). As Hong and Raudenbush (2005, p. 205) note in their abstract, “grade retention might harm high-risk students by limiting their learning opportunities.” Furthermore, a disproportionate percentage of those retained are from low socioeconomic backgrounds and/or are racial minorities (Alexander et al., 2003, Chap. 5).

Similar to the context for challenges regarding the inferences of the effects of smoking on lung cancer, there is an absence of randomized experiments of the effects of kindergarten retention on achievement, partly because such experiments would be unethical (Alexander et al., 2003). Therefore, researchers have turned to observational studies that employ statistical controls to estimate effects of retention on achievement. Of the studies of retention effects (Burkam, LoGerfo, Ready, and Lee, 2007; Jimerson, 2001; Shepard and Smith, 1989), we focus on Hong and Raudenbush (2005)’s analysis of nationally representative data in the Early Childhood Longitudinal Study (ECLS), which included extensive covariates including student background, emotional disposition, motivation, and pretests.

Hong and Raudenbush (2005) used the measures described above in a propensity score model to define a “retained counterfactual” group approximating what would have happened to the students who were retained if they instead had been promoted (e.g., Rubin, 1974; Holland, 1986). Hong and Raudenbush then estimated that the “retained observed” group scored 9.01 points lower on reading achievement than the “retained counterfactual” group at the end of first grade. The estimated effect was about two thirds of a standard deviation on the test, almost half a year’s expected growth (Hong and Raudenbush, 2005, p. 220), and was statistically significant ($\mathit{p}\le .001$, with standard error of .68, and t-ratio of $-13.67$). Ultimately, Hong and Raudenbush concluded that retention reduces achievement: “children who were retained would have learned more had they been promoted” (Hong and Raudenbush, 2005, p. 200).

Hong and Raudenbush (2005) relied on statistical controls with covariates to approximate equivalence between the retained and promoted groups. But they may not have conditioned for some confounding factor unmeasured in the data set, such as an aspect of a child’s cognitive ability, emotional disposition, or motivation. Instead of abandoning the process of inference altogether, we quantify the conditions necessary to change the inference to inform debate about the inference.

Consider Figure 1, in which the relationship of interest is between kindergarten retention (X) and reading achievement (Y), which could be impacted by an omitted confounding variable (e.g., motivation). The center of the Figure represents how the two components of confounding are resolved into the single term ${\mathit{r}}_{\mathit{\text{Retention}}·\mathit{C}\mathit{V}}\times {\mathit{r}}_{\mathit{\text{Achievement}}·\mathit{C}\mathit{V}}$ to impact the relationship between retention and achievement.

Drawing on Figure 1, consider the model that was estimated:

$$\begin{array}{c}\text{Achievement}={\mathit{\beta }}_0+{\mathit{\beta }}_1\text{Retention}+{\mathit{\beta }}_2\mathbf{\text{Z}}\mathrm{,}\end{array}$$ (8)

where Z is a vector of propensity strata representing observed covariates. Now assume ${\hat{\mathit{\beta }}}_1$ is statistically significant, rejecting the null hypothesis that ${\mathit{\beta }}_1=0$ for (8). But there may be a skeptic who challenges the inference. In the example of smoking and lung cancer, the skeptic Fisher challenged the inference based on omitted genetic information, thereby protecting the null hypothesis and corresponding policy.

Figure 1.

The impact of a confounding variable.

For the example of kindergarten retention on achievement, the skeptic’s model might be:

$$\begin{array}{c}\text{Achievement}={\mathit{\beta }}_0+{\mathit{\beta }}_1\text{Retention}+{\mathit{\beta }}_2\mathbf{\text{Z}}+{\mathit{\beta }}_3\text{Motivation}\mathrm{,}\end{array}$$ (9)

for which Motivation is omitted from the propensity model used to define the strata in (8). That is, the skeptic contends that ${\hat{\mathit{\beta }}}_1$ for (9) would not exceed the threshold for inference, even though it did so in (8).

Either in response to, or in anticipation of, such debates, researchers routinely employ controls for observed confounds through estimation techniques (e.g., regression analysis, propensity scores, regression discontinuity, instrumental variables). In fact, Hong and Raudenbush (2005, p. 215) used propensity strata to control for 207 observed covariates including pre-tests and teachers’ ratings of children. But what if ${\hat{\mathit{\beta }}}_1$ is still statistically significant even after controlling for observed confounds? Concerns about omitted variables might persist because it may be difficult to exhaustively account for all confounders with observed variables. The pragmatic question motivated by Cornfield et al. is then can the evidence be strong enough, relative to a threshold for inference, to inform action, even if there are potentially omitted variables (Frank et al., 2023)?

To address the question regarding the strength of evidence, we apply the ITCV to Hong and Raudenbush’s (2005) estimated effect of kindergarten retention on achievement. Hong and Raudenbush’s observed t-ratio of $-13.67$ can be converted to a correlation using (3b). Specifically, using a sample size of 7,639 (7,168 promoted students, 471 retained students), 221 covariates (including 14 propensity strata) used to estimate the effects, and assuming one extra covariate to be added to the model:

$$\begin{array}{c}{\mathit{r}}_{\mathit{\text{Retention}}·\mathit{\text{Achievement}}\mid \mathit{\text{Covariates}}}={\displaystyle \frac{\mathit{t}}{\sqrt{\mathrm{(}\mathit{n}-\mathit{q}-1\mathrm{)}+{\mathit{t}}^2}}}={\displaystyle \frac{-13.25}{\sqrt{\mathrm{(}7639-221-2\mathrm{)}+{\mathrm{(}13.25\mathrm{)}}^2}}}=-\mathrm{0.152.}\end{array}$$

Using equation (7), the degrees of freedom³ of 7416 yields a threshold correlation, ${\mathit{r}}^{\mathrm{\#}}$:

$$\begin{array}{c}{\mathit{r}}_{\mathit{\text{statsig}}}^{\mathrm{\#}}={\displaystyle \frac{{\mathit{t}}_{\mathit{\text{critical}}}}{\sqrt{\mathrm{(}\mathit{n}-\mathit{q}-1\mathrm{)}+{\mathit{t}}_{\mathit{\text{critical}}}^2}}}={\displaystyle \frac{-1.96}{\sqrt{\mathrm{(}7639-221-2\mathrm{)}+{1.96}^2}}}=-\mathrm{0.023.}\end{array}$$

From (6),

$$\begin{array}{c}\text{ITCV}={\displaystyle \frac{{\mathit{r}}_{\mathit{\text{Retention}}\mathrm{,}\mathit{\text{Achievement}}\mid \mathit{\text{Covariates}}}-{\mathit{r}}_{\mathit{\text{statsig}}}^{\mathrm{\#}}}{1-|{\mathit{r}}_{\mathit{\text{statsig}}}^{\mathrm{\#}}|}}={\displaystyle \frac{-0.152-\mathrm{(}-0.023\mathrm{)}}{1-|-0.023|}}=-\mathrm{0.132.}\end{array}$$

And the derivation of the ITCV can be verified from (5):

$$\begin{array}{c}{\mathit{r}}_{\mathit{\text{Retention}}\mathrm{,}\mathit{\text{Achievement}}\mid \mathit{\text{Covariates}}\mathrm{,}\mathit{C}\mathit{V}}={\displaystyle \frac{-0.152-\mathrm{(}-0.132\mathrm{)}}{1-|-0.132|}}=-0.023={\mathit{r}}_{\mathit{\text{statsig}}}^{\mathrm{\#}}.\end{array}$$

An omitted confounding variable would have to have an impact of ${\mathit{r}}_{\mathit{X}·\mathit{C}\mathit{V}\mid \mathit{\text{Covariates}}}\times {\mathit{r}}_{\mathit{Y}·\mathit{C}\mathit{V}\mid \mathit{\text{Covariates}}}=-.132$, with component correlations of ${.132}^{1\mathrm{/}2}=.36$ (taking opposite signs), to result in a partial correlation between retention and achievement of $-.023$ (associated with a p-value of .05). Correspondingly, if an omitted variable with impact greater in magnitude than .132 were included in the model (assuming ${\mathit{r}}_{\mathit{X}·\mathit{C}\mathit{V}\mid \mathit{Z}}={\mathit{r}}_{\mathit{Y}·\mathit{C}\mathit{V}\mid \mathit{Z}}$), the estimated effect of kindergarten retention on achievement would not be statistically significant (i.e., $\mathit{p}\mathrm{>}.05$).⁴

For an observational study, it is certainly quite likely that at least one confounding variable was omitted from the model. But the ITCV quantifies what the properties of that omitted variable must be to change the inference. Specifically, an omitted variable would have to be correlated with achievement at .36 and with retention at $-.36$ (signs are interchangeable) to nullify the inference (accounting for sampling variability) of an effect of kindergarten retention on achievement. In this way, the ITCV informs comparisons of the weight of evidence against concerns about an incorrectly specified model, ultimately informing discourse about inference that can be the basis for public action, such as in Cornfield et al.

3.3. Benchmarks for the ITCV based on correlations associated with observed variables

Beginning with Rosenbaum (1986), sensitivity analyses have extended Cornfield et al. by benchmarking the properties of an omitted variable necessary to nullify an inference against the properties of observed covariates (Altonji et al., 2005; Cinelli and Hazlett, 2020; Frank, 2000; Oster, 2019). For example, the ITCV can be expressed relative to the impact of an observed benchmark covariate Z (Frank, 2000; Frank et al., 2023):

$$\begin{array}{c}\text{ITCV}\mathrm{(}\text{benchmark}\mathrm{)}={\displaystyle \frac{\text{ITCV}}{{\mathit{r}}_{\mathit{Y}·\mathit{Z}}\times {\mathit{r}}_{\mathit{X}·\mathit{Z}}}}={\displaystyle \frac{{\mathit{r}}_{\mathit{Y}·\mathit{C}\mathit{V}}\times {\mathit{r}}_{\mathit{X}·\mathit{C}\mathit{V}}}{{\mathit{r}}_{\mathit{Y}·\mathit{Z}}\times {\mathit{r}}_{\mathit{X}·\mathit{Z}}}}.\end{array}$$ (10)

Therefore, $\text{ITCV}\mathrm{(}\text{benchmark}\mathrm{)}\mathrm{>}1$ indicates that, to nullify the inference for ${\mathit{\beta }}_1$, the impact of an unobserved covariate would have to be greater than the impact of the observed covariate (Altonji et al., 2005; Oster, 2019). In the kindergarten retention example, the impact of “student approaches to learning (SAL)” (the strongest covariate identified by Hong and Raudenbush, 2005) is ${\mathit{r}}_{\mathit{\text{SAL}}·\mathit{\text{Retention}}}\times {\mathit{r}}_{\mathit{\text{SAL}}·\mathit{\text{Achievement}}}=\mathrm{(}-0.1849\mathrm{)}\times \mathrm{(}0.4442\mathrm{)}=-0.08$. Correspondingly, the impact of an omitted variable would have to be $65\mathrm{\%}$ stronger than the impact of the strongest covariate to change the inference $-0.132\mathrm{/}\mathrm{(}-0.08\mathrm{)}=1.65$.

One can also benchmark conditional on other variables in the model, such as pretests. For example, once adjusting for pretests, the impact of SAL is:

$$\begin{array}{c}{\mathit{r}}_{\mathit{\text{SAL}}·\mathit{\text{Retention}}\mid \mathit{\text{Pretests}}}\times {\mathit{r}}_{\mathit{\text{SAL}}·\mathit{\text{Achievement}}\mid \mathit{\text{Pretests}}}=\mathrm{(}-0.0286\mathrm{)}\times \mathrm{(}0.0441\mathrm{)}=-\mathrm{0.00126.}\end{array}$$

The impact of an omitted variable necessary to nullify the inference (0.132) would have to be more than 100 times greater than the impact (0.00126) of the strong covariate SAL, once adjusting for pretests. This type of benchmarking can help stakeholders assess the plausibility of an omitted variable with the properties necessary to nullify the inference.⁵

4. Discussion

Sensitivity analyses do not alter estimates or inferences; they inform our interpretation and ability to communicate the robustness of those inferences. As in Cornfield et al., a fundamental goal is to express sensitivity in parsimonious and accessible terms that are nonetheless functions of the dual relationships associated with the confounder—the relationship with the focal predictor and with the outcome. By doing so, sensitivity analyses can help policymakers incorporate the evidence in their assessment about the benefits and costs of a particular policy or practice.

In this paper, we presented the Impact Threshold for a Confounding Variable (ITCV) that represents many techniques which foreground omitted variables in the linear model. The ITCV captures the dual components of confounding as the product of the two correlations associated with potentially omitted variables, and accounts for sampling variability by defining the threshold for inference in terms of the partial correlation (e.g., as associated with a p-value of 0.05). In the empirical example, the ITCV shows an omitted variable would have to be correlated at .36 with kindergarten retention and with achievement (taking opposite signs) to nullify Hong and Raudenbush’s (2005) inference of a negative effect of retention on achievement. The sensitivity analysis has turned the challenge “But there may be an omitted variable that biases the estimated effect of kindergarten retention on achievement” to a statement about the omitted variable such as “An omitted variable would have to be correlated at .36 (taking opposite signs) with retention and with achievement to change the inference of an effect of retention on achievement.” Policy then depends on discourse among a broad set of stakeholders that may include the plausibility of such an omitted variable (Cornfield, 1976).

4.1. Recommended Practices for Sensitivity Analysis

Sensitivity analyses assume that models have already been developed appropriately and are inclusive of alternative explanations represented by observed variables—Stated plainly, the first best practice is that sensitivity analysis should only be conducted after a researcher has fully specified a model including all relevant and available measures. This especially applies to leveraging longitudinal data, as including pretests (i.e., lagged dependent variables) can reduce bias by 60–$90\mathrm{\%}$ versus comparable randomized experiments (e.g., Shadish, Clark, and Steiner, 2008; see the review in Wong, Valentine, and Miller-Bains, 2017). See also recent work by Belloni, Chernozhukov, and Wei (2016), Young and Holsteen (2017) and Young (2018) regarding model selection. It is misleading to apply sensitivity analyses to models that have not already been rigorously vetted given the available data. But once the best models have been estimated and adjudicated, there may still be concerns about potential bias through omitted variables or sampling variability. Sensitivity analyses can then inform discussions about inferences by quantifying the terms of uncertainty about inferences rooted in the underlying frameworks applied to data analysis (omitted variables and potential outcomes).

Second, sensitivity analyses may best support rigorous science when used with conventional, transparent methods that can be interpreted by a range of consumers and producers of research. In so doing, sensitivity analyses allow one to plainly state the adjustments employed in a model while acknowledging that there may still be other factors unaccounted for. That is, sensitivity analyses focus scientific discourse on the factors that were and were not controlled for, rather than on estimation techniques that may be more difficult to track – see Holland (1989), who asked, “A menu of estimators or a sensitivity analysis? (p. 876)”. Ultimately, the transparency of the models and interpretation among researchers can transfer to discourse among a broad range of stakeholders in a policy.

It may be tempting to seek a threshold for a sensitivity index itself such as “An ITCV greater than __ indicates a robust inference.” But we believe sensitivity analyses should inform, not preempt by virtue of exceeding a threshold, the discourse among researchers, consumers, and stakeholders invested in an inference. Nonetheless, sensitivity indices can be evaluated against benchmarks defined by observed covariates.

5. Conclusion

Sensitivity analyses help interpreters of research to make evidence-informed decisions while acknowledging that there may be alternative factors at work or heterogeneity of effects in different populations. This supports pragmatic utilization of a mixture of study designs instead of reliance exclusively on RCTs (Cornfield, 1976) that may be difficult to replicate when sample sizes are small (Moody, Keister, and Ramos, 2022). It also acknowledges the reality that policymakers often do not have the luxury to wait for ideal evidence. Sensitivity analyses provide a precise language for discourse about imperfect evidence, ultimately supporting better, more informed, decision-making. This may be especially crucial when, as a society, we must weigh the expected benefits and harms of action against the consequences of inaction, such as in the case of smoking and lung cancer.

Funding Statement

This work was supported by U.S. Department of Education Institute for Education Sciences through R305D220022 to Michigan State University. The opinions expressed are those of the authors and do not represent views of the Institute for Education Sciences or the U.S. Department of Education.