The Wizard of Odds

In the 1939 Metro-Goldwyn-Mayer film, The Wizard of Oz, the benevolent Wizard is asked to solve the existential dilemmas of the Scarecrow, Tin Man and Cowardly Lion. Seemingly unable to provide them the intangible things that they seek, he instead gives them three mundane items that he happens to have in his bag. The munificence of the Wizard is not cast into doubt by this sleight of hand, however, because the characters discover that the traits they sought were things they had possessed all along. The article by VanderWeele and colleagues¹ responds to a crisis in reproductive epidemiology over how to handle measures of maturity and development. Variables such asgestational age and birth weight are ubiquitous in perinatal research, but problematic for all the reasons described in their paper. Analysts want to condition on some measure of maturation to put neonates on an equal developmental footing, but exposures of interest may affect the progress of the pregnancy, making conditional estimates biased as measures of total effect. At the same time, direct effects are generally not identifiable because of unmeasured common causes of the maturity indicator and the outcome, leading to the birth-weight paradox described by the authors. Reproductive epidemiologists therefore look eagerly to methods wizards for a magical solution to their problem: for the brains, heart and courage necessary for their daily travails. What have the methodologists got in their bag?

The first method proposed by VanderWeele et al ¹ is to estimate the total effect of an exposure such as smoking on neonatal mortality within strata defined by the estimated risk of a developmental intermediate such as low birthweight. The authors predict the risk of low birthweight through a regression model, and create categories of high and low risk. This is a straightforward extension of the typical practice of estimating total effects within strata of baseline covariates. In this case, the strata are defined as some more complicated function of the covariates, but the approaches are similar; they estimate the total effect of smoking among sub-categories of the population. The ability of this method to resolve the birth-weight paradox depends, however, on the discriminatory power of the risk score and the prevalence of the intermediate. The authors estimate the effect of smoking on neonatal mortality as OR_{high risk}=1.6 and OR_lowrisk=1.3, among those at high and low risk. Rather than implying a resolution of the paradox, however, these estimates are entirely compatible with a state of nature in which smoking actually has a protective effect among low-birth-weight infants.

Imagine we know p_am = Pr(Death|SET[Smoke = a], SET[LBW = m], where smoke=1 indicates maternal smoking and LBW=1 indicates infant low birthweight. If we posit values of p_am we can ascertain how well the authors’ stratification appears to resolve the paradox. Suppose, for example, that the birth-weight paradox is not actually a paradox, and that smoking is actually beneficial in one stratum and harmful in the other. If we take as parameters p₁₁=0.04, p₀₁=0.06, p₁₀=0.05, and p₀₀=0.03, then in the absence of confounding, the true controlled direct effect of smoking among normal-weight infants would be OR_(m=0) = 1.7 while among low-birth-weight infants it would be OR_(m=1) = 0.7. Suppose we apply the stratification proposed by the authors¹ and our classification scheme has some associated set of positive and negative predictive values (ppv_(a=1), ppv_(a=0), npv_(a=1) and npv_(a=0) ). The stratified total effects obtained by the proposed method are given by the formulae:

$$\begin{array}{l}{OR}_{\mathit{high}\mathit{risk}}=\frac{[p_{11}\ast {\mathit{ppv}}_{(a=1)}+p_{10}(1-{\mathit{ppv}}_{(a=1)})]/[1-\{p_{11}\ast {\mathit{ppv}}_{(a=1)}+p_{10}(1-{\mathit{ppv}}_{(a=1)})\}]}{[p_{01}\ast {\mathit{ppv}}_{(a=0)}+p_{00}(1-{\mathit{ppv}}_{(a=0)})]/[1-\{p_{01}\ast {\mathit{ppv}}_{(a=0)}+p_{00}(1-{\mathit{ppv}}_{(a=0)})\}]}\\ {OR}_{\mathit{low}\mathit{risk}}=\frac{[p_{10}\ast {\mathit{npv}}_{(a=1)}+p_{11}(1-{\mathit{npv}}_{(a=1)})]/[1-\{p_{10}\ast {\mathit{ppv}}_{(a=1)}+p_{11}(1-{\mathit{npv}}_{(a=1)})\}]}{[p_{00}\ast {\mathit{npv}}_{(a=0)}+p_{01}(1-{\mathit{npv}}_{(a=0)})]/[1-\{p_{00}\ast {\mathit{npv}}_{(a=0)}+p_{01}(1-{\mathit{npv}}_{(a=0)})\}]}\end{array}$$ (1)

For example, if the risk classification model has ppv_(x=1) = ppv_(x=0) = 0.9 and npv_(x=1) = npv_(x=0) =0.9, then equation (1) yields OR_{low risk}=1.5 and OR_{high risk}=0.7, which are similar to the controlled direct effects. However, an intermediate with a low prevalence such as low birthweight will have poor ppv unless the specificity is nearly 1.0. If the risk classification model has ppv_(x=1) = ppv_(x=0) = 0.3 and npv_(x=1) = npv_(x=0) =0.9 then equation (1) yields OR_{low risk}=1.5 and OR_{high risk}=1.2. A practitioner could be tempted to believe these results imply that the qualitative effect modification referred to as a “paradox” does not really exist when, in fact, it does. In the absence of extremely good predictive models for low birthweight, it is therefore hard to have much faith in the ability of this method to detect a “paradox” were it actually to exist. The authors’ finding of harmful effects of smoking in both risk strata cannot be taken as evidence of having “resolved” the paradox; in the absence of an excellent prediction tool for the intermediate, it is the answer one would expect regardless of the truth.

Jumping to the third method, the authors¹ propose estimation of the principal-stratum direct effect. Resolution of the problem by means of restriction to a stratum in which exposure has no effect on the intermediate predates the cited article of Frangakis and Rubin² --for example, in the work of Joffe et al.^3,4 But Joffe and colleagues identified this stratification on substantive grounds, whereas the authors’ approach here treats it as a latent factor that must be estimated with a model. The proposed sensitivity analysis parameter is an honest way of dealing with the inherent uncertainty of this estimation, but one that will often be difficult in practice. There is little substantive guidance available about the likely magnitude of this parameter, only its sign. Even if one happens to get this value right in the analysis, however, the effect estimate must be applied in public-health terms to a sub-population whose membership and prevalence are unknown.⁵ The authors are quite frank about all of these dilemmas, which leaves them not entirely sanguine about the prospects for this approach.

This leaves approach 2, which has been applied elegantly in previous papers,⁶ and which we agree will be of greatest value. This approach follows a long line of methodological development for effect estimation in the presence of unmeasured covariates.⁷ By specifying the prevalence of an unmeasured confounder within strata of the intermediate, as well as the magnitude of its effect on the outcome, the authors ¹ can correct the observed direct-effect estimates. As they note, one need not specify the parameters perfectly; it can be illuminating merely to estimate the extent of bias under alternative guesses at the bias parameters. But let’s go one step further by rephrasing the question: what values of the parameters given in the author’s expression (3) could resolve the birthweight paradox? The resolution of the paradox can be viewed in different ways; however, because the greatest concern seems to be that smoking could appear to be protective in some stratum, one might wonder what parameter values lead to bias equal to the observed OR, and therefore to an adjusted OR=1.0. To determine this, one can rearrange expression (3) given by the authors as:

$$\gamma =\frac{{\pi }_{1m}+B-B\ast {\pi }_{om}-1}{{\pi }_{1m}-B\ast {\pi }_{om}}$$ (2)

where π_am is the prevalence of the unmeasured confounder among those with smoking status a in low-birth-weight stratum m, B is the amount of bias necessary to render the observed association null, and is the unmeasured relationship between confounder and disease. In this case, a value of B=0.76 would lead to a perfectly null bias-adjusted OR and a resolution of the paradox. Rather than examine a few token examples, we plot the values of γ necessary at all combinations of π_0m and π_1m in Figure 1. Notably, for some prevalence combinations there is no value of γ that can explain the paradox according to this formula. Moreover, prevalence combinations that approach π_0m = B * π_νm require increasingly large confounder-outcome associations to explain the paradox, while prevalence combinations further from that line require less substantial effects. In this setting, with limited substantive knowledge, it may be more useful to examine a graph such as this rather than two or three discrete points.

Figure 1.

Magnitude of confounder-disease association necessary to result in an adjusted OR_m=1=1.0 (B=0.76) for values of π_0m and π_1m. Prevalence values contained between the vertical planes are those for which no possible value of γ can result in an adjusted OR_(m=1) (B=0.76).

Approach 2 involves estimation of controlled direct effects, which have an established place in the epidemiologic toolkit.⁸ The authors ¹ use a sensitivity parameter for unmeasured confounding of the intermediate, which is a pragmatic and effective elaboration. This seems to be the superior strategy of the three considered in this paper, as the authors themselves note. Yet it is important to add that a crucial consistency assumption is not met for an effect premised on intervention when no such intervention is realistic.⁹ We presumably have the technology, if not the malice, to fix all births as low birthweight by induction of premature labor. We have no mechanism for controlling pregnancies to normal birthweight, and, more importantly, these effect estimates have no pretense of corresponding to any real-world interventions. The authors therefore caution that the estimates should be granted no causal interpretation. Unfortunately, epidemiologists face this dilemma exactly because we need causal estimates. A more satisfying solution to this problem awaits improved knowledge of the unmeasured common causes of low birthweight and neonatal mortality. And so, like Dorothy at the end of the film, epidemiologists may find that the magic needed to resolve their problem actually lies in the substantive and methodological knowledge in their own backyard.