Commentary: Gilding the black box

Most epidemiology textbooks have the obligatory passage on ‘what is a cause?’ These discussions often start with Hume, pass reverently through Bradford-Hill and (if the book is of relatively recent vintage) end with Pearl. But as Hafeman and Schwartz¹ point out in their essay, few texts in our field go on to the question that really motivates these authors, which is ‘what is a causal structure?’ The closest thing I can find on my own bookcase might be Mervyn Susser's² ‘Causal Thinking in the Health Sciences’, now more than 35 years old and long out of print.

Despite the generally simplistic approach taken by our textbooks, the big breakthrough stories in biomedical research often sound like a sportscaster narrating the progress of a pinball machine game: ‘First the ball hits a lever, than bounces into a hole, where it triggers a sensor that opens a chute, and the ball slides down to the flipper …’. The answer to a causal question such as ‘how did I just lose that ball down the side chute?’ can only be answered by referring to (i) multiple events that happened in sequence, (ii) the imagining of alternate ways that the sequence could have occurred instead and (iii) by having the whole process situated in a context with regular and comprehensible laws (such as the physical layout of the pinball machine, or the physiology and environment of a living organism). We naturally tend to conceive of ‘explanations’ as Rube Goldberg devices, where some exposure leads to a predictable cascade of events, and finally to the ‘outcome’—a cancer, or a death, or whatever. And we can easily imagine holding some intermediate gear still, which interrupts the natural flow of this device, and in this way we find that the outcome depended on that step. This is an analysis of mediation.

Hafeman and Schwartz make the case that causal structure is ultimately what we really want to know in epidemiology, and I could not agree more. Surely, every one of us would love to stick our fingers into one or another part of the Rube Goldberg machine that generates our favourite health outcome to see what would happen. Sadly, epidemiology is probably the biomedical discipline most thoroughly removed from this sort of satisfying and enlightening exercise. Experimentalists in their laboratories get to do this with everything from Petri dishes to people, manipulating them in endless variations. We epidemiologists, on the other hand, are usually stuck with nothing but observational data. That is, our data are usually gathered by measuring the various configurations of the Rube Goldberg machine as it chugs along, but without being able to intervene in that process in any way. We get to manipulate the resulting data all we want, but that is a whole other ballgame. When we want to know what would have happened under some circumstance that was not observed, we need to use an observed substitute to get that value.³ For example, if we need to know the result for a group exposed to an event at some point in the process, had they instead been unexposed at that point, we might use an appropriately time- and covariate-matched unexposed group. Such substitutions are potentially dangerous because they may be unbalanced across important variables that we do not yet know are part of the causal structure. This is what is known as confounding.

While I have no disagreement with Hafeman and Schwartz's stated goal of trying to motivate interest in investigating causal structures, I am not as convinced by their decision to focus exclusively on natural/pure direct and indirect effects, rather than the alternate formulation based on hypothetical actions and known as controlled direct effects.⁴ There are several practical and conceptual advantages to this latter species of effects,⁵ and it seems to me both curious and unfortunate that Hafeman and Schwartz should give this interventionist formulation such quick dismissal. First of all, epidemiology is situated within the broader discipline of public health. This may be partly historical accident, but I think it does reflect a real interest in our field on interventions for health improvement. The point of trying to figure out that, say, smoking causes lung cancer is not to merely note this with the passive indifference with which one might observe some distant astronomical event. Rather, I think that most epidemiologists would study such a problem because they would want to know what needs to be done at the population level to lower the incidence of lung cancer. Total causal effect estimates provide for us the anticipated consequences of a manipulation of the exposure variable. But if we want to estimate the effect on the outcome of manipulating the exposure while at the same time fixing some consequence of the exposure to a specific value, this is a controlled direct effect, not a natural/pure direct effect. In the context of an exposure intervention, it is quite straightforward to conceive of a plan that also involves blocking intermediate M from occurring (i.e. SET[M = 0]) or forcing M to always occur (i.e. SET[M = 1]). It is much more problematic to figure out how one would enact a plan in which the value of intermediate M would be forced to take on the value that it would have had under some (potentially unobserved) reference level for exposure X (i.e. SET[M = M_(X=0)]).

A scenario that is often postulated in order to illustrate the natural/pure effect estimator is that there could exist an intervention that would disable the path between X and M, rather than an intervention to set M as in the controlled scenario. For example, Robins and Greenland⁶ considered cigarette smoking as the exposure, hypercholesterolemia as the intermediate and cardiovascular disease as the outcome. If one had a drug that could perfectly eliminate or induce hypercholesterolemia, one could postulate the controlled direct effect of smoking on cardiovascular disease, while hypercholesterolemia is forced by these medications to be absent or present. In order to envision the natural/pure direct effect, on the other hand, one would need a drug that merely prevents smoking from having any effect on hypercholesterolemia at all, so that people would have which ever value of the intermediate they would naturally have had in the absence of exposure.

A fundamental difficulty with this model for natural/pure direct effects, however, is outlined by Robins.⁷ Any directed path on a directed acyclic graph (DAG) can be thought of as a sequence of infinitesimally small steps. For the controlled direct effect in the DAG with exposure X and intermediate M, these myriad tiny steps between X and M are irrelevant as long as their error terms are not correlated with the outcome. For meaningful interpretation of estimated natural/pure direct effects, however, Robins argued that the deactivation of the pathway between X and M would have to occur at the very first infinitesimally small step in the maximally elaborated path between these nodes. Interruption of the path at this very first infinitesimal step would allow for the estimation of the outcome distribution that would have been observed if the intermediate M were to take the value it would have had under X = 0. Disabling the path at any subsequent step, however, would give a potentially different outcome distribution, since all of the other tiny steps prior to that point would retain the values that they exhibit under the attained exposure level, rather than under the reference exposure level. Since it is difficult to conceive of this maximally elaborated path, and even more difficult to think that a drug or other treatment would disable this path only at the very first mechanistic step, Robins expressed skepticism that the natural/pure definition of component effects could have any practical application in relation to potential public health interventions.

Overall, Hafeman and Schwartz have done a great service to the field in explaining these effect definitions in accessible and well-organized language. I would only caution that in focusing exclusively on the more abstract and impractical of these effect decomposition formulations, they risk merely ornamenting the opaque surface of the black box rather than really chipping away at its obfuscating exterior. Indeed, this is exactly what ‘black boxing’ has come to mean in the academic field of ‘science studies’: the way that improved technologies (such as these effect estimators) can actually make science more opaque by automating our thinking in formulaic ways.⁸ It would be ironic if these new statistical tools ended up taking us further from the ideals expressed in Susser's book,² which place a premium on the integration of detailed subject matter knowledge and logical reasoning with thoughtful but transparent statistical analysis. I am far from being any kind of statistical Luddite, but I would prefer to see greater use of those causal effect estimates, like controlled direct effects, which not only require fewer assumptions in order to be identified from observational data,^5,7 but also correspond to realistic and well-defined intervention plans. Using controlled effect measures, therefore, we will know not only something more about the Rube Goldberg machine that generates our favorite health outcome, but also about the expected result of sticking our fingers between the gears in some specific place. This is not only an analysis of mediation, but also one that has a more direct connection to decisions about policy.

Conflict of interest: None declared.