Commentary: Quantifying the unknown unknowns

Observational studies of the effects of exposures or medical treatments usually suffer from confounding. Whereas measured confounding variables can be adjusted for, it is often impossible to correct for unmeasured confounding. Unfortunately, the potential impact of unmeasured confounding, and whether an observed relation (or part of it) may be due to unmeasured confounding, is not often discussed.^1–3

To guide the thinking about unmeasured confounding, a useful classification is in known, yet unmeasured, confounding variables (the ‘known unknowns’), and unknown, and therefore unmeasured, confounding variables (the ‘unknown unknowns’). Given the former are known, their relation with exposure and outcome may be known too, and their potential confounding effect can often be quantified through bias analysis.^4–6

A starting point for the discussion about unknown unknowns could be to ask what an unmeasured confounding variable should look like in order to explain the observed effect, if in fact there is no exposure–outcome relation.^6,7 This can be quantified by the E-value, which is the minimum strength of association, on a risk ratio scale, an unmeasured confounding variable would need to have with both exposure and outcome in order to explain the observed effect.⁸ Although the E-value was proposed only recently, Blum et al. have already performed a systematic review of its use and interpretation, published in this issue.⁹

They found that studies that report similar E-values draw different conclusions about the potential for unmeasured confounding. Given that the E-value is merely a function of the effect estimate (e.g. estimated risk ratio), this basically shows that, despite similar effect estimates, the perceived potential for unmeasured confounding differs between studies. A simple reason is that one field of research may be more prone to unmeasured confounding than another.¹⁰ Furthermore, the study design, rather than effect estimate, is informative about the potential for unmeasured confounding. Consider three almost identical studies of the same exposure–outcome relation. All three studies find that exposure increases the risk of the outcome by, say, 33% (relative risk = 1.33). For each study, the E-value is 2. But once you know that one study is a large randomized trial, one is an observational study with extensive adjustment for confounding variables, and one is an observational study with adjustment for age and sex only, the potential for unmeasured confounding clearly differs between the different studies, despite the E-value being the same.

Critics of the E-value might argue that it does not quantify other sources of bias. Indeed, E-values are intended to quantify the discussion about unmeasured confounding, but not about, e.g. measurement error or missing values. But since it was never the intention of E-values to do so, the E-value is not to be blamed. Also, E-values assume that the strength of the relations of the confounding variable with exposure and with the outcome are the same. Those who consider this unrealistic, can apply slightly more complicated formulae,¹¹ of which the E-value is just a particular case. Furthermore, the E-value considers the perhaps unrealistic situation that the exposure–outcome relation is null. It is, however, possible to calculate E-values for non-null hypotheses.⁸ Finally, although E-values are often presented for a single unmeasured confounding variable, they can be thought of as a summary of multiple unmeasured variables.¹¹ Note that all these concerns are about the use of the E-value, not about the E-value itself.

A central issue in the debate about the E-value seems to be whether this tool is a useful first step when considering unmeasured confounding or whether it will do more harm than good by being too simplistic.^12–14 The latter concern is justified should the E-value prevent researchers from applying more advanced bias analysis methods. But current practice is that these methods are hardly ever used anyway.^1–3 Blum et al. argue that ‘facile automation in calculating E-values may compound the already poor handling of confounding’.⁹ However, that fear is not supported by the numbers found in their review. In 69 papers in which an E-value was reported, 18 made no comment about the potential impact of unmeasured confounding, whereas no such comment was made in 52 of the 69 matched control papers. These numbers suggest that, among researchers who reported an E-value, there appears to be more (not less!) attention to unmeasured confounding, which in any case is a starting point for further discussions about that topic.

Despite at least 60 years of literature about bias analysis for unmeasured confounding,¹⁵ little progress has been made in practice: usually unmeasured confounding is addressed only vaguely, and often not at all.^1–3 We should move beyond unsubstantiated statements about unmeasured confounding being present or not. E-Values are not meant to reduce discussions about unmeasured confounding to the mindless reporting of a single value. Instead, they could provide a starting point for a substantive debate about what unknown unknowns might look like.

Funding

This work was supported by grants from the Netherlands Organization for Scientific Research (ZonMW-Vidi project 917.16.430) and the Leiden University Medical Centre.

Conflict of interest

None declared.