Invited commentary: it’s not all about residual confounding—a plea for quantitative bias analysis for epidemiologic researchers and educators

Abstract

Epidemiologists spend a great deal of time on confounding in our teaching, in our methods development, and in our assessment of study results. This may give the impression that uncontrolled confounding is the biggest problem observational epidemiology faces, when in fact, other sources of bias such as selection bias, measurement error, missing data, and misalignment of zero time may often (especially if they are all present in a single study) lead to a stronger deviation from the truth. Compared with the amount of time we spend teaching how to address confounding in data analysis, we spend relatively little time teaching methods for simulating confounding (and other sources of bias) to learn their impact and develop plans to mitigate or quantify the bias. Here we review the accompanying paper by Desai et al (Am J Epidemiol. 2024;193(11):1600-1608), which uses simulation methods to quantify the impact of an unmeasured confounder when it is completely missing or when a proxy of the confounder is measured. We discuss how we can use simulations of sources of bias to ensure that we generate better and more valid study estimates, and we discuss the importance of simulating realistic datasets with plausible bias structures to guide data collection.

This article is part of a Special Collection on Pharmacoepidemiology.

This article is linked to “A simulation-based bias analysis to assess the impact of unmeasured confounding when designing nonrandomized database studies” (https://doi.org/10.1093/aje/kwae102).

Editor’s note: The opinions expressed in this article are those of the authors and do not necessarily reflect the views of the  American Journal of Epidemiology.

If an advanced life form existed outside of our current universe and they came to earth with the goal of scouring the published epidemiologic literature to understand what epidemiologists’ biggest problem was, they would quickly discover that the “limitations” section of publications would provide them with all the information they needed. Most likely, what they would conclude is that the biggest problem we face is uncontrolled confounding. It seems to be an obsession of ours.

This is also evident in how we teach epidemiology. We spend much time focused on confounding, likely convincing students that this is the biggest problem we face and the biggest source of bias (ie, deviation from the truth due to systematic error). Perhaps this is because confounding is easier to conceptualize than other forms of bias, or perhaps it is because we have more strategies for preventing or removing it than for any other form of bias and because we teach that one key advantage of randomized trials is reducing the probability of severe confounding. We teach it intuitively (with examples), conceptually (using counterfactuals¹), and structurally (with directed acyclic graphs [DAGs]²). We teach methods for removing or preventing confounding in the design (randomization, matching, etc) and analysis (stratification, standardization, regression, etc) phases of a study. We develop novel methods for dealing with confounding through design (crossover studies, instrumental variables, regression discontinuity, etc), and we teach methods for addressing time-dependent confounding (G-methods, inverse probability weighting, etc). Confounding seems to always be at the center of epidemiologic attention.

Despite the fact that methods for bias analyses for confounding in observational studies (as well as other sources of bias like misclassification)^3‑8 have been available and extensively detailed since the mid-20th century, they haven’t been used nearly as much as some form of the phrase “all observational studies suffer from uncontrolled confounding.” This phrase, as well as other qualitative, narrative assessments of residual confounding (many of which don’t even speculate on the magnitude and direction of the bias), is perhaps the most common phrase found in Discussion sections of published articles within our field. One wonders if those alien life forms, scouring our literature, would ask whether we understood that residual confounding could be quantified, given how infrequently these methods are used.

And yet, it is possible that confounding is not the biggest problem observational epidemiology faces, as other sources of bias, like selection bias, measurement error, missing data, and misalignment of zero time, may lead to even more bias when left unaddressed than confounding, though this would be very difficult to quantify. Additionally, to the extent that residual confounding does continue to have a large impact on epidemiologic research, it isn’t clear how much of this is due to unmeasured but known confounders and how much is due to poor measurement of those confounders, which in some circumstances leaves residual confounding in estimates.³ One wonders to what extent our problems with confounding are actually measurement error issues that have remained unaddressed, as has been expressed under the “all your data are missing” framework.^4,5

These observations lead to two aspects of confounding that we think deserve more attention: (1) simulation of unmeasured or mismeasured confounders to assess their impact on study results and (2) planning for analytical control of missing or mismeasured confounders. These approaches have the advantage that they can also be used for other sources of bias, not just confounding. As such, if we spent more time on them in our teaching, we might both improve our students’ understanding of confounding and give them flexible tools for understanding more sources of bias.

In a new paper published in the Journal, Desai et al⁶ take up these topics head-on. They simulate a dataset meant to mimic actual observational data on the relationships between treatment with celecoxib versus nonselective nonsteroidal antiinflammatory drugs and incident severe gastrointestinal bleeding to assess the impact of an unmeasured confounder. They simulate a confounding structure in relation to their key variables using a DAG as well as one confounder that is unmeasured, but with a measured proxy. They then assess the impact of the missing confounder under differing strengths of confounding and different correlations between the unmeasured confounder and the proxy confounder after adjustment for the measured confounders. This full data simulation approach differs from many bias analysis methods where the dataset already exists and the goal is to simulate the unmeasured confounder within the existing data structure. Because Desai et al simulate the full dataset, their method requires more assumptions about the data-generating mechanism than a traditional bias analysis, which itself provides more opportunities for the simulation parameters to deviate from what might be expected in a true study. As a result, the approach gives insights into possible impacts of bias. Given these challenges, when might a full data simulation approach be most useful?

One situation in which such an approach might be useful would be study planning before the data have been collected or accessed. To design studies that produce valid results, we must engage in the process of making trade-offs and decisions on how to use the limited time and resources we have available to us to achieve the goal. Sometimes this means choosing an approach to measure one key variable that lacks the validity we would like in favor of a more expensive and valid measure of another. In other cases, this means using a dataset that is missing an important confounder or, in a prospective cohort study, forgoing data collection on a key confounder because it would be too expensive. In such cases, is it reasonable and prudent to ask whether a proxy confounder like those simulated by Desai et al⁶ using indications of smoking-cessation counseling as a proxy for current smoking could help us get closer to a valid estimate? This proxy confounder approach can also be conceptualized as a measurement error issue and approached as a bias analysis for information bias.⁷ The results of their study suggest that in some cases it will and that in others, it may still leave reasonable residual confounding and additional bias analysis approaches may be warranted.

Full data simulations also encourage investigators to understand the causal structure of their question. In this case, the authors simulated data by first drawing out a DAG and then adding parameters to the DAG to guide the simulation, something we have been advocating for as a way to learn more about data structures through simulation.^8,9 This approach has the advantage, when teaching, of overtly encouraging students to understand the causal structure that causes confounding. It also reinforces students’ understanding of both DAGs and how confounding works in practice, as well as what actual correlations look like in confounded datasets. Additionally, it can be used to simulate other sources of bias, making it a flexible teaching tool. It also gives students a set of skills with which to continue exploring the impact of biases on their own after leaving the classroom.

The approach in the paper by Desai et al can also provide insights during grant-writing and protocol development. Simulating data prior to data collection (when prospective data collection is necessary) can demonstrate to funders that the investigators have a strong (even if imperfect) understanding of the environment in which they are working and that they can anticipate the sources of error they are likely to face. Such approaches can be complemented with a clear plan for how to address these sources of bias, possibly through additional data collection, validation studies, or a more traditional quantitative bias analysis within the dataset being used. Further, it can alert us to cases where an unmeasured confounder or a mismeasured confounder may lead to minimal bias, allowing the investigators to direct their resources away from data collection that would be expensive but yield limited benefit. It then allows us to target those resources towards important variables or towards generating validation data to support quantitative assessments of other sources of bias. Such an approach would also demonstrate good stewardship of donor resources and a strong attention to study vulnerabilities and plans to mitigate their impact.

In the example by Desai et al, what data would the simulation guide us to collect? First would be validation data on the confounder itself. If the missing confounder leads to a reasonable amount of bias, then validation data in which a sample of study participants have the confounder data collected could vastly improve any resulting bias analysis. Second, given that the data here are simulated, the confounder in the actual data may have a stronger (or weaker) impact on study results than in the simulated data due to differences in the distributions and the assumed impact of key variables. Assessments of these differences could improve future simulations. And third, given that this simulation does not include a correlation between the unmeasured and measured confounders, validation data could be collected to attempt to assess the extent to which this assumption is violated so that future simulations for study planning are improved.

All simulation approaches have limitations. One common challenge is in designing simulations that reliably approximate the data we expect to observe, and we often focus on a single source of bias, as these methods can be used for confounding, selection bias, and measurement error. We should also use caution in interpreting the results of these simulations, since we are almost always likely to simulate a far less complex data structure than we are likely to actually encounter, and as such we may miss important sources of interactions between biases. For example, one critique of the approach taken by Desai et al is that the simulated measured and unmeasured confounders were assumed to be uncorrelated, and no other sources of bias were present. In that simplified setting, the value of including the measured confounders in the simulation and their impact on the assessment of bias from the undermeasured confounder is not clear. This is a fairly common approach to take, as many methods for assessing the impact of uncontrolled confounding make this simplifying assumption.¹⁰ Since adjusting for the measured confounders would remove some of the unmeasured confounding if those factors were correlated, simpler simulations may over- or underestimate the amount of bias the missing confounder creates. To make these simulations as useful as possible, either those correlations need to be incorporated (possibly through a more complex DAG) or advice on the most relevant parameters to use to describe the simulations needs to be provided. That requires both being clear in what the unmeasured confounder is and having a clear understanding of its relationship to other covariates.

Given the clear importance of simulation and quantitative bias analysis in both the planning and analysis phases of epidemiologic research, and given how much time we spend teaching and talking about confounding, why are the methods not used more often? Perhaps it is because we don’t spend enough time teaching these methods. Or perhaps it is because it is not clear whose job it is to demand such approaches¹¹: authors, reviewers, editors, or funders. Whatever the reason, it is time for these methods to make their way to Madison Avenue for a marketing campaign worthy of their value, so that we end up with fewer published papers that suffer from serious bias and we dismiss fewer papers that have obvious sources of error that have limited impact on results. If we take seriously the mandate to maximize the use of limited resources, no doubt these methods will play a significant role in the future.

Funding

N.A. was supported by National Research Service Award T32 HD 104612.

Conflict of interest

The authors declare that they have no conflicts of interest. M.P.F. has published a textbook on quantitative bias analysis methods and receives royalties for that book.

Data availability

No data were used in this article.