Statistical Note: Confounding and Causality in Observational Studies

No teacher in medicine is better than experience. Observational studies that examine large repositories of data, such as electronic health record (EHR) databases and multicenter registries, afford a vantage of collective experience beyond what any individual can gain at the bedside. Recent years have seen a surge of observational literature propelled by increasing accessibility to EHR databases and a growing familiarity and availability of observational research tools such as the open-source R programming language and software libraries that facilitate advanced modeling.(1) Observational insights through pattern identification and statistical tests of association share both a bedside and mathematical space with causation. However, association and causation are not the same. Awareness of the potential role of confounding is essential when working to distill meaningful insights from observational data and studies, a task that can seem particularly daunting with the bloom of increasingly esoteric “machine learning” analyses throughout the medical literature, including Pediatric Critical Care Medicine.

In this issue of the Journal, a study by Beshish and colleagues describes an association between hyperoxemia during cardiopulmonary bypass and operative mortality among children with heart disease.(2) In the accompanying editorial, Dr. Peters highlights the natural temptation that we all have to infer causal relationships from patterns proximate to one another in time and space.(3) Indeed, several studies have examined the relationship between hyperoxemia and outcome among critically ill children and identified associations between supra-physiologic oxygen levels and mortality.(2, 4-12) It is a research question with a biologically plausible basis in that reactive oxygen species are recognized to wreak molecular havoc in cells and tissue. It is also a question readily investigated in data-rich intensive care unit registries and databases, in which multiple observations of F_iO₂, P_aO₂, S_pO₂ (fractional inspired oxygen, arterial partial pressure of oxygen, pulse oximetry oxyhemoglobin saturation, respectively) abound and can be analyzed in relation to outcome. But the fact remains that sicker patients are in general exposed to greater amounts of oxygen, either out of necessity or as part of resuscitation bundles. Hence, oxygen has been deemed a highly suspicious culprit, but just because oxygen is dressed in stripes does not make it a convict! So how does one derive meaningful information from observational studies while disentangling associations from confounding to form actionable insights?

One can start by considering the basic definition of confounding, which is “a blurring or mixing of effects.”(13, 14) A confounder has an association with both the disease, and/or predictor, and/or exposure, and the outcome of interest – but is not an effect of the exposure or a factor in the outcome or property of the underlying disease. For example, a large forest fire causes substantially more damage than a small bush fire and requires far more fire fighters to combat the blaze. Without knowledge of the causal mechanisms at play, one might identify that a greater number of fire fighters is associated with more catastrophe and could erroneously infer a causal relationship along the lines of more fire fighters results in more damage.

Since causal relationships are the relevant targets for treatments and interventions, we need strategies to control for confounding. Let us consider these using the example of pediatric hyperoxemia studies. As already mentioned, illness severity is an important confounder in many studies examining the relationship between hyperoxemia and outcome. A common technique to control for illness severity is multivariable modeling, often incorporating some version of a validated measure of disease severity, such as the pediatric risk of mortality score (PRISM), pediatric index of mortality (PIM) or pediatric logistic organ dysfunction score (PELOD).(2, 4, 6, 8-11) Each of these scoring systems has a track record that includes external validation and robust predictive performance. However, the development of each of these scores has had to account for the competing aims of being sufficiently comprehensive representations of illness severity while also being calculable using checklists and commonly available data. Further, the relatively low rate of mortality in pediatric critical care means that these models are trained on imbalanced datasets, with more to teach about life than death. Accordingly, these models only account for a subset, and certainly not all, of data representing illness severity. Modeling last rites as a variable, for example, may be associated with mortality after adjusting for severity with an established score, but does not indicate a causal relationship between last rites and death. Instead, identifying an association is only the first step, and further steps are necessary to tease out potential causality. Fortunately, readers can ask some relatively straightforward questions of their own before drawing conclusions from observational data.

Bradford Hill outlined a set of classic criteria to assess causality in observed relationships: strength, consistency, specificity, temporality, biological gradient (dose-response), plausibility, and coherence.(14) Strength refers to the observed effect size of an association. In the context of hyperoxemia research, this refers to how much more likely death is for patients exposed to hyperoxemia compared to those with apparent normoxemia, commonly expressed as odds. Greater odds of death provides some indication that the relationship is more likely to be causal, though there is no established threshold for declaring a causal relationship. Consistency refers to the reproducibility of the finding. As the number of observational studies citing an association between hyperoxemia and mortality accumulate from a range of institutions and patient subpopulations, the finding appears increasingly consistent. Specificity refers to an association consistently observed in a select context, such as hyperoxemia being associated with death post-cardiac arrest. The temporality criterion necessitates the timing of the exposure to be sensibly related to outcome. For example, hyperoxemia defined by P_aO₂ obtained during the episode of critical illness, rather than during apnea testing when assessing for brain death. The biological gradient criterion would require the magnitude of hyperoxemia, measured either by cumulative exposure, magnitude of an instance of exposure, or both, to have a proportionate effect on the odds of death. Several studies have demonstrated a plausible “U-shaped” relationship between oxygen level and outcome, with death more likely at both the extreme highs and lows of P_aO₂ values, though it is less clear if this is a relationship defined by polynomial smoothing rather than true dose-response.(6, 8, 11) Plausibility asks whether a biologically sound mechanism underlies the observed association. As noted already, substantial pre-clinical evidence, some findings dating back more than a century, contributes to the enthusiastic belief that hyperoxemia is causally linked to death.(15) Coherence would expect that the outcome of interest, mortality in this case, would be altered by removing exposure to hyperoxemia. Guidelines for the management of pediatric cardiac arrest recommend judicious use of oxygen in resuscitation, for example, though it is difficult to discern the impact of this single intervention amidst many other evolving aspects of care.(16)

With these criteria in mind, some additional steps can be taken in the design of observational studies to further interrogate potentially causal relationships. Well-constructed propensity scores, for example, can allow researchers to incorporate a greater number of potentially confounding variables in observational models without problematic overfitting. Following identification of a significant association, assessment for lurking confounders can be performed, which quantifies the prevalence and effect size of an unmeasured confounder that would be necessary to tip the model to fail to reject the null hypothesis of the observed association. Observational randomized analyses can also be performed to deal with uncontrollable confounders. This is commonly performed by dividing large datasets into a training set for model development and a testing set for model assessment. When smaller datasets preclude division into train and test groups, data splitting can be performed using k-fold cross validation. Intrinsically random, machine-learning approaches such as random trees, forests and fields and neural networks may be coupled with feature importance analyses to assess whether hyperoxemia is predictive of mortality in large, granular databases.(17) Such techniques provide added capabilities for dealing with unmeasured confounders in observational data, but such experiments are still subject to the limitations of the dataset on which they are being applied.

Ultimately, the gold standard to deal with confounding in clinical experimentation remains the randomized controlled trial. By randomizing comparable groups of patients to alternative treatment strategies, the intervention of interest ideally becomes the distinguishing characteristic of the experimental arms. There are, of course, myriad practical challenges to randomized controlled trials in pediatric critical care, including the relative rarity of diseases, cost, special ethical considerations, and challenges obtaining consent at the appropriate time, among others.(18) However, the same circumstances that have promoted the surge in observational studies in recent years, including the increasing usability of EHR data, may also pave the way for larger, most efficient trials in our field. As an increasing number of Bradford Hill criteria are met in observational studies assessing a link between hyperoxemia and outcome among critically ill children, the findings should not be viewed as irrefutable causal associations, but rather as supporting the need to invest in more conclusive studies of this important topic.