On Baseball, Counterfactuals, and Measuring Care Delivery Performance at the Emergency Department—Intensive Care Unit Interface

In professional baseball, despite use of videography and analytics to evaluate professional baseball players, it is difficult to measure fielders’ performance accurately. Multiple factors underlie this challenge. First, most batted balls are either surefire “outs” (e.g., routine pop-ups) or surefire hits (e.g., home runs) (1). The remaining opportunities are spread among nine fielders, leaving each fielder few chances to move the needle of performance. Additionally, the dichotomous “out” lacks important counterfactual information. What differentiates “routine” from extraordinary outs or identifies the error of omission when a ball would have been caught, had the fielder been appropriately positioned?

An analogous challenge exists in measuring care delivery performance in and around the intensive care unit (ICU). Among the heterogeneous population of critically ill patients, many have syndromes that they are extremely likely (e.g., uncomplicated diabetic ketoacidosis) or extremely unlikely (e.g., advanced malignancy with multisystem organ failure) to survive. For remaining patients—whose trajectories and outcomes would be most strongly affected by different care delivery approaches—outcomes like mortality are necessary but insufficient to evaluate the performance of the ICU treating them (2). With few randomized trials of care delivery practices, sophisticated observational methodologies are needed to draw inferences regarding the utility of many care delivery interventions.

Together, these factors make it hard to interpret much observational and quality improvement data from the ICU. One approach to this challenge that has become increasingly popular in health services research is the quasiexperimental interrupted time series (ITS) design. ITS controls for temporal trends by comparing outcomes observed after an intervention with the expected outcomes had the intervention not occurred (3). A key building block of the ITS is the concept of a counterfactual: a hypothetical scenario under which an intervention has not occurred. In the baseball analogy above, the counterfactual might be a fly ball that could have been caught had the manager positioned the right fielder differently.

In this issue of AnnalsATS (4), Anesi and colleagues (pp. 1599–1609) use an ITS design to address these challenges in performance measurement as they relate to an important set of clinical and administrative problems: how and where should care be delivered to critically ill patients admitted through the emergency department (ED)? These patients often face care delays and worse outcomes related to strained EDs, ICUs, or both (5–8). The authors investigated an ED-embedded critical care unit (ED-CCU), where some critically ill patients can be managed prior to ICU transfer or quick “downgrade” to ward status.

So far, evidence surrounding ED-CCUs has been sparse but supportive. Previous work found that an ED-CCU was associated with reduced patient mortality and unnecessary ICU admissions at a single academic center (9). To build on this evidence, Anesi and colleagues performed a retrospective pre-/post-cohort study at an urban academic quaternary care hospital to evaluate the relationship between opening an ED-CCU and clinical outcomes (e.g., length of stay [LOS], mortality, and ICU admission decisions) for patients with sepsis or acute respiratory failure. For this study, the counterfactual would have been an otherwise-identical hospital without an ED-CCU, at which critically ill patients continue to be admitted directly from the ED to traditional ICUs. After performing an ITS analysis and additional analyses to account for other important sources of potential bias (e.g., patients presenting on weekends—and their care—may be different than those presenting on weekdays [10]), the authors found that clinical outcomes neither improved nor worsened in association with ED-CCU availability.

In light of these negative findings, this study raises important questions for the future. First, what outcomes must be measured to ensure that a care delivery intervention is actually helpful (or not) (11)? Here, Anesi and colleagues evaluated multiple important endpoints, including minimization of both acute illness duration (total hospital LOS) and critical illness duration (ICU LOS). Appropriately, the authors considered time in the ED equivalent to ICU time; using the ED+ICU LOS as a key secondary outcome provides valuable context as to whether the ED-CCU influences the duration, or just the location, of critical care. If the latter is true, the ED-CCU may be no different than increasing the number of ICU beds.

Tied closely to these outcomes is a second major question: Which patients, if any, are likely to benefit from embedded ED-CCUs? Do the authors’ findings—that ED+ICU LOS was unchanged—suggest that the ED-CCU was not efficacious? Or, were the patients under study the ones most likely to benefit from this intervention? Potential benefits of an ED-CCU depend on the underlying causal mechanism(s) at play. Specifically, the ED-CCU is likely to influence a patient’s outcome if and only if 1) it facilitates care that is somehow better than the alternative and 2) the patient’s illness is neither so severe nor so mild that the outcome is already highly probable. It is unsurprising, then, that this study’s lone suggestion of benefit was for the least-sick patients with sepsis, for whom appropriate disposition and interventions are known to be beneficial (12, 13). Future work might evaluate patients who could avoid the ICU with expedient correction of one clinical issue, such as those with diabetic ketoacidosis.

Third, were potential ED-CCU benefits negated by concurrent harm? For example, many patients would encounter additional clinician and nursing handoffs—well recognized as a source of medical error and potential harm (14, 15)—as a result of “stopping over” en route to their inpatient destination. Additionally, directing a patient to the ED-CCU could itself prompt tests or procedures of relatively low value but nonzero risk (e.g., the “just-in-case” arterial or central line).

Finally, the question of resources must be considered; because establishing and maintaining care delivery innovation like an ED-CCU is likely to be expensive, the intervention must improve patient outcomes, system-level outcomes, or both to have a chance at being cost effective. In light of this study’s finding that an ED-CCU did not demonstrate a clear effect on several patient-oriented outcomes, further work evaluating system-level outcomes is needed.

In the end, we are left with ongoing uncertainty regarding ED-CCUs. Perhaps this uncertainty should not be surprising; just as it takes several seasons to obtain an accurate assessment of a fielder’s defensive performance (16), it may take multiple evaluations of ED-CCUs—with different patients, in different settings, measuring different outcomes—to understand whether these innovations are worth pursuing over the long run.