Challenges of Comparative Effectiveness Research with Observational Data

Every day, patients and their doctors confront decisions without the evidence they need to inform the choice. Guidelines are replete with recommendations that rely on expert opinion because needed studies are absent (1). Moreover, even when high-grade clinical trial evidence exists, there are concerns that the reported efficacy in idealized settings is not reflective of what can be achieved in actual practice. As a result, there is growing interest in developing evidence that flows from the experience of clinical practice. In fact, the concept of the learning health care system derives from the notion that data produced in the course of clinical care can be transformed into practical evidence about what comparative strategies can achieve under typical clinical conditions (2).

There are two main strategies that can be applied in typical clinical settings to provide evidence regarding effectiveness. One strategy, the pragmatic clinical trial, employs an experimental design while focusing routine practice settings (3). The interest in these trials is growing, but the pace of knowledge production is slow and the expense is relatively high, with a few exceptions. The other strategy is to leverage observational data and variation in practice patterns to produce meaningful comparisons of alternative strategies. This approach can produce new evidence faster and more efficiently, but questions about the validity of the findings have limited their impact. An important issue is whether comparative effectiveness research employing observational data can be strong enough to influence practice.

This issue of the Annals contains an article by Hansen and colleagues that provides a good case study for the predicament of comparative effectiveness research employing observational data (4). These Danish investigators sought to determine whether an early invasive strategy, defined as coronary angiography within 72 hours, was superior to a more conservative therapy, defined as delayed angiography or no angiography, for patients hospitalized with an acute coronary syndrome. They leveraged national administrative databases and the adoption, in 2009, of compulsory protocols requiring diagnostic coronary angiography for these patients within 72 hours of admission and employed advanced methods for addressing selection bias. They reported that patients undergoing an early invasive strategy achieved remarkably better outcomes compared with the conservative strategy.

The study seeks to draw a causal inference between the use of an invasive strategy and the better outcomes. However, the design severely limits the capacity of this study to support such an inference. At best, the study provides no evidence to impugn the belief that the early invasive therapy is better.

There are many threats to the validity of the causal inference in this study. First, as noted by the authors, they lack clinical data on the patients, including electrocardiographic findings, cardiac troponin measurements, Global Registry of Acute Coronary Events (GRACE) score, type of myocardial infarction, and left ventricular ejection fraction. Most importantly, they lack information about the indication for the procedure, which is particularly important since prior work has compared routine invasive strategies (applied to everyone) versus selective invasive strategies (reserved for people who develop symptoms despite optimal medical therapy) (5). The absence of this key clinical data makes it impossible to evaluate the success of the propensity matching for key measurable, but missing, clinical data. In addition, judgments about referral for procedures may take into account frailty, dementia, and other functional disabilities that are poorly documented in the chart. There is also an issue of the inclusion of patients with unstable angina and the unreliability of the codes along with the possibility that the procedure and its outcome could influence the coding. Finally, there is a key temporal issue, as the invasive strategy was compulsory in the later years of the study, which is particularly important for a condition that in most countries is trending toward improved outcomes over time.

Another key issue is that the validity of the comparison requires that both groups be considered candidates for either strategy (6). This study considered all patients who did not undergo angiography to be in the comparison group, but there is no evidence that they all were candidates for an early invasive therapy.

The predicament is that this study - comparative effectiveness research with observational data in a leading journal, using national data, and applying strong analytic methods - cannot strongly support causal inference. The quality of the data remains a critical issue as do the questions about whether the methods can overcome concerns about residual confounding. Almost every comparative effectiveness article using observational data, however well done, must be circumspect in asserting casual inference.

Moreover, even if the results are valid, and the current study cannot demonstrate its validity, the perceived weaknesses undermine its ability to move individuals from their prior beliefs about the comparative effectiveness of the strategies. Many observational studies agree with trial results, but some do not (7). According to guideline standards, the observational data, at best, provide level of evidence ‘B’. Moreover, that assignment of strength of evidence does not even consider the quality of the data used.

A reasonable question is whether we should conduct these studies if their ability to influence guidelines and practice is so limited. Moreover, can we develop methods that can provide us greater confidence in the ability of studies using observational data to guide practice? Given that the comparative effectiveness of complex interventions may be influenced by not only by the relative efficacy of the strategies, but also by the multiplicity of factors involved in their application in practice, it is also critical that this research isolate implementation factors contributing to the results and how they might affect the translation of the results in other settings.

There remains much work to do before comparative effectiveness studies using observational data become meaningful for influencing clinical practice. We need to improve the quality of data, strengthen analytic methods with attention to assessing comparative effects and modifying factors, and reach consensus on validation approaches (8, 9). Meanwhile, these studies will remain interesting, but fall short of being strongly influential in changing minds about the comparative performance of each strategy.