Blinding Them With Science? Evidence-Based Medicine as a Barrier to Health Care Value

The term “evidence-based medicine” (EBM) was introduced in 1992 in a seminal paper by Gordon Guyatt as a solution to “an exploding volume of literature . . . deepening concern about burgeoning medical costs, and increasing attention to quality and outcomes.”¹ Over the ensuing decades, EBM has been integrated into the medical culture and incorporated almost universally into medical school and residency curricula.^2,3 In addition, Guyatt's recognition of the need to reduce health care costs and improve quality has entered the mainstream consciousness, framed increasingly around the notion of “value.”

Value can be conceptualized as the ratio of health outcomes and costs.⁴ Skills in EBM are critical to optimizing value, since a deep understanding of evidence is required for predicting health outcomes in individual patients. In particular, clinicians must recognize the clinical impact of interventions, grapple with uncertainty in the evidence, and uncover bias in published studies in order to fully balance the benefits and harms of potential approaches. More than 20 years of EBM immersion should have thoroughly prepared us for these tasks—but has it?

The study by Caverly et al⁵ in this issue of the Journal of Graduate Medical Education suggests that EBM education has failed to prepare physicians for high-value practice. The authors presented medical residents and attending internal medicine physicians with 4 vignettes that described drug studies with different types of endpoints: total mortality, disease-specific mortality, a surrogate outcome (simply called a “risk factor” in the vignette), and a composite outcome with a surrogate component. Participants were asked to rate the extent to which each study proved that the new drug “might help people.” Improvement in the composite outcome, as proof of drug benefit, was rated most highly by both residents and attending physicians. While participants were not asked to directly compare endpoints, fewer than half rated all-cause mortality as better proof of benefit than improvement in a surrogate endpoint, and fewer than a quarter of participants rated all-cause mortality as better proof than a composite endpoint. Despite limitations in this study approach, the findings suggest that physicians lack the skill to accurately weigh the relative importance of different types of endpoints in clinical trials, and they tend to overvalue surrogate and composite endpoints.

The overvaluing of surrogate and composite endpoints threatens health care value, because improvements in surrogate endpoints may occur without improvement (or with worsening) of clinical outcomes. For example, class 1C antiarrhythmic agents were routinely prescribed to post-myocardial infarction patients with asymptomatic ventricular arrhythmias after myocardial infarction for arrhythmia suppression, until the Cardiac Arrhythmia Suppression Trial found that these drugs actually increased mortality compared to a placebo.⁶ Use of dual angiotensin-converting enzyme inhibitor and angiotensin receptor blocker therapy for a variety of indications grew rapidly based on the possible benefit in surrogate outcomes (eg, proteinuria in nephropathy) until complications such as hypotension and hyperkalemia were clarified.⁷ In both of these cases, prescribing based on surrogate outcomes likely harmed large numbers of patients. Further, since pharmaceutical industry marketing is often based on surrogate outcomes,⁸ a failure of physicians to recognize the limitations of surrogate outcomes may facilitate successful industry marketing of new expensive (and possibly minimally effective) drugs, resulting in reduced value for patients.⁹

Why, despite EBM education, are physicians unable to appreciate the greater value of a reduction in mortality compared to an improvement in a surrogate outcome? First, evaluating the appropriateness of endpoints is not adequately emphasized in EBM education. Despite the ubiquitous “PICO” structure for clinical questions, with “O” representing the outcome of interest, there is little instruction in the relative weight of different outcomes, and the complexity of composite outcomes defies simple explanation. Instruction in the applicability of evidence to patient care includes consideration of whether all clinically relevant outcomes were reported.¹⁰ However, applicability issues tend to be deemphasized in EBM teaching in favor of teaching about internal validity; thus, discussions of outcomes may be cursory. Reflecting this lack of emphasis, standard tools for evaluating physicians' EBM skills do not test understanding of the relative value of outcome measures.^11,12 Finally, evidence hierarchy¹³ is routinely taught as a central EBM concept. This hierarchy emphasizes the importance of study design, where randomized trials are highly valued without consideration of specific study characteristics (such as the chosen primary outcome), so a relatively inexperienced EBM practitioner would likely consider a randomized trial with a surrogate primary outcome to be high-level evidence.

Clearly, inclusion of specific study characteristics in the evidence hierarchy would render it overly complex and unusable, but perhaps that's the point. Understanding evidence is legitimately complex, and attempts to oversimplify the process may perversely lead to misinterpretation of evidence and the incorporation of low-level evidence into clinical practice. The findings of Caverly and colleagues⁵ may represent the tip of the iceberg of evidence misinterpretation. While few studies have assessed physician skills in identifying appropriate evidence for clinical adoption, physicians have poor numeracy,¹⁴ fail to discount for conflicts of interest when weighing evidence,¹⁵ and appear to be influenced by industry marketing⁹ that tends to present evidence poorly.

How can educators better train physicians to use evidence to improve value for patients? First, we can emphasize basic concepts in EBM education, rather than the details of critical appraisal or instruction in calculating quantifiers, such as the number needed to treat and the likelihood ratio. This teaching should include the importance of clinically relevant outcome measures, appropriate comparators, and adequate follow-up time in clinical trials as well as the possible influence of conflicts of interest. These basic concepts need to be reinforced repeatedly throughout training. Second, after more than 2 decades of EBM education, we need to recognize that evidence interpretation is complex, and that many (perhaps most) physicians may never master it. For these learners, the ability to identify and retrieve reliable high-quality evidence is critical,¹⁶ but the ability to perform critical appraisal is less important. All trainees must become skilled at accessing high-quality evidence-based guidelines (from a variety of national organizations) and topic summaries (from sources such as BMJ Clinical Evidence¹⁷). Trainees should also know how to access summaries and interpretations of individual high-impact clinical trials (from sources such as ACP Journal Club¹⁸ and McMaster PLUS¹⁹) and high-quality systematic reviews (from sources such as Cochrane²⁰). Accomplishing this mastery may take time away from traditional EBM education, and will also require humility and the acknowledgment of the complexity of EBM. At the same time, it will result in a physician workforce with the ability to use the best evidence and make high-value clinical decisions for patients.

The study by Caverly et al⁵ shows us that current EBM education may not provide physicians with the skills required to make the best decisions for patients. Refocusing on EBM basics will remove the blinders, help physicians recognize good and bad evidence, and improve the value of care provided to all patients.