Teaching Evidence-based Medicine Skills Can Change Practice in a Community Hospital

Abstract

OBJECTIVES

Several studies have evaluated whether evidence-based medicine (EBM) training courses can improve skills such as literature searching and critical appraisal but to date, few data exist on whether teaching EBM skills and providing evidence-based resources result in change in behavior or clinical outcomes. This study was conducted to evaluate whether a multifaceted EBM intervention consisting of teaching EBM skills and provision of electronic evidence resources changed clinical practice.

DESIGN

Before/after study.

SETTING

The medical inpatient units at a district general hospital.

PARTICIPANTS

Thirty-five attending physicians and 12 medicine residents.

INTERVENTION

A multicomponent EBM intervention was provided including an EBM training course of seven 1-hour sessions, an EBM syllabus and textbook, and provision of evidence-based resources on the hospital network.

MEASUREMENTS AND MAIN RESULTS

The primary outcome of the study was the quality of evidence in support of therapies initiated for the primary diagnoses in 483 consecutive patients admitted during the month before and the month after the intervention. Patients admitted after implementation of the EBM intervention were significantly more likely to receive therapies proven to be beneficial in randomized controlled trials (62% vs 49%; P = .016). Of these trial-proven therapies, those offered after the EBM intervention were significantly more likely to be based on high-quality randomized controlled trials (95% vs 87%; P = .023).

CONCLUSIONS

A multifaceted intervention designed to teach and support EBM significantly improved evidence-based practice patterns in a district general hospital.

As a result of studies demonstrating substantial gaps between research evidence and the care provided in usual clinical practice,¹ there is an increasing emphasis on the teaching of evidence-based medicine (EBM) skills in undergraduate, postgraduate, and continuing medical education programs. These initiatives are based on the untested assumption that teaching health care providers the skills necessary to practice EBM (formulating questions, literature searching, critical appraisal, and application of research evidence) will change clinical performance. Although a number of studies have examined whether training courses can improve specific EBM skills such as literature searching or critical appraisal,² limited data exist on the impact of teaching EBM skills and providing evidence resources on clinical decision making or clinical outcomes. This study was conducted to address this question by evaluating whether the care offered to medical patients at a district general hospital in the United Kingdom was more evidence based after implementation of a multifaceted EBM training program.

METHODS

We performed a before and after study of the quality of evidence in support of therapies initiated for the primary diagnosis of patients admitted to a medical inpatient unit of a district general hospital. It was conducted at Queen's Hospital in Burton-upon-Trent, Staffordshire, United Kingdom (a 465-bed district general hospital without a university affiliation but with a fully integrated information support system as one of the two nationally funded pilot sites for the development of the electronic patient record). There were 35 attending physicians and 3 teams (consisting of 2 junior and 2 senior residents) in the department of medicine, none of whom had received prior training in clinical epidemiology or EBM.

Intervention

The EBM intervention was multifaceted. First, we reviewed all discharge summaries for a 2-week period (July 1998) to identify the most common admitting diagnoses. Therapies were identified for each common medical diagnosis and literature searches were conducted to retrieve evidence supportive of these therapies. For each topic, 1-page summaries of the evidence (critically appraised topics; CATs³) were prepared and entered into a database. Second, we provided all participants with the syllabus Practising Evidence-based Medicine and relevant excerpts from the book Evidence-based Medicine: How to Practise and Teach EBM.^4,5 Third, we conducted an EBM training course over seven 1-hour sessions in October and November 1998 (these sessions occurred during regularly scheduled teaching rounds and involved small-group teaching similar to that provided during the annual Oxford Workshops on How to Teach Evidence-based Medicine). Each session began with a clinical scenario and generation of a clinical question by the learners. During the session, we identified an article relevant to the question and used this to develop and hone critical appraisal skills for a variety of study designs. Clinical topics for discussion included the diagnosis of iron deficiency anemia, prognosis following stroke, therapy for dementia and atrial fibrillation, and association between calcium antagonists and cancer. We also taught efficient strategies to find evidence employing EBM resources including the Cochrane Library, Best Evidence (a compendium of ACP Journal Club and Evidence Based Medicine), and medline, as well as how to develop and access the CATs for 65 common topics which had previously been prepared. Fourth, these EBM resources were installed on the hospital electronic network which the attending physicians and house officers could access from a ward-based computer (prior to this intervention, only the Oxford Textbook of Medicine and a few locally developed guidelines were available on the hospital network). Participants were able to access these resources through two PCs located on the medical ward.

Outcome Assessment

To evaluate our EBM intervention, we obtained all discharge summaries for patients admitted for more than 24 hours to the medicine wards at Queen's Hospital in September 1998 and January 1999 (immediately before and after the program, and 2 months during which the same medical teams were attending on the inpatient units) and 2 of the investigators independently assigned a primary diagnosis and primary intervention to each discharge. The investigators were blind to admission date. Any disagreements were resolved by consensus and, in a few instances, independent assessment by a third investigator. Using methods employed in an earlier study,⁶ we defined the primary diagnosis as “the disease, syndrome, or condition entirely, or if there were several diagnoses, most responsible for the patient's admission to hospital” and the primary intervention as “the treatment or other manoeuvre that represented the most important attempt to cure, alleviate or care for the patient in respect of his primary diagnosis.”

After designation of the primary diagnosis and intervention for each patient, Best Evidence, the Cochrane Library, and medline were searched to find evidence for each intervention. Two clinical epidemiologists independently classified the strength of evidence for each intervention using a previously employed scheme⁶:

Class 1: Those interventions that have been proven to be beneficial in systematic reviews of randomized controlled trials (RCTs) or from individual RCTs.

Class 2: Those interventions with convincing nonexperimental evidence; for example, interventions whose face validity is so great that randomized trials were unanimously judged by the 2 investigators to be both unnecessary, and, if a placebo would have been involved, unethical (e.g., antibiotics for pneumonia, pacemaker for complete heart block).

Class 3: Those interventions without substantial evidence, which includes interventions in common use but meeting neither of the above 2 criteria or interventions shown to be harmful or useless in systematic reviews of RCTs or from individual RCTs.

The primary interventions were defined as “evidence based” if they were class 1 or 2. Finally, the clinical epidemiologists independently categorized the quality of the evidence underlying each class 1 intervention. They used previously described criteria in deciding whether or not an RCT or systematic review of RCTs was of high quality (RCT with blinded assessment of outcomes, intention-to-treat analysis, follow-up of at least 80% or losses to follow-up too few to materially affect the results, and sufficient sample size to detect a clinically important difference with power>80%).⁴ Any disagreements were resolved by consensus.

The data were entered into and analyzed using SPSS (SPSS Inc., Chicago, IL) for Windows version 11.0.

Approval was received from Queen's District Hospital to review anonymous discharge summaries for this project.

RESULTS

During September 1998, 262 patients were admitted to the Queen's District Hospital medical inpatient units; 275 patients were admitted to these same units in January 1999. We excluded 4 patients who were admitted for diagnostic work-up (3 for bronchoscopies, 1 for a lactose tolerance test), 3 patients who discharged themselves against medical advice before therapy was instituted, 41 patients who were admitted for observation (virtually all for chest pain or syncope) and did not receive specific therapy, and 6 patients who received supportive end-of-life care only. The primary diagnoses in the remaining 483 patients were generally similar in both time periods, although there were significantly more admissions for obstructive airways disease exacerbations in January 1999 than in September 1998 (P =.001; Table 1). The age (mean, 63, standard deviation [SD], 9.4 years vs mean, 62, SD, 8.2 years) and gender (44% vs 46% women) distributions were similar in both time periods. The attending physicians cared for a mean of 36.3 (SD, 12.7) patients and each resident cared for a mean of 32.2 (SD, 17.5) patients each month.

Table 1.

Ten Most Common Primary Diagnoses in Patients Admitted Pre/Post EBM Intervention

Diagnosis	September 1998 (N =239) n (%)	January 1999 (N =244) n (%)
Acute coronary syndrome (MI, unstable angina)	44 (18)	45 (18)
Infection (all sites)	29 (12)	37 (15)
Bronchospastic airways disease (asthma, COAD)	28 (12)	56 (23)
Angina, stable	27 (11)	16 (7)
Heart failure	16 (7)	17 (7)
Stroke/transient ischemic attack	10 (4)	10 (4)
Peptic ulcer disease, esophagitis	10 (4)	8 (3)
Atrial fibrillation	9 (4)	8 (3)
Overdose	8 (3)	5 (2)
Chronic liver disease	7 (3)	3 (1)

No other diagnoses occurred in more than 1% of patients in either year. EBM, evidence-based medicine; MI, myocardial infarction; COAD, Chronic obstructive airways disease.

Patients admitted after implementation of the EBM intervention were significantly more likely to receive evidence-based therapy than those treated before the intervention (82% vs 74%; P =.046). In particular, patients admitted after the intervention were significantly more likely to receive therapies proven to be beneficial in RCTs (62% vs 49%; P =.016; Table 2). Furthermore, even among the subset of patients receiving therapies shown to be beneficial in RCTs, the therapies offered after the EBM intervention were significantly more likely to be based on high-quality evidence (95% vs 87% based on high-quality RCTs; P =.023; Table 2). Sensitivity analyses demonstrated that patients admitted with coronary disease (71 preintervention, 61 postintervention) were significantly more likely to receive evidence-based therapy after the intervention (98% vs 87%; P =.02), while the quality of evidence for the selected therapy was not significantly different for patients with obstructive airways disease pre/postintervention (91% vs 86% evidence-based; P =.47).

Table 2.

Evidentiary Basis for Prescribed Therapies Pre/Post EBM Intervention

	Preintervention (N =239) n (%)	Postintervention (N =244) n (%)	P Value
Strength of evidence
Therapy supported by evidence from randomized trials or systematic reviews of randomized trials	118 (49)	152 (62)	.016
Therapy supported by convincing nonexperimental evidence	60 (25)	48 (20)
Therapy without substantial evidence, or therapy not supported by evidence from randomized trials or systematic reviews of randomized trials	61 (26)	44 (18)
Quality of evidence for those therapies supported by randomized trials
High-quality randomized trial or systematic review of randomized trials	103 (87)	145 (95)	.023
Poor-quality randomized trial	15 (13)	7 (5)

EBM, evidence-based medicine.

Agreement between the investigators was good regarding the primary diagnosis (κ= 0.92) and management (κ=0.76).

DISCUSSION

We have shown that a multifaceted intervention designed to teach EBM skills and implement evidence resources significantly improved practice patterns in a district general hospital. After the EBM intervention, more patients were prescribed therapies proven to be efficacious in randomized trials, and the trials supporting these therapies were significantly more likely to be high quality than before the EBM intervention. The observed absolute improvement of 13% exceeds the 10% absolute improvement which has long been accepted as the minimal clinically important difference for studies of educational interventions.⁷

The degree to which practice on the medical inpatient units at this district general hospital was evidence based after the EBM intervention is very similar to that reported for medical inpatient units at university-affiliated tertiary care hospitals with attending physicians holding postgraduate degrees in clinical epidemiology. For example, using the same definition we did, 82% of primary interventions were deemed to be evidence based at the John Radcliffe Hospital in Oxford, United Kingdom and 84% at the Ottawa General Hospital in Ottawa, Canada.^6,8 This includes 53% (and 57%, respectively) of primary interventions which were deemed to be class 1 (i.e., supported by RCTs). Thus, we have shown that the attainment of evidence-based practice is indeed possible in busy clinical settings after implementation of appropriate resources and teaching of EBM skills.

The results of two recent randomized trials suggest there may be benefit to EBM training although differences in the formulation of the intervention make it difficult to compare with the current study.^9,10 In the first study, information management was compared with training in EBM for secondary prevention of cardiac disease in primary care.⁹ The combination of these interventions showed some improvement in management of cholesterol. However, it appears that the study intervention was focused on searching and retrieving evidence from the Internet and medline around this particular condition and did not include education on formulating questions, applying the evidence, or assessing our performance. Moreover, the intensity of the intervention is unclear. The second study evaluated the impact of an EBM educational intervention among public health workers but did not report impact on actual behaviors.¹⁰

However, our study is a before/after case series and does not carry the same weight as a randomized trial. Ideally, to complete a methodologically rigorous study of EBM, we would aim to expose “control” clinicians to an evidence-poor teaching intervention and allow them to become out of date and unaware of potentially life-saving evidence accessible to and known by the evidence-based clinicians in the experimental group. However, this approach is not ethical and alternative designs must be explored. We were unable to identify an appropriate control site with the same patient, house staff, and attending physician mix and which had the same informatics infrastructure. This hospital had a fully integrated information support system as one of two nationally funded pilot sites. And, we were unable to do an interrupted time series given the time constraints due to house staff rotation.

Our choice of study design limits the inferences that should be drawn from this study. Thus, while our study suggests that EBM training does meaningfully impact on clinical decision making, randomized trials of EBM teaching are clearly needed and one of us (SES) has embarked on just such a study. This randomized trial of family physicians has been designed to determine whether an online EBM educational intervention can change behavior and clinical outcomes. Our study may also be criticized for reporting on process measures (therapy prescribed) rather than clinical outcomes such as mortality. However, we chose to focus on process measures as they are more sensitive indicators of quality of care than changes in clinical outcomes which take months or years to manifest.¹¹ Finally, we do not have any data on the frequency with which the various evidence resources were accessed by the clinicians at Queen's Hospital. However, other investigators have shown that if you provide evidence resources in a convenient and readily accessible format, clinicians with training in EBM will use them.^12,13

In summary, we have demonstrated that the practice of clinicians in a district general hospital changed in a statistically significant and clinically meaningful way after completion of an EBM training course and provision of evidence resources. The implications of our study are further amplified by evidence that clinicians trained in EBM are more likely to remain up to date for longer after their training than clinicians without EBM training.¹⁴ Given that surveys of frontline clinicians confirm widespread enthusiasm for EBM and a desire to learn the key skills such as evidence retrieval and critical appraisal,^15–19 we believe that training in the practice of EBM should remain a key component of undergraduate and postgraduate education. Proponents of knowledge translation have advocated that changing behavior requires comprehensive approaches directed toward patients, physicians, managers, and policy makers.²⁰ The results of this study suggest that a multifaceted approach to teaching EBM can change behavior.