Impact of an Evidence-Based Medicine Curriculum Based on Adult Learning Theory

Abstract

OBJECTIVE

To develop and implement an evidence-based medicine (EBM) curriculum and determine its effectiveness in improving residents' EBM behaviors and skills.

DESIGN

Description of the curriculum and a multifaceted evaluation, including a pretest-posttest controlled trial.

SETTING

University-based primary care internal medicine residency program.

PARTICIPANTS

Second- and third-year internal medicine residents (N =34).

INTERVENTIONS

A 7-week EBM curriculum in which residents work through the steps of evidence-based decisions for their own patients. Based on adult learning theory, the educational strategy included a resident-directed tutorial format, use of real clinical encounters, and specific EBM facilitating techniques for faculty.

MEASUREMENTS AND MAIN RESULTS

Behaviors and self-assessed competencies in EBM were measured with questionnaires. Evidence-based medicine skills were assessed with a 17-point test, which required free text responses to questions based on a clinical vignette and a test article. After the intervention, residents participating in the curriculum (case subjects) increased their use of original studies to answer clinical questions, their examination of methods and results sections of articles, and their self-assessed EBM competence in three of five domains of EBM, while the control subjects did not. The case subjects significantly improved their scores on the EBM skills test (8.5 to 11.0, p =.001), while the control subjects did not (8.5 to 7.1, p =.09). The difference in the posttest scores of the two groups was 3.9 points (p =.001, 95% confidence interval 1.9, 5.9).

CONCLUSIONS

An EBM curriculum based on adult learning theory improves residents' EBM skills and certain EBM behaviors. The description and multifaceted evaluation can guide medical educators involved in EBM training.

Emphasizing the direct examination of evidence from clinical research to support clinical decision making, evidence-based medicine (EBM) has been proposed as a “new paradigm” for the practice of medicine. ^1,2 This approach requires unique skills of the physician, including the abilities to articulate a focused clinical question, efficiently search the literature, apply rules of evidence to clinical studies, and interpret the results for an individual patient.

With the proliferation of clinical trials and observational studies, the usefulness of an increasing amount of clinical maneuvers can be confirmed. ³ However, many physicians on the front lines of patient care do not apply this growing evidence base, ^4–6 perhaps owing to lack of access, ^7,8 skills, ^9–11 or acceptance. In response to these needs, medical educators, ^12,13 as well as advisory groups such as the Accreditation Council for Graduate Medical Education (ACGME) ¹⁴ and Association of American Medical Colleges (AAMC), ¹⁵ have called for the introduction of clinical epidemiology, biostatistics, critical appraisal, and medical informatics into medical school and graduate medical education curricula.

Despite consensus on the need to cultivate EBM skills, such training has proven difficult. A number of curricular methods have been reported for teaching critical appraisal and clinical epidemiology in medical schools. ^16–21 In residency programs, this training largely takes place in journal clubs, ^22–28 or in related literature-based curricula. ^29–31 These curricular efforts provide valuable guidance to medical educators in this area. However, many suffer from incomplete description, insufficient evaluation, or unproven effectiveness. Furthermore, focusing on knowledge of clinical epidemiology, critical appraisal skills, or exposure to emerging literature, journal clubs are not sufficient to train residents to practice evidence-based medicine, which requires interpretation of the evidence in the context of individual patient decision-making.

In September 1995, we instituted an EBM curriculum that focuses on individual patient decision-making for medical residents in the Yale Primary Care Residency Program. In this article, we present a description of the curriculum and the results of a multifaceted evaluation. We include the description because it represents a novel approach, places the evaluation in context, and is exportable to other programs considering EBM training.

DESCRIPTION

Development

In developing this curriculum, we began with a needs analysis. A review of our global program curriculum, informal discussions with faculty, and a precurriculum survey all pointed to insufficient training in EBM. The 25–item survey examined five areas: the residents' attitudes toward EBM, prior critical appraisal training, self-assessed EBM competence, medical reading habits, and preferences for information sources. Examples of questions and responses in these areas are listed in Table 1 Although the residents valued EBM, they had limited training and low confidence in their abilities, and they infrequently examined the evidence in their decision-making.

Table 1.

Precurriculum Survey Results (Selected Questions)

In conceiving an education strategy, we committed ourselves to satisfying the assumptions that underlie adult learning theory or “andragogy” ^32–34:

1. Adult learners need to know why they need to learn something before undertaking to learn it.

2. Adults prefer responsibility for their decisions and desire to be viewed as capable of self-direction.

3. Adults accumulate a greater volume of experience, which represents a rich resource for learning and necessitates individualization of learning strategies.

4. Adults become ready to learn things when they need to know them in order to cope effectively with real life situations.

5. In contrast to children's subject-centered orientation to learning, adults are life-centered (or task-centered).

6. While adults are responsive to some external motivators, their most potent motivators are internal.

Based on our review of this model, we enumerated a list of optimal learning conditions for our curriculum format: self-initiation, self-direction, realistic learning solutions, internal motivators, problem-centered organization, a variety of resources, and the opportunity to receive and offer feedback.

Next we performed a literature review of previous approaches to EBM training. Although no EBM-specific curricula appeared in the literature, several related journal clubs and critical appraisal seminars have been reported. ^16–31 In addition, educators have conducted workshops on EBM training at several academic meetings. Our review revealed seven attributes favored by teachers of EBM:

1. Small-group, learner-centered format for seminars;

2. General medicine as opposed to subspecialty faculty;

3. One-on-one resident-faculty opportunities;

4. Immediate clinical relevance;

5. Residents as both educators and learners;

6. Integration of EBM into mainstream of clinical work; and

7. Faculty role modeling of EBM.

Finally, we worked with the program directors to fit our curriculum, both logistically and philosophically, into the larger program schedule. As we desired the opportunity to meet with a small group of residents over a period of time, we decided to include our curriculum within the residents' yearly 12-week ambulatory block, in which they rotate through community internists' offices and non-internal-medicine specialty selections. The residents on this rotation convene weekly for a series of seminars that constitute an ambulatory core curriculum. For our EBM curriculum, we reserved 7 of the 24 total hours in this series.

Implementation

For the 1995 to 1996 academic year, we implemented an evidence-based medicine curriculum for junior and senior medical residents in a university–based primary care internal medicine residency program. Our overall goal was to enhance the residents' decision making and counseling for individual patients through the acquisition, evaluation, and application of evidence from the medical literature. Through their participation in this curriculum, they develop the skills, behaviors, and attitudes required for this “evidence-based” practice of medicine. These attributes are reflected in our learning objectives, listed in Table 2.

Table 2.

Curriculum Objectives for Evidence-Based Medicine

Educational Settings.

We convene seven, weekly, 1-hour tutorials for 5 to 14 residents during their yearly ambulatory block rotations. Each tutorial session is directed by a resident and facilitated by a general medicine faculty member. In preparation for the tutorials, the residents work in other important settings, including the patients' bedside, the MEDLINE terminal, and individual meetings with faculty.

Instructional Strategies.

Each of the seven tutorials guides the residents through a real clinical scenario, representative of one of six prototypical clinical questions, including therapy, prognosis, harm, diagnosis, prevention, and decision making. In the seventh, they can choose any question type, but must use a systematic review as evidence.

Tutorial preparation.

The resident scheduled to direct the tutorial recalls a recently encountered patient, whose evaluation, management, or counseling generated uncertainty but required action. Without exception, we insist that the case represent an actual individual patient, not a hypothetical patient inspired by an interesting article or a commonly encountered problem. He or she then meets briefly with a facilitator to distill the issue down to a focused answerable question. For guidance in this task, the resident refers to the “well-built clinical question” editorial, ³⁵ in the syllabus.

After this, the resident, guided by either the hospital librarian or one of the authors, performs a MEDLINE literature search to identify an article that addresses the clinical question. In the search strategy, we encourage the use of methodologic filter term to restrict the capture to articles of the highest quality. The most sensitive and specific terms have recently been identified in a series of ACP Journal Club editorials, ^36–39 which are included in the syllabus.

Finally, the resident meets again with the facilitator to appraise the article, consider the implications for the patient, and plan the tutorial. In addition, he or she reviews locally prepared handouts, the Evidence-Based Medicine Working Group's Users' Guides to the Medical Literature, ^40–48 and other selected articles. In the syllabus, these pieces are organized in chapters corresponding to the prototypical clinical questions. For general biostatistical or epidemiologic questions, the resident refers to handouts and articles in a general reference section.

For their tutorial preparation, the remaining residents read the article under consideration and the corresponding chapter in the syllabus.

Tutorial format.

The directing resident leads the seminar, following the format in Table 3. At each section, he or she invites a group discussion. We view the last section, “interpretation of the results for this patient,” as the most critical part of the seminar. At this point, we encourage the residents to recognize that the report of an effect of an intervention or exposure, even if valid, represents an average effect for an average patient. The residents anticipate the effect for their patient by examining subgroups or incorporating the patient's baseline risk. Furthermore, the residents determine a clinically meaningful measure of effect. For example, they usually calculate the patient's absolute risk reduction, the number needed to treat per favorable effect, and the number of adverse events per favorable effect for a therapy under consideration. In making their decision, the residents also consider the patient's preferences and risk aversion and the practical realities of the particular situation. Finally, they appreciate the need to make the best decision under uncertainty. The residents must take a stand, regardless of the limitations of the evidence, on how they will proceed with this patient's evaluation, management or counseling.

Table 3.

Tutorial Format

Role of faculty

While allowing the residents to direct the tutorials, the faculty facilitators play a critical role. Prospective facilitators, in preparation for this role, participate in a parallel faculty development EBM curriculum. In addition to refining their own EBM skills, they learn EBM tutorial facilitating techniques developed for this curriculum, which are listed in Table 4. At the end of each tutorial, facilitators offer feedback to the residents.

Table 4.

Facilitating Techniques

METHODS

Effectiveness Evaluation

To evaluate the effectiveness of the curriculum in improving residents' EBM behaviors and skills, we conducted a pretest-posttest controlled trial. The subjects were the 34 second- and third-year residents at a university-based primary care internal medicine residency program for the 1995 to 1996 academic year. According to their previously determined rotation schedule, residents assigned to two consecutive ambulatory blocks for a total of 20 weeks received the curriculum. Residents on other rotations for this same period served as controls.

Before and after the study period, both groups completed a questionnaire, which consisted of a survey of EBM behaviors, a survey of self-assessed EBM competence, and an EBM skills test (best obtainable score = 17). The skills test required free text responses to questions based on a clinical vignette and a redacted journal article (portions of the abstract and discussion deleted). For each question, the respondent listed two points to justify his or her level of agreement with statements such as “the results can be applied to this patient,” based on the Users' Guides to the Medical Literature. ^40–48 In addition, the respondents were required to calculate the relative risk reduction and absolute risk reduction from the data. The EBM skills test was modified from Stern et al., ⁴⁹ who asked similar questions but required Likert scale responses relating to an article alone. Two raters (the authors) independently and blindly scored all tests against a “gold standard” developed from faculty responses.

We assessed posttest-versus-pretest changes within groups using paired Student's t-tests and χ ² tests. For the EBM skills test, we also compared mean score changes in case versus control subjects using an independent samples t-test and performed a stratified analysis to assess the effect of confounding by differences in baseline characteristics. The residents gave oral informed consent and received an information sheet indicating the nature of their participation but not the specific study hypothesis.

Process Evaluation

For the first four curriculum cycles, we recorded tutorial attendance, faculty time commitment, and the cases presented and noted any emerging administrative difficulties. We also observed the residents' preparation for and participation in the tutorials, noting their reactions and the small-group dynamics, and informally debriefed faculty facilitators about their experience.

Institutional Impact Evaluation

We asked the program directors to review our progress after one year of implementation. In addition, the residents completed a Likert scale satisfaction questionnaire at the end of each cycle.

RESULTS

Effectiveness Evaluation

Out of 34 residents enrolled in the effectiveness study, 28 (82%) completed the pretest and posttest surveys only, and 26 (76%) completed both the pretest and posttest surveys and EBM skills test. Baseline characteristics of the residents are listed in Table 5. The case subjects included a higher percentage of third-year residents and had more research experience, while the control subjects had a higher level of critical appraisal training. None of the differences were statistically significant, in part due to the low power of the study.

Table 5.

Baseline Characteristics of Residents

Scoring the EBM skills test (possible scores 0 to 17), the two raters agreed within 2 points on 98% of tests and within 1 point on 67%. The correlation coefficient between raters was 0.87 (p= .01).

Table 6 shows changes in the residents' medical reading habits and information preferences. After participating in the curriculum, the case subjects significantly increased the frequency with which they examined methods and results sections of articles and the frequency with which they referred to original studies to answer clinical questions, while the control subjects did not. Neither the case nor the control subjects showed significant changes in numbers of hours spent reading per week, articles read per week, or literature searches per month. Changes in the residents' self-assessed competencies are illustrated in Table 7. The case subjects demonstrated significant improvements in their abilities to evaluate study design, evaluate statistics, and apply results to individual patients, while the control subjects did not. Neither group significantly improved their abilities to pose a focused question or perform a literature search, but these were rated high on the pretest.

Table 6.

Medical Reading Habits and Information Preferences

Table 7.

Self-Assessed Competence*

After participating in the curriculum, the case subjects significantly improved their scores on the EBM skills test (8.5 to 11.0, p= .001), while the control subjects did not (8.5 to 7.1, p= .09). Comparing the posttest scores for case and control subjects, the mean difference was 3.9 points (p= .001, 95% confidence interval [CI] 1.9, 5.9). This effect was preserved after stratifying by the residents' baseline characteristics (data not shown).

Process Evaluation

For the first four curriculum cycles, the mean resident attendance was 5.5 sessions out of a possible 7. The number of participants per session ranged from 5 to 14. After observing many tutorials, we found that 7 to 10 participants made for the most stimulating dynamic. Fewer participants precluded a diversity of viewpoints and productive discussions, and a larger group afforded those inclined a measure of anonymity and disengagement.

Given the residents' complicated and geographically dispersed schedules, organization of the curriculum presented significant logistical challenges. A high level of administrative support was required to ensure preparation of the directing resident, link the directing resident with a faculty facilitator, and ensure distribution of the articles.

Some of the residents were initially reluctant to discuss real cases. Reasons for their reticence included a lack of “good studies” relevant to their clinical questions, lack of “interesting issues” in their immediate experience, and the desire to review a particular article for its own sake. As we considered the use of the residents' real experiences a cornerstone of the curriculum, we insisted on this discussion despite their reservations. Over time, the residents began to appreciate this approach and viewed the EBM tutorial as a forum in which to work out their clinical dilemmas.

Faculty time intensity was significant. The authors attended nearly all of the sessions (approximately 35 in the first year) and actively facilitated about half of them each. Thus far, we have recruited two other faculty to facilitate one session per curriculum cycle. For each 1-hour tutorial, faculty facilitators spent from 1.5 to 3.5 hours in preparation, including meeting with the resident.

Recruitment of faculty facilitators was difficult, probably because of the time commitment and perceived skills requirement. Given the tutorial format, however, we stressed that critical appraisal “expertise” is less important than tutorial teaching dexterity and clinical savvy. Faculty valued the opportunity to expand their own EBM repertory and further develop their small-group teaching skills.

Institutional Impact

The curriculum was well received by the residents, faculty, program directors, and departmental leadership. A survey of the participants showed them to be highly satisfied with the level of learner participation, the syllabus, and the utility of the curriculum in helping them acquire, appraise, and apply “the evidence” in making decisions about individual patients (data not shown). The program directors agreed that our curriculum met the needs suggested by both our internal review and the ACGME. Hence, they decided to continue to support it as an ongoing experience. Finally, after we presented the curriculum at a regional and national meeting, several other programs have expressed an interest in adopting a similar approach.

DISCUSSION

We have presented an evidence-based medicine curriculum for medical residents. The educational strategy includes a tutorial format, use of residents' actual clinical experiences, resident selection of cases and clinical questions, a variety of resources, and one-on-one faculty opportunities. We also provide faculty development, both in EBM skills and EBM facilitating techniques.

Our effectiveness evaluation demonstrated a positive impact on the residents' EBM behaviors and skills. The pretest-posttest controlled trial design provides strong evidence for a true effect on the learners. Furthermore, an EBM skills test showed good reliability, captured the residents' thinking process in free text, and measured actual EBM skills, rather then surrogate abilities such as epidemiologic knowledge. After participating in the curriculum, the residents increased the frequency with which they referred to original articles, their reading of methods and results sections, and their self-assessed EBM competence. And, most importantly, they improved their ability to appraise the report of a clinical study and interpret the results for an individual patient. Other facets of our evaluation illuminated a noteworthy administrative burden and faculty time commitment and a favorable institutional reception.

Among the published curricula, ours distinguishes itself by the rigor of its evaluation and its efficacy in improving real EBM skills. No other approach to EBM training in graduate medical education has been shown to improve residents' ability to appraise the evidence in the context of individual patient decision making. In the only randomized controlled trial of a journal club, Linzer et al. showed that journal club participation improved resident's self-reported reading habits and epidemiologic knowledge, but did not affect their ability to appraise a test article. ²⁷ Residents in a general medicine faculty-directed journal club reported reading articles more critically compared with residents in a journal club led by a chief resident and a subspecialist, but neither group changed the way they used the medical literature in their practice. ²⁴ The two groups performed equally on an epidemiology test, but there was no pretest to establish the true effect of the curriculum. In Seelig's uncontrolled trial of a journal club augmented with an adult learning theory intervention, residents improved their critical appraisal knowledge and self-assessed skills but did not change their perceived value of journals for keeping up. ²⁶ Kitchens and Pfeifer reported a literature-based curriculum, which improved residents' performance on a clinical epidemiology test in the second phase of a crossover trial. ²⁹ When junior and senior family medicine residents received Gehlbach's critical appraisal seminars, ³⁰ they scored higher than unexposed interns on a clinical epidemiology test, but the evaluation strategy lacked a pretest and a comparable control group.

Our curriculum's effectiveness, we believe, derives largely from its fidelity to adult learning theory. Using their own actual clinical scenarios, often as they evolve, the residents acutely understand why they need to learn something, take responsibility for their learning, exploit their experience as a resource, link their readiness to learn with the exigency of real life situations, and orient their learning by life tasks. Other curricular elements that facilitate adult learning include the variety of resources and resident direction of the tutorials. More pedagogic approaches have not enjoyed the same efficacy.

Our emphasis on actual individual patient decision making also distinguishes our program from other curricular efforts and fills an important niche in graduate medical education. Most residency programs offer traditional journal clubs, ^22,23 or similar literature-based seminars, which focus on epidemiologic knowledge, general critical appraisal skills, or keeping up with emerging literature. These curricula, however, are not sufficient to train residents to practice EBM, which requires interpretation of the evidence in the context of individual patient decision making. Working through decisions for their own patients, residents in our curriculum appreciate the importance of incorporating “the evidence” into the larger process of clinical judgment. Beyond appraising the validity of associations, they must consider their particular patient's anticipated benefits and risks, risk aversions, preferences, and practical realities. The exigency of these scenarios, furthermore, reinforces the difficulty yet inevitability of decision making under the uncertainty of limited evidence.

We recognize several potential limitations of our curriculum. First, in terms of the evaluation strategy, the participants were not randomly assigned. However, they were arbitrarily assigned based on the program schedule, which was determined before the trial was contemplated. Second, there is a possibility of a test-training effect since we used an identical posttest; that is, it is possible residents learned how to take the same test better the second time around rather than really learning new skills. However, if this were the case, we should have seen an improvement in the control subjects' scores, but they actually deteriorated. Third, our evaluation was short term, so we cannot assess the durability of the skills. Lastly, although the EBM skills test can be scored reliably and has content validity, it lacks validation in other settings. Similar instruments, however, have shown the ability to discriminate between different levels of expertise. ^20,49

In addition, although our focus on individual patient decision making represents an advance over journal clubs and generic critical appraisal seminars, the residents' experi-ence does not completely simulate “real world” EBM. The directing resident enjoys the luxury of undistracted time and individual faculty attention as he or she works through the steps of an evidence-based decision. Although this is helpful for a first formative experience, it does not confront the dilemma of a busy practitioner, who must practice EBM in the face of significant time and possibly technologic constraints. As an initial attempt to address this issue, we encourage the residents to utilize their EBM skills and resources in their clinical decision making beyond the tutorial series. We cultivate EBM in various residency settings, including work rounds, attending rounds, morning report, and continuity clinic. In addition, using a format similar to the “critically appraised topic,” ⁵⁰ we have recently begun to catalogue and disseminate the residents' clinical questions.

In terms of future plans, we anticipate that electronic linking of our teaching venues will overcome some of the logistical problems. We intend to retest the residents after one year to see if their skills extinguish. Lastly, we plan to bring EBM to the community internist's office, a major site for ambulatory training in our program. We will begin with an assessment of the practitioners' information needs, current resources, and EBM skills and attitudes. From these data, we expect to provide them with tailored faculty development in practicing and teaching EBM and electronic linkage to medical information. We will then evaluate the educational impact on the residents rotating through these sites.

We have presented the development, implementation, and evaluation of an EBM curriculum for medical residents at our institution. Our focus on decision making for the residents' actual individual patients represents an important curricular initiative. Our description of each phase, analysis of our experience, and demonstration of effectiveness, we believe, can guide medical educators involved in EBM training.

REFLECTIONS

Love Poem of the Rural Doc

Because he loved his work, he suffered,

Loved to suffer and suffered to love

And suffered to know the work of love

As he had suffered the love of work

Until finally he assumed their suffering,

Heel driven against the blade, he shoveled

Great spoonfuls into his pickup truck

Until it sagged beneath the weight

Spilling into every corner of the bed.

He drove the dried fields at the edge

Of town. It was dusk. The sweep

of wind was the only music.

Whorled lines of square hands loved

The labor of the shovel handle as

He unloaded what he could, then with

The bed-door open, sped through the fields

Leaving a cloud like a duster plane

And as the suffering settled, he kept driving,

One hand on the wheel, one out the window,

Chill filtering between his fingers, but

It didn't matter. Nothing mattered because

he was exhausted and he was in love.

*

To conclude the evening hours, he performs

a complete physical on Mrs. Ruth Hartigan

then closes up, walks home. Along the way,

a sense of belonging circulates the dirt-veined town.

Visitors can palpate this. Basketballs percuss

gravel driveways. Four-wheeled bicycles graze

the lawns, fat purple handgrips antler their frames,

and dry weeds digest the edge of every home,

ossified in place. The neighbor's words lick

with canine sensitivity, while moths soften

the lamplights, perfected in the sky. He lives here.

He has no choice.

This town grips him like ivy.

Ambivalence, anguish, couplings, revenge,

to which he is privy, glue the pieces, artwork

shown at the children's farm district fair.

It is not difficult to love this work:

crepe and Elmer's, lives in fingerpaint,

Mrs. Hartigan's red handprint is

the blossom on his construction paper tulip

and the moon floats in their tea cups.

O yes, loving them all equally, the moon.

*

Vroom. Vroom. Does he see a mountain

of tomatoes? Two helium balloons surprise him

as he opens the trunk. The pictorial encyclopedia

of Civil War equipment, written by a dentist,

plays in the forefront, while Chris drags

a small felled tree to the garbage in the background.

Chris is 3 years old. Who will put the fruit away?

He thanks goodness the other kids are sleeping or

Darlene, love of his life, would never understand

pretend reading in speech-language-impaired children,

and he would feel guilty as ever changing shoes

to go to the ER.

*

“Thank you, Carole, for the Easter Pie,

Although it was not really pie at all,

But rice pudding. The pineapple chips

Were especially appreciated.

Thank you for the way you wear your rouge,

Although it truly isn't vanity at all,

Rather bravado, your transcendence of death.

Thank you for your suffering, most kind

And sacrificial, although martyrdom

Is certainly not your style.

And mostly, thank you for your aneurysm.

You woke, worked, slept, dreamed

And woke again, day after day,

Until one day, a very special day,

You popped a hole in your heart

Like a bottle of champagne

To commence a small celebration

Of what we do for a living

And why we love it.”

Phillip J. Cozzi,MD Elmhurst, Ill.