Finding the evidence: teaching medical residents to search MEDLINE

INTRODUCTION

Increasingly, medical residency programs are teaching their trainees how to practice evidence-based medicine (EBM), with one-third of internal medicine programs now offering freestanding EBM curricula [1]. Most EBM curricula described in the literature focus on teaching critical appraisal of original journal research [2]. However, other aspects of EBM are equally important. One crucial step is efficiently finding the best literature that answers clinical questions. As part of a newly implemented EBM curriculum, the authors have developed a workshop to teach our internal medicine housestaff how to search MEDLINE efficiently and effectively to find the answers to clinical questions arising from patient care.

PROGRAM DESCRIPTION

In the fall of 1997, we conducted a needs assessment to ascertain our residents' skills in literature searching. We performed a convenience sample by distributing a survey at various meetings of residents. Thirty-six percent of the respondents described their MEDLINE searching skills as only fair or poor. In addition, 56% felt teaching in this area would be extremely or very useful for them.

To address the identified learning needs of our residents, we developed a three-hour workshop on using Ovid's version of MEDLINE to find high-quality information to answer clinical questions. After a pilot test of the workshop, the session was implemented with all second-year medicine residents (N = 42, male = 61.9%, mean age = 30). We taught the workshop six times (once per month over a six-month period), with an average of seven residents per workshop.

WORKSHOP CONTENTS

In advance of the workshop, the residents were assigned to read a series of articles about using MEDLINE and searching to find answers to clinical questions [3–5]. The workshop began with a thirty-minute didactic presentation on MEDLINE searching. Topics included Medical Subject Headings (MeSH) versus textword searching, “explode” feature, search term truncation, Boolean operators, and use of the methodological filters developed by Haynes et al. [6]. Lastly, we provided an overall approach for converting clinical questions from patient encounters into MEDLINE search strategies [7, 8] (Table 1). The approach emphasized retrieval of the largest possible sets (“mega-sets”) for each of the strategy components to find all of the potentially relevant literature. The methodological filters were then used to identify the most clinically relevant articles.

Figure 1 MEDLINE performance checklist

The class then moved to the library computer center, where the residents were provided with personal computers for the remainder of the workshop. Searches were performed with Ovid's MEDLINE search interface. We guided the residents step by step in performing two MEDLINE searches for clinical questions from two hypothetical clinical scenarios, using the search functions and strategies described in the earlier didactic session. A handout contained materials from the didactic session, the clinical scenarios, and step-by-step instructions for the MEDLINE searches. Finally, class participants independently performed practice searches from a third scenario, with instructors available for assistance. All scenarios related to problems that would be seen in primary care medical practices. Two instructors supported the workshop; one primarily led the step-by-step instructions, and the other provided individual assistance for participants as needed.

PROGRAM EVALUATION

One week before the workshop, residents performed a pretest MEDLINE search, based on a clinical question from a hypothetical patient care scenario. Following the workshop session, the residents were assigned new scenarios for searching MEDLINE as a posttest. For example, one of the assigned questions was “In patients with Alzheimer's disease, do any vitamin supplements improve their cognitive function or slow progression of disease?” In addition, one to eleven months after the workshop, for a subsequent session of the EBM curriculum, residents independently performed MEDLINE searches to answer clinical questions from one of their own patient encounters. Performance checklists were designed for the MEDLINE searches. The performance checklists evaluated whether residents demonstrated the correct use of important MEDLINE strategies at least once in their searches and whether they retrieved the appropriate study type and articles to answer the clinical questions (Figure 1). Search evaluations were performed by one of the authors (Vogel).

We compared the proportions of residents appropriately completing each searching task before, after, and in “long-term” searches, using the McNemar change test (Analyze-It Software). The residents' use of MEDLINE searching strategies and techniques all increased significantly (P < 0.05) in searches completed in the week after the workshop, compared to baseline. For all of the searching tasks in the long-term search, one to eleven months after the workshop, a significantly higher percentage of residents correctly used MEDLINE searching skills compared to their pretest searching (P < 0.05). However, in long-term searches, residents employed the explode function, textword searching, truncation, and the “OR” function significantly less frequently (P < 0.05) than their posttest search.

Written feedback was sought at two times: at the end of the workshop and as part of overall EBM course evaluations a week later at the end of their ambulatory block. The workshop was generally very well received by the residents. In feedback at the end of the ambulatory block, more than half the residents rated the workshop as one of the best learning experiences of their rotation. Two participants suggested using residents' own clinical questions to teach searching in the future.

DISCUSSION

We have developed a workshop that improves residents' ability to search MEDLINE effectively for information to answer clinical questions. For all searching techniques evaluated for the workshop, the percentage of residents using the technique appropriately improved significantly after the workshop. In self-directed MEDLINE searches later in the year, residents continued to demonstrate improved skills compared to their baseline abilities, although there was a significant decline in the use of some skills compared to their searches right after the workshop.

A MEDLINE search back to 1990 yielded only a few studies related to developing and conducting classes on MEDLINE searching for residents. Most articles that concerned MEDLINE training discussed it as part of the curriculum in medical school, the more recent articles as a component of EBM instruction [9–14]. Other articles attempted to measure improvement in searching skills of residents and attending physicians after participating in MEDLINE training classes. These studies reported some improvement but also concluded that designing a good study about teaching methods was difficult, making additional research needed [15–17]. We also found articles discussing classes devoted to teaching more general information retrieval skills, but these articles did not detail structure and content of individual sessions in comparison to what we describe here [18–20].

The evaluation of our workshop had several significant limitations. First, there was no control group for comparison of searching skills over time. Therefore, we could not exclude the possibility that apparent improvement in abilities was related to some factor other than our instruction. Second, as only one of the authors evaluated the residents' searches, the assessment of searching skills might have had some bias, possibly resulting in overstatement of the magnitude of the benefit of the workshop. Third, we only evaluated the skills of residents in these structured searching assignments; we could not assess how frequently or thoroughly residents might perform searches on their own as a result of participating in this workshop.

In the future, we need to evaluate how teaching MEDLINE searching has an impact on residents' abilities to find information for questions about their actual patients. It would be important to know that participants not only improve their searching skills from workshops like this, but also are more likely to search for information in this way. We hope participants would become more inclined to perform literature searches to find the most up-to-date information to answer clinical questions, as opposed to looking at less recent or less valid resources.

Table 1 Overall MEDLINE searching strategy presented in workshop