Real-time, evidence-based medicine instruction: a randomized controlled trial in a neonatal intensive care unit

Abstract

Purpose: The study assesses potential for improving residents' evidence-based medicine searching skills in MEDLINE through real-time librarian instruction.

Subjects: Ten residents on a rotation in a neonatal intensive care unit participated.

Methodology: Residents were randomized into an instruction and a non-instruction group. Residents generated questions from rounds and searched MEDLINE for answers. Data were collected through observation, search strategy analysis, and surveys. Librarians observed searches and collected data on questions, searching skills, search problems, and the test group's instruction topics. Participants performed standardized searches before, after, and six-months after intervention and were scored using a search strategy analysis tool (1 representing highest score and 5 representing lowest score). Residents completed pre- and post-intervention surveys to measure opinions about MEDLINE and search satisfaction.

Results: Post-intervention, the test group formulated better questions, used limits more effectively, and reported greater confidence in using MEDLINE. The control group expressed less satisfaction with retrieval and demonstrated more errors when limiting. The test and control groups had the following average search scores respectively: 3.0 and 3.5 (pre-intervention), 3.3 and 3.4 (post-intervention), and 2.0 and 3.8 (six-month post-intervention).

Conclusion: Data suggest that measurable learning outcomes were achieved. Residents receiving instruction improved and retained searching skills six-months after intervention.

INTRODUCTION

Physicians of the new paradigm rely heavily on sound knowledge of the rules of evidence and the ability to access, select, and interpret useful references from the medical literature [1]. To support and encourage such practices from new students and clinicians, staff at both Taubman Medical Library and the Medical School of the University of Michigan had been experimenting with the teaching of evidence-based medicine (EBM) but with no collaboration or communication between them. Within individual departments in the medical school, EBM training varied from doing nothing at all to conducting quite extensive literature searches using EBM search strategies and critically appraising articles for journal clubs. Thus, there was very little consistency. In the fall of 1998, faculty from the Department of Pediatrics and Communicable Diseases approached librarians at Taubman about teaching a hands-on session for faculty on searching for evidence-based literature in MEDLINE and the Cochrane Database of Systematic Reviews. This session would be part of a one-day seminar for the Faculty Development Program in General Pediatrics. The success and positive reaction of the session led to a second faculty session in December 1998.

Curious how such teaching would play out in a “real-time” clinical situation, Schumacher, one of the present investigators, invited librarians to attend rounds in the neonatal intensive care unit (NICU) in January 1999. After rounds, residents with questions about patient care worked one-to-one with a librarian and learned EBM searching techniques in MEDLINE. The librarian also attended weekly resident meetings to discuss their searches. The goal was to provide instruction on EBM search techniques and to review basic MEDLINE searching. Through repeated quick sessions, the investigators speculated that residents would develop a pattern of seeking information for clinical problem solving, instead of perceiving MEDLINE searching as prohibitively time intensive. Survey feedback from residents indicated that this experience was very positive and educational for them.

Desiring to take this pilot project one step further, the investigators created a proposal to conduct a randomized controlled trial (RCT) in the NICU and submitted it to the Medical Education Scholars Program (MESP). Funded through the medical school, this unique program is designed to prepare faculty for leadership roles in medical education and to provide assistance in conducting research into improving medical education. MESP accepted the proposal, and the first of a series of three cohorts commenced in September 1999. This paper details the study, with results from the first cohort, to assess the potential for improving residents' EBM searching skills in MEDLINE through real-time librarian instruction.

LITERATURE REVIEW

The librarians conducted extensive literature searches in the MEDLINE, ERIC, and Library Literature databases to prepare the proposal for MESP. To aid in the design of the RCT and data collection methods, the librarians sought to identify RCTs of end-user MEDLINE training, studies on teaching EBM searching skills, studies identifying the most problematic aspects of MEDLINE searching for medical students and physicians, and studies seeking to evaluate the quality of the many EBM search hedges for MEDLINE. For this last query, the Internet search engine HotBot was used to supplement the traditional bibliographic databases noted above.

In the past fifteen years, many studies have examined the effectiveness of training sessions for end users of MEDLINE or the success of end users as independent searchers. In one study, Walker et al. found that difficulties associated with the logic of the MEDLINE database accounted for nearly half of the unproductive searches performed by their study participants [2]. Additional difficulties found in this same study were formulating search strategies poorly, specifically using a Boolean “AND” with redundant terms, and using general terms instead of appropriate subheadings. These errors were demonstrated in earlier studies like that of Slingluff, who found that the structures of databases and use of appropriate terms were two aspects of searching with which users had the most trouble [3]. Kirby pointed to inadequate search strategy formulation as the cause of most unsuccessful end-user searches [4]. Understanding the areas of difficulty presented in these specific studies resulted in the design of improved data collection methods for this RCT.

The medical literature also yielded further rigorous studies of end-user searching. Erikson and Warner conducted an RCT to examine the impact of an individual MEDLINE tutorial session on obstetrics and gynecology residents. Although the sessions were well received, they found no improvement in the outcome measures analyzed [5]. Haynes led an RCT to investigate whether the clinical use of MEDLINE was enhanced by assigning preceptors and providing feedback on individual searches. A considerable problem in this trial was that participants were not required to consult with their preceptors, thus very few sought expert help with their searches. While this study found no improvement in the quality of searches performed by users with access to preceptors or those receiving feedback on their searches, Haynes noted that search performance in the test and control groups improved after a basic introduction to searching [6]. To avoid the pitfalls of the Haynes study, the present study was designed so that a minimum of several training sessions with librarian contact would be required.

At the time of this study, there were almost no studies during clinical rounds or in clinical educational environments that were RCTs and involved real-time problem solving or finding evidence-based literature. Michaud, through a demonstration model, attempted to encourage faculty and residents to use EBM in their daily practice. House staff were receptive to the information obtained from the medical literature and willing to base clinical decisions on relevant findings [7]. However, Michaud also discovered that accessibility was key, and, in this case, accessibility meant not only access to computers but also to the knowledge of how to perform a literature search, formulate a clinical question, and critically appraise findings. Sackett and Straus published a brief report on the use of an “evidence cart” containing print materials, locally produced critically appraised topics, and access to MEDLINE and the Cochrane Library during clinical rounds. Sackett stated that although the study had limitations, they learned that “evidence made available within seconds during rounds altered the clinical approach of at least 1 team member 48% of the time, but when the evidence was not readily available, the clinicians rarely searched for it” [8]. This finding reinforced the desire to keep each encounter between NICU residents and librarians to a brief duration and to begin immediately following rounds, before other clinical demands would take priority.

METHODS

Study design

An RCT study design was selected to evaluate whether real-time instruction and feedback by medical librarians resulted in improved EBM searching in OVID MEDLINE. Ten residents on a one-month rotation in the NICU participated in this study. They represented programs in pediatrics, emergency medicine, medicine/pediatrics, and family medicine. According to standard practices in the NICU, residents were arbitrarily assigned to one of four teams for the month-long assignment to the unit. For the purposes of this study, the investigators assigned teams to the test group or to the control group by a random drawing, resulting in the test and control groups each consisting of five residents. Teams rather than individuals were randomized, as the researchers felt that the potential risk for inadvertent unmasking of assignment within teams was too high. All NICU staff were blinded as to group assignment. Only the librarians knew to which study group individual residents were assigned.

Residents answered pre- and post-intervention surveys to measure changes in attitudes and behaviors related to information seeking. Questions were almost identical on both surveys. The post-intervention survey is included in Appendix A. Additionally, the residents independently performed a series of three standardized searches pre-, post-, and six-months post-intervention, emailing the strategy and retrieval from each search to the investigators. The standardized searches were presented as written clinical scenarios involving pediatric patients and were based on systematic reviews in the Cochrane database. This allowed for an analysis of the precision and recall of search retrieval by using the list of studies included in each Cochrane Review as a predefined set of quality EBM articles that could be compared with the retrieval of the residents' searches. Further, the librarians evaluated all strategies with a locally designed search strategy analysis tool (Appendix B). Using a scale of 1 to 5 with 1 being highest, points were deducted for a variety of common searching errors. These outcome measures are discussed in more detail below.

Intervention

As part of their orientation to the NICU, all residents attended a lecture on clinical decision making and strategies for asking good questions. Each received a handout covering the lecture content. The ten residents completed the pre-intervention survey at the beginning of the rotation to assess initial attitudes and information-seeking behaviors. Pre-intervention survey questions focused on their preferred sources for patient care information, frequency of searching for patient care information, and level of satisfaction with their personal searching skills. Residents also performed the pre-intervention standardized search and emailed the retrieval and strategy to the librarians for analysis as a baseline of their searching skills.

During the study, residents, in consultation with the attending physicians, formulated patient care questions to answer by performing literature searches. The questions were generated by discussions of patient care during the day's rounds in the unit. Information gained by answers to the questions was thought to have potential to affect individual patient care (either alter or confirm decisions). A librarian arrived in the NICU two to three days per week, immediately following rounds. The study began with two librarians alternating days, while a third librarian observed to gain experience and participate later in the study. The librarian sat with each resident individually in a private office in the NICU, while the resident attempted to find clinically relevant information in MEDLINE. For every session, the librarian completed a data collection form to capture information on the search experience. The resident stated the question to be answered, and the librarian noted whether or not it included each of the four elements of a good question: population, intervention, comparison intervention, and outcome (PICO).

Depending on whether a resident was in the control group or the test group, each session followed one of two paths. For those in control group, the librarian observed the search, noting on the data collection form any independent knowledge of searching and any problem areas encountered. No feedback or instruction was provided. Residents were told, if they inquired, that they were being observed to determine where users had difficulty with the MEDLINE interface. For the test group participants, the librarian provided active instruction based on the nature of the question and the searcher's level of skill. Instruction frequently included breaking the question into searchable elements, locating and using appropriate Medical Subject Headings (MeSH), applying subheadings, keyword searching, and truncating terms. Residents received EBM search hedges, developed by librarians at the University of Rochester Medical Center Library [9], on a floppy disk and learned how to use the hedges and save them under their MEDLINE passwords for future use. Data collected from these interactions included the type of instruction provided, problems encountered, and areas of independent knowledge.

Post-intervention

All residents completed the second survey following the intervention. This survey was designed to measure any changes in attitude toward doing literature searches for patient care information, satisfaction with searching skills, and ability to appraise the articles critically (Appendix A). Study participants performed the post-intervention standardized search. To assess long-term retention of skills learned during the intervention phase, participants also conducted another standardized search six-months post-intervention.

Evaluation of standardized searches

The investigators scored the three independently conducted searches for precision and recall. Recall was defined as the number of relevant citations retrieved divided by the total number of relevant articles listed in the Cochrane review along with any relevant articles identified by a search done subsequent to the Cochrane review, using the stated Cochrane search strategy. Precision was defined as the number of relevant articles divided by the total number of articles retrieved.

A team of three librarians evaluated the quality of each search strategy using the search strategy analysis tool. The instrument was designed to quantify failures in search logic, technique, or both in approaching the clinical question. The librarians judged the success of each search based on whether searchers showed evidence of having formed good questions, used MeSH where appropriate, applied logical limits, and searched for keywords in an effective manner, if keyword searching was included. This team reviewed each search twice to ensure consistency of evaluation and applied the scale to determine overall numerical score.

RESULTS

Data collection forms

The librarians observed a total of thirty-five searches during the intervention phase of the study. Participants in the test group performed nineteen of the searches, while those in the control group conducted sixteen. This represented a range of two to five searches performed per resident. For the test group, librarians most frequently provided instruction on use of MeSH, use of evidence-based search strategies, and keyword searching techniques. The control group most often showed difficulty in using MeSH effectively, formulating good search questions, and locating quality articles that would facilitate practicing evidence-based medicine.

Pre- and post-intervention surveys

Residents' responses to the pre- and post-intervention surveys were compared. While not all questions yielded statistically significant differences, several demonstrated very important post-intervention differences. For example, residents' rankings of their preferred sources of clinical information and satisfaction with their searching skills changed noticeably. Working on a scale of 1 to 5 with 1 being highest, the test group ranked MEDLINE higher on average as a preferred resource than the control group prior to the intervention. Although both the test and control groups elevated MEDLINE in their rankings in the post-intervention survey, the test group's ranking increased by 1.4 while that of the control group increased by only 0.6 (Table 1). Furthermore, when asked to assess satisfaction with personal searching skills, rankings of both the test and control groups improved. The test group showed greater satisfaction before and after the intervention, but the control group's search satisfaction rose by a slightly greater degree, 0.6 compared to 0.8, respectively.

Table 1 Survey results for preferred source and personal searching skill satisfaction

Several survey questions attempted to measure the level of personal confidence in searching and in search success. Residents ranked their agreement with statements using a five-point scale, with 1 indicating strong disagreement and 5 indicating strong agreement. On the pre-intervention survey, the test and control groups, on average, ranked their ability to find relevant patient care information at 3.4 and 3.0, respectively. After the intervention, these averages had changed to 4.2 for the test group and 3.2 for the control group. The test group's confidence in their ability to find relevant information increased notably, by 0.8, while the control group's increase was a more modest 0.2 (Table 2). Similar results were represented by the magnitude of the increase in the test group's confidence in finding adequate search terms in MEDLINE versus that of the control group, 0.8 and 0.2, respectively. Even more illustrative were the attitudes of both groups about their abilities to formulate effective search strategies in MEDLINE. The test group's average increased by 0.4, while the control group's average decreased by 0.8, indicating that the control group found developing search strategies more difficult after participating in the intervention.

Table 2 Pre- and post-intervention survey results of residents' perceptions of personal searching abilities

Search strategy evaluation

The librarians scored search strategies from the individual pre-, post-, and six-months post-intervention searches using the search strategy analysis tool (Figure 1). In the pre-intervention search evaluation, the test group scored a mean average of 3.0, while the control group averaged 3.5. Failure to develop an answerable clinical question was the most common error, with residents often leaving out key elements of PICO. In total, 100% of the control group and 50% of the test group made this error. In the post-intervention search strategy evaluation, the test group scored an average of 3.3 and the control group averaged 3.4. Again, the failure to formulate an answerable clinical question was the most common error, with 100% of the control group and 60% of the test group making this error. In this case, the failure to ask a good question was believed to be attributable to the ambiguity of the questions used for the post-intervention search. This issue is addressed further in the discussion. Lastly, in the evaluation of the six-months post-intervention search strategies, the test group improved its average score to 2.0, while the control group's average scored decreased to 3.8. Individuals in the control group still exhibited difficulty asking an answerable question, and four out of five used MeSH incorrectly. In the test group, four out of five residents asked an answerable question and applied the EBM search strategies that they learned during the intervention, thus demonstrating the long-term retention of these skills.

Figure 1.

Search strategy evaluations of standardized searches

Recall and precision

The final measurement of search success was analysis of precision and recall of the individual pre-, post-, and six-months post-intervention searches. Mean recall for the control group was 18% of 11 Cochrane articles in pre-intervention retrieval, 17% of 13 Cochrane articles in post-intervention retrieval, and 27% of 20 Cochrane articles in six-months post-intervention retrieval. Mean recall for the test group was 44% of 11 Cochrane articles in pre-intervention retrieval, none of 13 Cochrane articles in post-intervention retrieval, and 41% of 20 Cochrane articles in six-months post-intervention retrieval.

In pre-intervention, the highest individual precision for the control group was tabulated at four Cochrane articles from a retrieval of forty-three (9%), and the lowest was tabulated at zero Cochrane articles from a retrieval of sixty-five (Figure 2). For the test group, the highest individual precision was six Cochrane articles from a retrieval of twenty-two (27%), and the lowest precision measure was four Cochrane articles from a retrieval of 104 (4%). In post-intervention, the highest individual precision for the control group was tabulated at two Cochrane articles from a retrieval of twenty-five (8%), and the lowest was zero of twenty-four Cochrane articles from a retrieval of twenty-five. Lastly, in the six-months post-intervention control group, the highest individual precision measure was tabulated at ten Cochrane articles of thirty-two (31%), and the lowest was tabulated at zero Cochrane articles of fifteen. In the six-months post-intervention test group, the highest individual precision was measured at fourteen Cochrane articles of fifty-one (27%).

Figure 2.

Precision of standardized searches

DISCUSSION

The three outcome measures involved in the study were: (1) analysis of pre- and post-intervention surveys; (2) analysis of strategies from pre-, post-, and six-months post-intervention standardized searches; and (3) precision and recall measures from the three standardized searches. From the perspective of the librarian investigators, the most significant outcome measure was the analysis of search strategies from the standardized searches. Using the search strategy analysis tool, the librarians quantified and thus systematically compared subjects' search strategies. Secondarily, survey analysis provided an overall picture of the information-seeking attitudes and trends among the test and control group participants. Lastly, precision and recall from the three standardized searches provided insight into the possible clinical value of participants' searching skills.

The pre- and post-intervention surveys provided a qualitative measure of residents' information retrieval attitudes and practices. Answers to questions about being satisfied with searches, finding proper MeSH terms, using search syntax, and formulating search strategies provided the authors with insight into the participants' confidence as searchers. As a whole, the test group's confidence demonstrated a marked improvement from pre- to post-intervention surveys, while in most areas such as confidence in finding MeSH terms and formulating search strategies, the control group showed little or no change in confidence. Interestingly enough, however, both test and control groups stated that the program enhanced their educational experience in the NICU and that acquired searching skills would aid in future information-seeking endeavors.

Just as basic principles of EBM hinge on asking good questions, the researchers also had the responsibility to ask a good study question. The question was two-fold: Does real-time librarian instruction in a dynamic clinical environment improve residents' EBM searching skills in MEDLINE, and do they retain those skills? According to search strategy analysis and search success, the researchers discovered, through the scores of the test group, that librarian instruction resulted in the improvement of searching skills and in the long-term retention of these skills.

The immediate post-intervention search question might have introduced an unexpected variable. The subjects as a whole had difficulty identifying the type of question being asked in the scenario, mistaking a “therapy” question for one of “prognosis.” Results of precision and recall for this search seemed to confirm this difficulty. Both test and control groups had lower precision and recall measures in post-intervention than in either pre-intervention or six-months post-intervention. The authors postulated that this particular search question might have seemed ambiguous to the subjects.

The “real-time” aspect of the study was highly noteworthy. The fact that searches were performed and customized instruction was provided in a high-intensity environment during periods of patient care may have had a positive affect on retention of searching skills. Residents were familiar with real-time instruction in other aspects of their medical education and training, so why not in information-seeking behavior as well? The immediacy of results was a motivating factor behind search success in the NICU. Thus, the librarians believed that the real-time nature of this study positively affected the searching confidence and retention of EBM searching skills in the test group.

LIMITATIONS

The study sought to use an independent “gold standard” against which to measure search precision and recall. From clinicians' perspectives, precision and recall might represent the “bottom line” of their abilities to retrieve information with the potential to affect patient care. The lists of articles included in Cochrane reviews might be viewed as a gold standard. However, the librarians found the precision and recall of Cochrane articles to be less significant measures of successful searching skills. Specifically, Cochrane contributors conduct very broad literature searches to locate RCTs on a topic, hand search some of the literature, and occasionally include unpublished studies. Therefore, the list of articles included in a Cochrane systematic review would not be completely comparable to the retrieval from a MEDLINE search.

In the intense clinical environment of the NICU, librarians working with residents found that interruptions during searching sessions were common. Residents were often paged, interrupted to answer patient care questions, or, on some occasions, called away to see patients. It was a challenge to make the instructional encounter as standardized as possible among the subjects.

The residents were asked not to discuss their searching encounters with others. However, the subjects were curious about the presence of librarians in the NICU, and the researchers could not be certain that the subjects maintained confidentiality.

CONCLUSION

The purpose of the study was to determine if real-time searching and EBM instruction would impact searching skills of residents in the NICU and, more significantly, if they would retain the skill sets to which they were introduced during the intervention phase. Residents receiving instruction clearly improved searching skills and maintained those skills six-months after completion of the study. Their attitudes toward searching also demonstrated positive changes. These outcomes support Lindberg's recommendation that including MEDLINE training in the curriculum could foster in students the attitudes and information-management skills necessary for them to become practicing physicians who are both motivated and able to access computer-based information [10]. To validate the outcomes of this RCT, similar real-time studies should be conducted.

APPENDIX A

Final information survey

Neonatal intensive care unit

APPENDIX B

Search strategy analysis tool

Search success or failure analysis