How are medical students trained to locate biomedical information to practice evidence-based medicine? a review of the 2007–2012 literature

Abstract

Objectives:

This study describes how information retrieval skills are taught in evidence-based medicine (EBM) at the undergraduate medical education (UGME) level.

Methods:

The authors systematically searched MEDLINE, Scopus, Educational Resource Information Center, Web of Science, and Evidence-Based Medicine Reviews for English-language articles published between 2007 and 2012 describing information retrieval training to support EBM. Data on learning environment, frequency of training, learner characteristics, resources and information skills taught, teaching modalities, and instructor roles were compiled and analyzed.

Results:

Twelve studies were identified for analysis. Studies were set in the United States (9), Australia (1), the Czech Republic (1), and Iran (1). Most trainings (7) featured multiple sessions with trainings offered to preclinical students (5) and clinical students (6). A single study described a longitudinal training experience. A variety of information resources were introduced, including PubMed, DynaMed, UpToDate, and AccessMedicine. The majority of the interventions (10) were classified as interactive teaching sessions in classroom settings. Librarians played major and collaborative roles with physicians in teaching and designing training. Unfortunately, few studies provided details of information skills activities or evaluations, making them difficult to evaluate and replicate.

Conclusions:

This study reviewed the literature and characterized how EBM search skills are taught in UGME. Details are provided on learning environment, frequency of training, level of learners, resources and skills trained, and instructor roles.

Implications:

The results suggest a number of steps that librarians can take to improve information skills training including using a longitudinal approach, integrating consumer health resources, and developing robust assessments.

INTRODUCTION

Evidence-based medicine (EBM) is the integration of best available research evidence, patient preferences, and clinical acumen to make optimal decisions in patient care 1. The practice of EBM has been associated with improved patient outcomes and physician lifelong learning 2. Physicians practice EBM by recognizing and articulating gaps in their knowledge, seeking information to fill their knowledge gaps, appraising the evidence, and applying it to the care of their patients 3. For over twenty years, librarians have supported EBM by offering information skills training for medical students 4. Unfortunately, EBM is suboptimally practiced due to a variety of factors, including challenges related to searching and retrieving biomedical literature 5–8. The suboptimal practice of EBM poses a risk to patient care, and the difficulties related to information skills suggest a need to improve teaching in this domain 2. As a precursor to improving information retrieval skills, more needs to be known about how these important skills are taught.

Literature reviews on information skills training have focused on the evaluation of the training. For example, in 2003, Brettle reviewed the literature attempting to answer the question of whether or not information skills training improves search skills 9. Unfortunately, due to the lack of rigor of the published literature at that time, she was unable to draw a definitive conclusion. A decade later, Just readdressed this topic across all levels of medical education, with a focus on evaluation instruments and measurement approaches, concluding that training increased at least 1 measure of information searching skills, such as the formulation of search strategies and database selection 4. While both reviews are valuable contributions to medical librarianship, these reviews do not delve into the critical question of: How are medical students trained to locate and manage biomedical information to practice EBM? Lacking this knowledge impedes medical librarians' abilities to design information skills curricula and, if desired, to replicate the efforts of colleagues who are engaged in information training. Another recent literature review identified and described 20 educational interventions for teaching all of the steps of EBM, with the goal of providing educators examples to utilize in their own teaching practice 10. However, although 90% of the interventions in that review included instruction related to information search and retrieval education, the review did not provide details as to the content or methods of training specific to that particular skill.

The current study, therefore, aims to systematically review the literature specific to that part of EBM training that relates to information search and retrieval, and to identify and describe how these critical skills are taught in EBM curricula at the undergraduate medical education (UGME) level. These findings will then be considered in light of the broader medical education literature.

METHODS

The PRISMA statement and guidelines for identifying articles in medical education informed the execution of this systematic review 11. To begin, author Kung searched MEDLINE (via PubMed), Scopus, Web of Science, Education Resources Information Center (ERIC), the Cochrane Library, and Google Scholar using combinations of keywords and, where appropriate, controlled vocabulary terms. Search terms included: evidence based medicine; evidence based practice; EBM; EBP; information storage and retrieval; information literacy; information seeking; literature search; database search; PubMed; MEDLINE; education, medical; students, medical; undergraduate medical education; medical students; UGME; and UME. The initial searches were run on April 15, 2013, and limited to English language articles published from 2007 to 2012. To ensure a comprehensive retrieval of articles, the searches were rerun on June 1, 2013, because in some cases, inclusion of citations in databases can be delayed. The full text of all search strategies is available in the online only appendix. To supplement the database searches, Kung also manually searched the reference lists of retrieved articles and Academic Medicine, BMC Medical Education, the Journal of the Medical Library Association, Medical Education, Medical Reference Services Quarterly, and Medical Teacher.

The authors considered articles for inclusion if they (1) involved undergraduate medical students, (2) described information retrieval skills training to improve EBM skills, and (3) were published from 2007 to 2012. Due to the many changes that have taken place in the digital landscape recently, the authors excluded articles more than five years old (published prior to 2007). We focused on UGME to identify how students are first taught EBM, so we excluded articles featuring other student groups such as residents or other health professional students, as well as information retrieval training that was not intended to support EBM. We also excluded training that focused broadly on EBM training and only mentioned the inclusion of information skills training, without providing any details. In cases where an author published more than one paper on a single intervention, we included the article that contained the most details of the intervention.

Kung performed a title abstract review of all retrieved articles and identified those that seemed to meet the inclusion criteria. Both authors then independently reviewed the full text of these articles to determine fitness for inclusion. Additional articles were identified for exclusion because they did not focus on searching, lacked details of searching, did not address searching, or failed to focus on training. Following the review, inclusion was determined via conference call. We discussed discrepancies until consensus was reached.

Data from each included article were extracted using a modified version of the Best Evidence in Medical Education (BEME) data-abstraction tool for systematic reviews 12. Using this tool, we extracted data on skill sets covered, educational settings used, learner levels present, and the roles of instructors. Modifications to the tool enabled extraction of information specific to information retrieval skills and EBM, including the addition of Khan and Coomarasamy's hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine 13. The use of this hierarchy, which categorizes teaching in terms of what its interactive and didactic qualities were and whether or not it was clinically integrated, provided a framework for the characterization of teaching methods used. We both independently extracted data from each article and, via conference call, compared findings and resolved any discrepancies through discussion.

RESULTS

The search process initially identified 621 articles (Figure 1). Following the removal of duplicate articles, review of titles and abstracts of the 433 articles left 23 articles that appeared to meet the inclusion criteria. After full-text review, 12 articles remained for analysis 14–25 (Table 1).

Figure 1.

Search process

Table 1.

Studies identified for analysis

Nine of the twelve (75%) trainings were conducted in the United States 14, 15, 17–20, 22, 23, 25. Trainings were also held in Australia (1, 8%) 16, the Czech Republic (1, 8%) 21, and Iran (1, 8%) 24. All trainings were held at a single institution.

Eight (67%) of the trainings took place solely in classroom settings 14, 16–20, 22, 24; 2 (17%) were held completely online 23, 25. We defined classrooms as traditional classrooms, lectures, computer labs, and any other physical space that enables teaching medical students in a formalized situation. Two (17%) of the trainings were a hybrid of online and classroom sessions 15, 21. A single training took place in a clinical setting, which we identified as a hospital ward 21.

The frequency of trainings varied. Trainings were characterized as a single session training if the students, in-person or online, engaged in only a single session. Sessions in which students received training in more than 1 session were considered multiple session trainings. Five (42%) of the trainings were considered single-session training 16, 18, 22, 23, 25, and 7 (58%) were multi-session trainings 14, 15, 17, 19–21, 24. For multi-session trainings, the average number of sessions was 6, ranging between 2 and 8 sessions. Morley and Hendrix presented 5 sessions in what was considered an elective course 19. Chen also presented 5 sessions that were integrated into an existing problem-based learning curriculum 14. Two studies described sessions that were integrated into clerkships 17, 20, while Taheri's study presented the greatest number of sessions, consisting of a 4-day workshop, including 4 2-hour lectures and 4 1-hour small group sessions 24. The remaining 2 studies 15, 21 did not specify the number of sessions offered.

Five trainings (42%) targeted preclinical students 14, 18, 19, 23, 25. Six (50%) focused exclusively on clinical students 16, 17, 20–22, 24. A single study described trainings spanning both the preclinical and clinical years and was characterized by its authors as a longitudinal program 15.

The majority (83%) of trainings were integrated into the medical school curriculum at their institutions 14–18, 20–23, 25. At the preclerkship level, information skills sessions were often affiliated with problem-based learning courses 14, 18, 23 and preclerkship transition courses 25. Six trainings were integrated into clinical clerkships 15–17, 20–22. One study featured an elective course 19, which was offered to both third- and fourth-year students. In a single instance, it was unclear if the session was integrated into the medical school curriculum 24.

Sixty-seven percent of interventions indicated that they trained learners to search PubMed 15, 18–21, 23–25. In three interventions 14, 16, 17, MEDLINE, a subset of PubMed, was mentioned as a distinct information resource for which students received training. In one instance, MEDLINE was accessed via Ovid 17, whereas in the two other interventions, it is unclear if MEDLINE was introduced via the PubMed interface or via a subscription service, such as Ovid or EBSCO.

In addition to PubMed, a spectrum of information resource types was introduced, including UpToDate (25%), AccessMedicine (17%), DynaMed (17%), Google (17%), Cochrane EBM Reviews (8%), DxPlain (8%), National Guideline Clearinghouse (8%), and ACP Journal Club (8%). These resources represent a wide variety of information resources, including databases, electronic textbooks, decision support tools, and evidence summaries. A single study also introduced MedlinePlus 17, a consumer health resource. Two interventions also presented the reference management tools RefWorks 19 and MyNCBI 15.

The interventions addressed a variety of information retrieval skills and information resources. Five interventions (42%) emphasized the importance of constructing a well-developed search and selecting multiple information resources to locate evidence 15, 16, 19, 22, 24. Four interventions also introduced the concept of patient/population, intervention, comparison, and outcomes (PICO) 15, 17, 20, 25, which is a mnemonic that guides the formulation of searchable clinical questions 3. Specific to PubMed, 3 interventions 20, 23, 25 (25%) instructed students on how to search using Medical Subject Headings (MeSH) and PubMed limits/filters 20, 25. Using Khan and Coomarasamy's hierarchy (Table 2) 13, we classified 10 of the interventions as Level 2a 14, 15, 18–25. The remaining 2 studies 16, 17 were described as Level 3. None of the interventions were classified as Level 1 or Level 2b. Details on the overall evaluation of the training sessions were scant, limiting our ability to draw conclusions as to the efficacy of the training interventions. In 3 interventions (25%), the instructors conducted pretest evaluations 19, 22, 24, and 7 interventions (58%) included post-session assessments. Post-session assessment included locally created multiple-choice quizzes 15, 23, 24, oral presentations 19, 21, and written reports 14, 22.

Table 2.

Khan and Coomarasamy's framework

Librarians and clinicians co-taught 4 (33%) of the interventions 14, 15, 18, 21. Alternatively, librarians independently taught 5 of the interventions 16, 17, 20, 23, 25, while clinicians were primary instructors in 2 trainings 22, 24. Beyond instruction, librarians played other significant roles. For the 2 trainings that were administered online, librarians participated in the design of these modules. Eight (67%) interventions featured librarians evaluating students, and a single intervention reported a clinician and librarian collaboratively evaluating the students' work 21. Table 3 summarizes the key findings.

Table 3.

Results summary

DISCUSSION

EBM is an essential skill that enables physicians to provide patient-centered care and to practice medicine as lifelong learners 2. The ability to locate best evidence is an essential component of EBM and is frequently included as a component of UGME curriculum 10. To characterize how information search skills are taught to support EBM in UGME, we systematically reviewed the published literature. Our review resulted in the analysis of twelve articles with findings in relation to the literature on learning environment, frequency of training, timing of training, resources and skills trained, teaching modalities, and instructor roles, including librarians.

Learning environment

The context of training impacts how students learn and eventually practice medicine 26. We identified information skills training sessions in a variety of learning settings or contexts. Seven of the identified trainings were presented in classroom settings, and two were set in clinical environments. Research has demonstrated that classroom-based skills training, which occurs outside the real-life context of the skill, can hamper the transfer of the skill into authentic settings and can lead to de-contextualized knowledge 27. This indicates that classroom-based EBM training may not be an ideal practice. However, it is not always efficacious for training to occur in a clinical setting due to logistics and patient safety issues 28. To circumvent these issues, other groups, such as surgeons, have used medical simulation to provide safe and realistic learning environments that mirror the clinical environment 29. We did not detect the use of simulation of clinical settings in our findings and suggest that medical educators and librarians consider this modality for future information skills training.

Two trainings were presented completely online, and two trainings were a hybrid of online and classroom training. Online EBM instruction has been demonstrated to be as efficacious as face-to-face EBM training 30. More specific to information retrieval skills, a randomized trial that was focused on delivering online MEDLINE training concluded that information skills training delivered online can improve students' information retrieval skills 31. The use of online learning environments aligns with recent calls to action for medical educators to further integrate online media into their learning activities 32.

Frequency of training

Half of the analyzed trainings were presented as single sessions. Information retrieval training delivered over a single session is suboptimal for learning 33, 34. To optimize learning, learners require sufficient opportunities to practice a skill and receive guidance for learning benefits to accumulate 35. Therefore, EBM educators should consider expanding the number of information retrieval skills sessions offered to ensure that learners are afforded adequate practice opportunities. In addition to multiple learning opportunities, educators should consider the timing of sessions. In medical education, educators have adopted the model of distributed practice, which refers to periods of training that are interspersed with rest periods rather than one continuous block of practice or “massed practice” 36. In practice, Moulton found that if residents experience training schedules that allow for distributed practice, they retain and transfer skills better 37. EBM educators should consider expanding the scope of training to include additional sessions whenever possible.

Timing of training

Although there seems to be agreement that EBM training, including information skills training, should ideally be offered across all years of the medical school curriculum 38, 39, we identified only a single longitudinal intervention spanning both the preclinical and clinical years 15. The remaining articles described five preclinical information retrieval trainings and six clinical-level trainings. This finding aligns with the lack of consensus as to when to teach EBM 25, 40. For example, some studies concluded that EBM instruction is ideal during the first two years of medical school (preclinical) 40, 41, while others suggested EBM training should be provided in years three and four of the curriculum, when students are commonly enrolled in clinical clerkships 4, 42. Due to lack of consensus and the benefits of continual practice through learning activities 43, it seems advisable that, when possible, medical educators consider teaching EBM, including information retrieval skills, using a longitudinal approach.

Resources and skills trained

The prevalence of PubMed as the most commonly used database in information retrieval training aligns with Just's and Brettle's earlier findings 4, 9 and with practices observed in clinical settings 44. Also in agreement with previous research on the varied use of information resources in practice, we noted the inclusion of additional information resources beyond PubMed 45, such as DynaMed, ACP Journal Club, and UpToDate. Yet, consistency was limited as to which resources the trainings featured. For example, a single training featured UpToDate and PubMed 19, whereas another training included Google Scholar and MedlinePlus but not UpToDate 17. With the exception of PubMed, this demonstrates a lack of consensus on which information resources to feature in training. This finding has repercussions for graduate medical education (GME). As GME programs recruit residents from a variety of medical schools, each featuring their own approaches to information skills training programs, it may be difficult to know which information resources are familiar to their residents and, therefore, difficult for medical librarians to calibrate information skills training. We suggest that librarians, in conjunction with medical educators, organize a baseline list of information resources to be introduced at the UGME level. This list would ideally focus on publicly available resources and have flexibility for institutions to add resources from their local collections as they saw fit.

EBM advocates encourage physicians to provide their patients with best evidence as a component of the EBM process and shared decision making 46. However, best evidence that is valuable to a physician, such as a meta-analyses featured in PubMed, may or may not be of value to a patient due to its complex nature. Therefore, it is notable that only a single training presented a consumer health information resource. This is a valuable missed opportunity for librarians, who have a long history of promoting consumer information 47, to introduce medical students to these important tools, which can facilitate the shared decision-making process and empower the patient. In the future, librarians should consider introducing consumer health resources into EBM training.

The identified trainings covered several information skills, including how to formulate clinical questions, use a controlled vocabulary (MeSH), and use limits 20, 23, 25. In general, these search skills should translate to other databases and information resources; however, it is notable that all of the trainings introduced these skills in the context of PubMed. Future trainings might demonstrate the use of these skills in the other resources, such as PsycINFO. Although the introduction of these skills was mentioned in the trainings, there was scant detail on how and to what level these skills were taught. In alignment with Brettle's finding, this lack of detail makes it difficult to replicate these interventions, if warranted, indicating a need for additional detail.

Teaching modalities

Students optimally learn when they are actively engaged in problem solving 48. Therefore, it is encouraging that the majority of sessions were identified as interactive, although classroom based. For example, in the preclinical years, Chen challenged learners to use their newly acquired information skills to complete written short reports that the students discussed in their problem-based learning small groups 14. Unfortunately, the majority of the analyzed articles provided scant details on the nature of the learning activities presented in the training sessions, again making them difficult to replicate. As Eldredge suggests, it is critical for librarians to carefully consider their pedagogical approaches to ensure a place in the “crowded curriculum” 49, which suggests that robust examples in the literature could be highly valuable for librarians to learn from their colleagues.

Similar to Brettle's findings from her systematic review of the literature on search skills, inadequate descriptions in the analyzed articles made it difficult to draw conclusions about the efficacy of interventions 9. In the future, EBM educators and medical librarians should consider or, if necessary, create robust assessments and provide increased details of these assessments in the published literature.

Evidence-based medicine instructors

Similar to recent findings by Dorsch and Perry, medical librarians are highly engaged in EBM in a variety of capacities, including as curricular designers and teachers of information skills 50. This is positive for both medical librarians and students, because information skills trainings tend to be well received and accepted by students when taught by librarians 21. Additionally promising are the co-teaching and educational interactions between physicians and librarians. This partnership approach demonstrates interprofessional collaboration, providing a positive role model for learners.

This study must be considered in the context of its limitations. We only studied the last five years and focused on published accounts of training, which might have excluded programs that have not been described in the literature. We also chose to focus on trainings with undergraduate medical students, although it is possible that there would be value in examining information skills training in other health professional disciplines and at other learner levels, such as the residency level. We limited our study to medical students in order to focus on their formative years in information skills training with EBM, which potentially narrowed the scope of the review. Also due to the lack of detail in the articles, we were unable to draw conclusions regarding the efficacy of interventions and, therefore, are unable to make recommendations for their adoption.

CONCLUSION

EBM is an essential skill that enables physicians to provide patient-centered care and to practice medicine as lifelong learners. To practice EBM, physicians must be able to locate best evidence, which requires information skills and, therefore, information skills training. We have systematically reviewed the literature to characterize how EBM search skills are taught in UGME by providing details on learning environment, frequency of training, level of learners, resources and skills trained, and instructor roles, including medical librarians. Our findings suggest a number of ways in which information skills training for EBM can be improved, including using a longitudinal teaching approach, addressing consumer health resources, integrating a variety of databases and online search engines, and developing a baseline list of resources to include in training sessions.