Abstract
Objectives:
Constructing an answerable question and effectively searching the medical literature are key steps in practicing evidence-based medicine (EBM). This study aimed to identify the effectiveness of delivering a single workshop in EBM literature searching skills to medical students entering their first clinical years of study.
Methods:
A randomized controlled trial was conducted with third-year undergraduate medical students. Participants were randomized to participate in a formal workshop in EBM literature searching skills, with EBM literature searching skills and perceived competency in EBM measured at one-week post-intervention via the Fresno tool and Clinical Effectiveness and Evidence-Based Practice Questionnaire.
Results:
A total of 121 participants were enrolled in the study, with 97 followed-up post-intervention. There was no statistical mean difference in EBM literature searching skills between the 2 groups (mean difference = 0.007 (P = 0.99)). Students attending the EBM workshop were significantly more confident in their ability to construct clinical questions and had greater perceived awareness of information resources.
Conclusions:
A single EBM workshop did not result in statistically significant changes in literature searching skills. Teaching and reinforcing EBM literature searching skills during both preclinical and clinical years may result in increased student confidence, which may facilitate student use of EBM skills as future clinicians.
Highlights.
A single workshop did not improve the medical literature searching skills of third-year medical undergraduate students.
Students taking a workshop were more confident in constructing answerable clinical questions and identifying information gaps.
Student confidence in writing a clinical question, identifying information sources, and performing a literature search was high six-months post-workshop.
Implications.
Students' perceptions of competency may not be accurate reflections of their skills.
Workshops may be valuable for the increase in confidence they provide.
INTRODUCTION
Competency in evidence-based medicine (EBM) provides clinicians with the ability to identify, evaluate, and integrate evidence into clinical decision making. Two of the five critical steps in achieving competency in EBM are to (1) construct an answerable question from the clinical environment and (2) effectively and efficiently search the medical literature to identify the best available evidence to answer the question [1]. EBM users must be proficient in these skills before evidence can be appraised (step 3), integrated into clinical practice (step 4), and evaluated (step 5).
Various training modules and courses in EBM are now commonly implemented in medical schools worldwide [2]. The aim of such EBM programs is to provide an integration of knowledge, cognitive skills, and behaviour that promotes lifelong learning for future medical graduates [3]. There is a small, but growing evidence base evaluating how to best educate medical students and clinicians in the principles of EBM [3, 4]. The limitation of the current evidence base is that majority of these studies have focused on evaluating critical appraisal skills, with few focused on training medical undergraduates in the critical steps of constructing an answerable question and effectively searching the literature. Those studies that have been published report an improvement in the EBM skills of undergraduate medical students using a variety of interventions [5–7].
A before and after study of sixty third-year medical students in 2000 reported improvement in EBM skills following a mini-course in EBM, consisting of four sessions about writing clinical questions, searching MEDLINE, appraising articles critically, and applying evidence [5]. A 2005 non-randomized controlled trial reported the positive impact of a single MEDLINE workshop delivered to fourth-year medical students [6]. Students were provided with a clinical scenario and asked to develop a search strategy, which was assessed by a librarian using a search strategy analysis instrument. Students who attended the workshop produced higher quality search strategies, compared to those who did not attend the workshop. Similarly, a 1998 randomized controlled trial (RCT) allocating first-year medical students to a single training session on formulating questions and searching also identified improvement in question design and search skills for students receiving the intervention [7].
Although these various studies have identified the positive impact of EBM workshops on students, none have identified when specific EBM skills, such as effectively searching the medical literature, should be taught—be it during a medical student's preclinical or clinical years of study. The level of “clinical maturity” of students can affect their perception of the importance and uptake of EBM principles in practice. A study of junior doctors' (postgraduate trainees in obstetrics and gynaecology) knowledge of and beliefs in EBM identified a belief that EBM was an essential skill relevant to their clinical practice, despite few having partaken in formal training in the principles of EBM [8]. Conversely, first-year medical students who have not been exposed to the clinical environment have been reported to perceive EBM as a static discipline, not relevant to clinical medicine [9].
Many medical courses have adopted the spiral curriculum, in which specific topics, skills, themes, or concepts are continuously revisited throughout the curriculum [10]. This spiral approach to learning aims to revisit past experience and promote deeper learning by building on existing knowledge. Obtaining a high level of knowledge and mastery of EBM skills requires a combination of formal education and application of these principles in a clinical context.
This RCT aimed to determine the effectiveness of delivering a single formal workshop in EBM literature searching skills to undergraduate medical students who had received prior training in EBM. The specific objectives of the study were to:
compare the EBM literature searching skills of medical students who participate in a single formal workshop during their first clinical year of studies, compared to medical students who have only received informal EBM training during their preclinical years of study; and
determine medical students' self-perceived competency in EBM literature searching skills.
METHODS
Study design and setting
A single-centre, single-blinded RCT with intention-to-treat analysis was performed with third-year medical students undertaking the undergraduate bachelor of medicine/bachelor of surgery (MBBS) degree at Monash University, a five-year undergraduate course. Students spend the first two years based at the university, outside of the clinical environment. In these first two preclinical years, students undertake core subjects in anatomy, physiology, and pharmacology. During the preclinical years of the Monash MBBS degree, all students also participate in several sessions about medical information sources, using the population, intervention, comparison, outcomes (PICO) framework to write answerable clinical questions and searching the medical literature. During years three to five of the course, students are placed in a clinical learning environment. Students spend their entire third year of the degree at one clinical site but rotate between sites for the remainder of their clinical years. The Monash MBBS undergraduate degree, and by extension the students, are comparable to similar MBBS degrees in Australia and internationally that train undergraduate students in the principles of EBM across preclinical and clinical years of the degree [11].
Recruitment
Third-year medical students were recruited from three teaching hospitals associated with the course (Monash Medical Centre, Dandenong, and Casey) from the Southern Health network of hospitals. To meet eligibility, participants were required to be a third-year Monash MBBS student at the time of the study. Students who were unwilling to participate in the study or did not wish to provide consent were excluded from the study.
Randomization
Participants were randomly assigned independently by the Southern Health clinical site administrator by block randomization to either the intervention or control groups (Figure 1). A computer random number generator was used to generate a randomization list in blocks of four.
Figure 1.
Flow diagram of randomized controlled trial
Intervention
Students randomized to the intervention participated in an EBM literature searching skills (EBM-LSS) workshop. The EBM-LSS workshop was two hours in duration and was delivered in the training room of the Hargrave-Andrew Library at the Clayton campus of Monash University. The EBM-LSS workshop began with a formal presentation by the subject librarian, which covered (i) how to construct an answerable question from the clinical environment, (ii) major sources of medical information, and (iii) how to effectively and efficiently search the medical literature to identify the best available evidence to answer the question.
The studies, syntheses, synopses, summaries, and systems (5S) model was used to compare and contrast the content and structure of the medical information sources and to highlight approaches to searching them effectively [12]. Students then completed an interactive, computer-based searching session and concluded the workshop by completing self-directed learning tasks (with the subject librarian providing support when requested).
Control
Students allocated to the control group did not attend the EBM-LSS workshop during the study period. These students had the opportunity to attend the EBM-LSS workshop once the study had concluded (i.e., once competency in EBM literature searching skills and self-perceived competency in EBM had been assessed).
Outcome measures
The primary outcome to be measured in this study was competency in EBM literature searching skills, which was measured by using the previously validated Fresno tool [13]. The Fresno tool provides users with a clinical scenario and a series of open-ended questions relating to the five EBM steps. The Fresno tool evaluates skills in question formulation, knowledge of information sources, choice of study design, search strategy development, and relevance of selected evidence. Rather than using the clinical scenario in the Fresno tool, students were presented with the following scenario to avoid recall bias from previous exposure to the Fresno tool:
Your patient is a 50 year old man with a family history of colorectal cancer. The patient recently received a pamphlet in the post advertising the National Bowel Cancer Screening program. You explain to your patient that the Screening program uses the Faecal Occult Blood Test (FOBT). Whilst understanding that no test is 100% accurate, the patient wants to know whether it would be useful for him to undertake the FOBT—will it save his life?
Based on this scenario, students were assessed for their ability to construct a clinical question (step 1 of the EBM process) and effectively search the medical literature to identify the best evidence to answer the clinical question (step 2 of the EBM process). The maximum score attainable for the Fresno tool was adjusted to reflect that only the first 2 steps of the EBM process were being evaluated in this study. Therefore, the maximum overall score possible was 29, with the maximum subcomponent scores of writing a clinical question, identifying information sources, identifying appropriate study types, and performing an effective literature search being 3, 6, 12, and 8 points, respectively.
Student self-perceived competency in EBM literature searching skills was assessed using the previously validated Clinical Effectiveness and Evidence-Based Practice Questionnaire (EBPQ) [14]. The EBPQ is a self-reported measure of implementation of EBM in the clinical environment. It can be used to evaluate educational programs as questions relate to the measurement of the practice, attitudes, and knowledge/skills associated with EBM. Only the knowledge/skills domain of the EBPQ was utilized in this study, as measuring students' practice of and attitudes toward EBM would be unnecessary since they do not practice medicine in a clinical setting.
These outcomes were measured across both intervention and control groups at 1 week post-implementation of the intervention. Control students were given the opportunity to attend the EBM-LSS workshop at the conclusion of the initial study period (study phase 1) to ensure equivalence among the student cohort. All students who attended the EBM-LSS workshop completed a questionnaire, six months after attending the workshop (study phase 2) that evaluated their confidence in (i) writing a clinical question, (ii) identifying information sources, (iii) identifying appropriate study designs to answer a clinical question, and (iv) performing a literature search efficiently. Students were asked to evaluate their confidence on a 5-point Likert scale (1 = poor, 5 = excellent). Students were also asked to reflect whether the EBM-LSS workshop improved their EBM competency on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree).
Blinding
Blinding of investigators and participants was not possible as the subject librarian and students were aware of their allocation. The outcome assessor and data analyst were blinded to the allocation.
Analyses
Sample size calculation
A previous study, which implemented the Fresno test to assess searching skills, was referred to when estimating the sample size for this trial [15]. This previous study identified a mean difference of 13 points on the Fresno test to be statistically significant in identifying competency between groups in effective EBM literature searching. A mean difference of 13 points, with a standard deviation of 10, is meaningful to discriminate between “novice” and “expert” users of EBM principles [13]. It was determined that with a power of 90%, alpha of 0.05, the required sample size per group was 21, for a total sample size of 42 participants.
Analyses
Data were analyzed using the principle of intention-to-treat. Mean difference in EBM literature searching skills competency between the intervention and control groups, as determined by the Fresno tool, were explored using a Student's t-test. Descriptive statistics were used to characterize participants' self-perceived competency in EBM literature searching skills at one week and six months post-intervention.
Ethical approval for this study was granted by the Monash University Human Research Ethics Committee.
RESULTS
A total of 121 of 134 eligible students were enrolled in the study, with 62 participants (51.2%) allocated to the intervention and 59 participants (48.8%) allocated to the control group (study phase 1). There was no statistical mean difference in EBM literature searching skills between the participants who attended the formal workshop and participants who did not (mean difference = 0.007 (P = 0.99). There was no statistically significant difference between the 2 groups with respect to writing a focused clinical question (step 1 of the EBM process), identifying information sources, or performing a structured literature search (step 2 of the EBM process) (Table 1). Students who received the EBM-LSS workshop were more likely to choose a systematic review (as it is perceived to offer the best level of evidence) to answer the question in the clinical scenario; however, this was not a significantly statistical difference between groups (RR = 1.26, 95% CI 0.91, 1.74).
Table 1.
Student competency scores across evidence-based medicine (EBM) literature searching skills as identified by the Fresno tool
Participants in the EBM-LSS workshop were significantly more confident in completing step 1 of the EBM process (converting a clinical scenario into a question) and had a greater perceived awareness of information resources (step 2 of the EBM process) (Table 2). Students in the EBM-LSS workshop also had a greater confidence in identifying gaps in their professional practice.
Table 2.
Student self-perceived competency across EBM literature searching skills as identified by the Clinical Effectiveness and Evidence-Based Practice Questionnaire
A total of 97 (80.2%) participants were followed-up post-intervention (60 in the intervention group and 37 in the control group) (study phase 2). Twenty-four participants (2 intervention and 22 control group) declined to complete the follow-up survey. The remaining students completed a 6-month post-workshop follow-up survey, which assessed their confidence in EBM skills such as preparing an answerable clinical question and literature searching skills. More than 75% of respondents believed that their EBM literature searching skills were very good or excellent across the 4 measured competencies (Table 3). This contrasts with the scores obtained from the Fresno tool 6 months earlier, which indicated that students in both groups exhibited “novice” rather than “intermediate” or “expert” skill in EBM literature searching. Despite the absence of any measureable improvement after the EBM-LSS workshop, more than 70% of participants (both intervention and control groups) agreed that the EBM-LSS workshop was effective in improving their EBM literature searching skills (Table 4).
Table 3.
Student confidence in EBM literature searching at 6 month after the EBM literature searching skills (EBM-LSS) workshop attendance (n = 97)
Table 4.
Student perceived worth of the EBM-LSS workshop at 6 month post-EBM-LSS workshop attendance (n = 97)
DISCUSSION
This study indicated that the delivery of a single EBM-LSS workshop to third-year undergraduate medical students did not significantly improve student ability to perform the first two steps of the EBM process. Students who had been randomized to an EBM-LSS workshop had greater confidence in their ability to write answerable clinical questions and in their awareness of relevant information sources, compared to students who did not receive the EBM-LSS workshop. Although no data were collected before the implementation of the EBM-LSS workshop, student confidence in performing critical EBM literature searching was significantly higher immediately after the intervention group took the workshop in the intervention group than in the controls. However, this self-perceived increase in confidence should be viewed with caution, since it does not necessarily correlate with a higher competency in practicing EBM [16].
A recently published study documented the effectiveness of combining an online educational intervention with clinical rotations in a cohort of third-year undergraduate medical students [17] and involved extended, faculty mentored instruction throughout the year. The authors concluded that there was a significant improvement in EBM skills after the educational intervention (as measured by the Fresno tool). The results of our study might differ from this and previously published studies for a variety of reasons. Our study evaluated the impact of a single workshop, while other studies evaluated the impact of a series of workshops [5, 17]. Delivering a series of workshops, rather than a single workshop, would promote reinforcement and development of student skills over a period of time and, therefore, be more likely to result in significant improvements.
The EBM-LSS workshop in our study did not focus solely on MEDLINE, as a number of other studies reported [5, 6], but highlighted a range of different information sources. Focusing on more than one source limits the time available to explore specific sources in depth, which may also have limited the potential for significant improvements in student literature searching skills. However, this approach was used to highlight to students that it is not always necessary to start their search with MEDLINE, as they may be able to locate reliable evidence using another source, such as the Cochrane Library. The different approaches to practicing EBM—the “doing,” “using,” and “replicating” modes [18]—were also highlighted as this may influence which source they choose to search first.
Previous research has indicated that clinical maturity, perceived relevance of EBM in the clinical environment, and continued practice of evidence-based skills in this context may influence a medical student's competence in EBM skills [8, 9, 19, 20]. Whilst there is no significant difference in EBM skills between the intervention and control groups in this study, the results indicate that students who are taught these skills during the clinical years of their study demonstrate greater self-confidence in adopting the first two EBM steps. This increased confidence may be attributed to observing how practicing clinicians use these skills in the clinical environment and positively influence the manner in which these students implement EBM skills during their clinical years of study and beyond [21].
Integrating a formal workshop during the clinical years of study provides students with opportunities to practice searching for and identifying evidence that can be directly related to the clinical context. It is important that students are aware that they can adjust their approach to suit the clinical environment and do not always have to carry out all five steps of EBM, the “doing” mode [18]. They may also incorporate evidence in a “using” or “replicating” mode, in which one or more of the five steps are not carried out by them directly [18].
Limitations
There was no assessment of EBM literature searching skills before and after the intervention. This is important to note as the authors were not able to measure the impact of previous training programs implemented during the preclinical years of student training and the potential confounding effect on this trial. Given that it was not possible to blind the students in this study, it is possible that contamination between the two arms of the study (with students sharing knowledge about the EBM-LSS workshop intentionally or indirectly) might have been a confounding factor in this study. Whilst it did not affect the power of the study, it must also be noted that twenty-two students from the control arm did not complete the outcome assessment.
The Fresno tool evaluates skills in question formulation, knowledge of information sources, choice of study design, search strategy development, and relevance of selected evidence. Students were not directed to use MEDLINE but were given the choice to select a relevant source. The Fresno tool assigns an “excellent” score if three PICO elements are used in the search strategy. However, for the scenario that the students completed in the intervention, a relevant Cochrane systematic review was able to be located in the Cochrane Library with a basic search using only one PICO element. This may have resulted in students obtaining low scores in the “performing a literature search” component of the Fresno tool, even though they might have located a Cochrane systematic review.
It can be argued that the most important outcome is to ensure that students can locate and identify the best evidence; how many PICO elements they include or where they look may be secondary to this. A clinical scenario in which there was limited evidence (i.e., no Cochrane systematic review) might have been better at discriminating skill levels between the students. It should also be noted that the authors used adapted versions of the Fresno and EBPQ tools, both of which have been validated on practicing health professionals, rather than undergraduate medical students.
CONCLUSIONS
Medical students may develop effective, if not efficient, skills in searching the medical literature and finding evidence during the preclinical years. However, as students transfer into the clinical years of their medical education, it is valuable to reinforce and develop these skills further in the clinical environment. Increased confidence may be an important factor in students continuing to search for and use evidence effectively and efficiently in their practice in the future.
Acknowledgments
The authors thank the involvement of the third-year Monash University MBBS students for their participation in this study. The authors also thank Dominic Upton and Penelope Upton for the use of the EBPQ.
REFERENCES
- 1.Straus S, Glasziou P, Richardson W, Haynes B. Evidence-based medicine: how to practice and teach it. Edinburgh, UK: Churchill Livingstone, Elsevier; 2011. [Google Scholar]
- 2.Dinkevich E, Marksinson A, Ahsan S, Lawrence B. Effect of a brief intervention on evidence-based medicine skills of pediatric residents. BMC Med Educ. 2006;6:1. doi: 10.1186/1472-6920-6-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Ilic D. Teaching evidence based practice: perspectives from the undergraduate and post-graduate viewpoint. Ann Acad Med Singapore. 2009;38(6):559–63. [PubMed] [Google Scholar]
- 4.Coomarasamy A, Khan K. What is the evidence that postgraduate teaching in evidence based medicine changes anything. a systematic review. BMJ. 2004 Oct 30;329(7473):1017. doi: 10.1136/bmj.329.7473.1017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Ghali W, Saitz R, Eskew A, Gupta M, Quan H, Hershman W. Successful teaching in evidence-based medicine. Med Educ. 2000 Jan;34(1):18–22. doi: 10.1046/j.1365-2923.2000.00402.x. [DOI] [PubMed] [Google Scholar]
- 6.Gruppen L, Rana G, Arndt T. A controlled comparison study of the efficacy of training medical students in evidence-based medicine literature searching skills. Acad Med. 2005 Oct;80(10):940–4. doi: 10.1097/00001888-200510000-00014. [DOI] [PubMed] [Google Scholar]
- 7.Rosenberg W, Deeks J, Lusher A, Snowball R, Dooley G, Sackett D. Improving searching skills and evidence retrieval. J R Coll Physicians Lond. 1998 Nov–Dec;32(6):557–63. [PMC free article] [PubMed] [Google Scholar]
- 8.Hadley J, Wall D, Khan K. Learning needs analysis to guide teaching evidence-based medicine: knowledge and beliefs amongst trainees from various specialities. BMC Med Educ. 2007;7:11. doi: 10.1186/1472-6920-7-11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Astin J, Jenkins T, Moore L. Medical students' perspective on the teaching of medical statistics in the undergraduate medical curriculum. Stat Med. 2002 Apr 15;21(7):1003–6. doi: 10.1002/sim.1132. [DOI] [PubMed] [Google Scholar]
- 10.Bruner J. The process of education. Cambridge, MA: Harvard University Press; 1960. [Google Scholar]
- 11.Finkel M, Brown H, Gerber L, Supino P. Teaching evidence-based medicine to medical students. Med Teach. 2003 Mar;25(2):202–9. doi: 10.1080/0142159031000092634. [DOI] [PubMed] [Google Scholar]
- 12.Haynes R.B. Of studies, syntheses, synopses, summaries, and systems: the “5S” evolution of information services for evidence-based health care decisions. ACP J Club. 2006 Nov–Dec;145:A8. [PubMed] [Google Scholar]
- 13.Ramos K, Schafer S, Tracz S. Validation of the Fresno test of competence in evidence based medicine. BMJ. 2003 Feb 8;326(7384):319. doi: 10.1136/bmj.326.7384.319. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Upton D, Upton P. Development of an evidence-based practice questionnaire for nurses. J Adv Nurs. 2006 Feb;53(4):454–8. doi: 10.1111/j.1365-2648.2006.03739.x. [DOI] [PubMed] [Google Scholar]
- 15.Kim S, Willett L, Murphy D, O'Rourke K, Sharma R, Shea J. Impact of an evidence-based medicine curriculum on resident use of electronic resources: a randomized controlled study. J Gen Intern Med. 2008 Nov;23(11):1804–8. doi: 10.1007/s11606-008-0766-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Lai N, Teng C. Self-perceived competence correlates poorly with objectively measured competence in evidence based medicine among medical students. BMC Med Educ. 2011;11:25. doi: 10.1186/1472-6920-11-25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Aronoff S.C, Evans B, Fleece D, Lyons P, Kaplan L, Rojos R. Integrating evidence based medicine into undergraduate medical education: combining online instruction with clinical clerkships. Teach Learn Med. 2010 Jul;22(3):219–23. doi: 10.1080/10401334.2010.488460. [DOI] [PubMed] [Google Scholar]
- 18.Straus S.E, Green M.L, Bell D.S, Badgett R, Davis D, Gerrity M, Ortiz E, Shaneyfelt T.M, Whelan C, Mangrulkar R. Evaluating the teaching of evidence based medicine: conceptual framework. BMJ. 2004 Oct 30;329(7473):1029–32. doi: 10.1136/bmj.329.7473.1029. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Bradley P, Oterholt C, Nordheim L, Bjorndal A. Medical students' and tutors' experiences of directed and self-directed learning programs in evidence-based medicine: a qualitative evaluation accompanying a randomized controlled trial. Eval Rev. 2005 Apr;29(2):149–77. doi: 10.1177/0193841X04269085. [DOI] [PubMed] [Google Scholar]
- 20.Dorsch J.L, Aiyer M.K, Meyer L.E. Impact of an evidence-based medicine curriculum on medical students' attitudes and skills. J Med Lib Assoc. 2004 Oct;92(4):397–406. [PMC free article] [PubMed] [Google Scholar]
- 21.Ilic D, Forbes K. Undergraduate medical student perceptions and use of evidence based medicine: a qualitative study. BMC Med Educ. 2010;10:58. doi: 10.1186/1472-6920-10-58. [DOI] [PMC free article] [PubMed] [Google Scholar]