Abstract
Objective To evaluate the effects of standalone versus clinically integrated teaching in evidence based medicine on various outcomes in postgraduates.
Design Systematic review of randomised and non-randomised controlled trials and before and after comparison studies.
Data sources Medline, Embase, ERIC, Cochrane Library, DARE, HTA database, Best Evidence, BEME, and SCI.
Study selection 23 studies: four randomised trials, seven non-randomised controlled studies, and 12 before and after comparison studies. 18 studies (including two randomised trials) evaluated a standalone teaching method, and five studies (including two randomised trials) evaluated a clinically integrated teaching method.
Main outcome measures Knowledge, critical appraisal skills, attitudes, and behaviour.
Results Standalone teaching improved knowledge but not skills, attitudes, or behaviour. Clinically integrated teaching improved knowledge, skills, attitudes, and behaviour.
Conclusion Teaching of evidence based medicine should be moved from classrooms to clinical practice to achieve improvements in substantial outcomes.
Introduction
The knowledge and skills needed for critical appraisal of literature and practice of evidence based medicine (EBM) are often taught through standalone courses and workshops in classrooms away from clinical practice. An early (and now out of date) review showed that in these educational interventions, gains in knowledge were poorer among postgraduates than undergraduates.1 Without reinforcement in subsequent practice, even the modest knowledge gains from such courses are likely to deteriorate over time. Postgraduate and continuing education received in this way is unlikely to lead to any meaningful changes in clinical care. In theory, teaching and learning that is integrated into routine practice should bring greater benefits.
We examined the effects of postgraduate teaching in EBM and explored the effect of the teaching methods (whether standalone or integrated into clinical practice) on various outcomes.
Methods
We searched Medline, Embase, ERIC, Cochrane controlled trials register (CCTR), Cochrane database of systematic reviews (CDSR), database of abstracts of reviews of effects (DARE), Health Technology Assessment database (HTA), Best Evidence, Best Evidence Medical Education (BEME), and Science Citation Index (SCI) using the following search terms and their word variants: “evidence”, “critical”, “appraisal” or “journal club” combined with “AND” to “teach$”, “learn$”, “instruct$”, or “education”. We also searched reference lists of known systematic reviews.1-4 The final electronic search was conducted in April 2004.
We included studies that evaluated the effects of postgraduate EBM or critical appraisal teaching compared with a control group or baseline before teaching, using a measure of participants' learning achievements or patients' health gains as outcomes. Learning achievement was assessed separately for knowledge, critical appraisal skills, attitudes, and behaviour.
Knowledge relates to issues such as remembering materials as well as grasping the meaning—for example, defining and understanding the meaning of number needed to treat (NNT). If this knowledge can then be applied accurately to given problems this will be regarded as a gain in critical appraisal skills—for example, the ability to generate a number needed to treat when baseline risks and odds ratios are provided. Spontaneously acknowledging a need for the use of a certain piece of knowledge or skill in practice will be regarded as a change in attitude—for example, recognising without prompting the need for different NNTs for different clinical scenarios and intending to calculate the respective NNTs for different levels of risk. Finally, a change in behaviour occurs when one seeks the necessary information and applies the knowledge and skills to solve the issue in practice—for example, searching the literature, finding relevant baseline risks and odds ratios, and calculating necessary NNTs to guide clinical practice.
We excluded studies on teaching of EBM in undergraduate education. We graded the quality of the evidence in these articles as either level 1 (randomised controlled trials) or level 2 (non-randomised studies that either had a comparison with a control group or a before and after comparison without a control group). We could not use meta-analysis because of the obvious heterogeneity in features, quality, and assessment tools in individual studies. We weighted our conclusions by quality of methods.
Results
The literature search identified 42 potentially useful citations. We examined the full manuscripts of all of these citations and identified 23 articles (including one yet to be published) relevant for inclusion in our review (tables 1 and 2).5-26 Of the 19 articles that we rejected, 15 examined populations unsuitable for our review (for example, undergraduates or non-medical staff), two examined an unsuitable intervention (for example, the effect of dissemination of EBM guidelines rather than teaching of EBM), and two were reviews of primary studies already included in our review. Of the 23 included studies, four were randomised (level 1) trials (R Taylor, personal communication),5,22,23 and 19 were non-randomised studies (level 2), comprising seven non-randomised controlled studies and 12 before and after comparison studies. Teaching methods included workshops, seminars, and journal clubs alone or in various combinations (tables 1 and 2). Eighteen studies (R Taylor, personal communication),5-21 including two randomised trials (R Taylor, personal communication)5 (level 1), evaluated a standalone teaching method (table 1), while five studies,22-26 including two randomised trials,22,23 evaluated an integrated teaching method (table 2). Integrated teaching focused on training in EBM components (such as question formulation, literature searching, and critical appraisal) in real time clinical ward rounds or basing the EBM teaching sessions on encounters with patients on the wards and in clinics. The outcomes reported were knowledge, skills, attitude, and behaviour. None of the studies assessed patients' health.
Table 1.
Primary studies of standalone teaching of critical appraisal skills and EBM in postgraduate trainees
| Teaching methods | Assessment | Results | |
|---|---|---|---|
| Randomised controlled studies (level 1) | |||
| Linzer (1988)5 | Journal club using small group discussion | Knowledge: MCQ | Knowledge: improved |
| Behaviour: self assessment (reading habits) | Behaviour: no change | ||
| R Taylor, personal communication* | Half day critical appraisal skills workshop based on problem based small group model | Knowledge: MCQ | Knowledge: improved |
| Skills: objective assessment | Skills: no change | ||
| Attitude: self assessment | Attitude: no change | ||
| Behaviour: self assessment | Behaviour: no change | ||
| Non-randomised studies (level 2) | |||
| Controlled studies: | |||
| Gehlbach (1980)6 | Eight hour seminar | Knowledge: MCQ | Knowledge: improved |
| Kitchens (1989)7 | Critical reading seminars and small group discussion | Knowledge: MCQ | Knowledge: improved |
| Green (1997)8 | Tutorials and one-on-one teaching | Knowledge: free text responses to questions | Knowledge: improved |
| Behaviour: self assessment | Behaviour: reading quality improved | ||
| Fu (1999)9 | Journal club using EBM approach: 1.5 hours/week for 12 weeks | Knowledge: MCQ | Knowledge: no change |
| Skills: MCQ | Skills: no change | ||
| Behaviour: self assessment | Behaviour: no change | ||
| Bazarian (1999)10 | Journal clubs using EBM approach | Knowledge: scores | Knowledge: no change |
| Behaviour: self assessment | Behaviour: no change | ||
| Smith (2000)11 | EBM course: 2 hours/week for seven weeks | Attitude: self assessment | Attitude: no change |
| Skills: written tests and self assessment | Skills: improved | ||
| Behaviour: self assessment | Behaviour: improved | ||
| Ross (2003)12 | 10 workshops lasting 1-2 hours. Each workshop consisted of 30-40 minutes lecture followed by interactive session | Knowledge: MCQ | Knowledge: improved |
| Behaviour: analysis of recorded resident-preceptor interactions (use of EBM constructs) | Behaviour: improved | ||
| Before-after comparison studies (without controls): | |||
| Mulvihill (1981)13 | Seminars | Knowledge: not reported | Knowledge: improved |
| Seelig (1991)14 | Journal club based on adult learning theory | Knowledge: not reported | Knowledge: improved |
| Behaviour: self assessment (reading habits) | Behaviour: no change in number of articles read but reading quality improved | ||
| Langkamp (1992)15 | Didactic sessions followed by journal club | Knowledge: MCQ | Knowledge: no change |
| Seelig (1993)16 | One hour seminar based on principles of adult education | Knowledge: self assessment | Knowledge: no change |
| Skills: self assessment | Skills: no change | ||
| Behaviour: self assessment | Behaviour: no change | ||
| Hillson (1993)17 | Lectures and journal clubs | Skills: self assessment | Skills: improved |
| Caudill (1993)18 | Seminars followed by journal clubs | Knowledge: scores | Knowledge: no change |
| Skills: self assessment | Skills: no change | ||
| Attitudes: self assessment | Attitudes: no change | ||
| Ibbotson (1998)19 | Workshops | Knowledge: self assessment | Knowledge: improved |
| Kellum (2000)20 | Three didactic sessions on critical appraisal techniques, followed by journal club presentations | Skills: objective assessment through questionnaires, and self assessment of “confidence in the ability to critically evaluate research literature” | Skills: improvement found in both objective assessment and self assessment |
| Schoenfeld (2000)21 | Three day seminar on EBM and critical appraisal | Knowledge: self assessment | Knowledge: improved |
MCQ=multiple choice questions.
A randomised controlled trial of critical appraisal skill training for healthcare professionals (cited with permission).
Table 2.
Primary studies of clinical practice based (integrated) EBM teaching among postgraduate trainees
| Teaching methods | Assessment | Results | |
|---|---|---|---|
| Randomised controlled studies (level 1) | |||
| Bradley (2002)22 | “Real time” teaching of formulation of clinical questions and literature searching instructed by librarians based on queries raised during the day's ward round | Skills: numerical score and assessment by investigators | Skills: improved |
| Attitudes: self assessment | Attitudes: improved | ||
| Behaviour: self assessment, and assessment by investigators | Behaviour: improved | ||
| McGinn (2002)23 | “EBM teaching rounds”: daily ward rounds (except Mondays) focusing on development of searchable questions, literature search, critical appraisal, and application of evidence, based on cases presented on clinical rounds | Knowledge: self assessment | Knowledge: improved |
| Behaviour: self assessment (change in management of patients) | Behaviour: improved | ||
| Non-randomised studies (level 2) | |||
| Before-after comparison studies (without controls): | |||
| Khan (1999)24 | “EBM journal club”: one hour session every week covering formulation of searchable questions, literature search, critical appraisal, and summarising of evidence based on clinical queries from wards or outpatient clinics | Knowledge: self assessment | Knowledge: improved |
| Attitudes: self assessment | Attitudes: improved | ||
| Behaviour: self assessment (reading habits) | Behaviour: improved | ||
| Grad (2001)25 | Eight, one hour weekly sessions: first two sessions were lectures, but other sessions were based on problems encountered in clinical practice. Sessions focused on formulation of clinical questions, review of current practice, literature search, critical appraisal, and incorporation of evidence into practice | Skills: self assessment | Skills: improved |
| Attitudes: self assessment | Attitudes: improved | ||
| Behaviour: self assessment (use of information resources) | Behaviour: improved (greater use of secondary sources of information) | ||
| Haines (2003)26 | “EBM ward round”: two hour session every other week led by clinical specialist and epidemiologist on development of searchable clinical questions, literature search, critical appraisal, and summarising of evidence based on case presentations of patients currently being treated | Behaviour: assessment by investigators (for example, change in practice and guidelines) | Behaviour: improved |
Does knowledge improve?
Of the 23 studies, 17 assessed knowledge (fig 1). The weight of evidence, including the evidence from the three randomised trials (R Taylor, personal communication)5 that reported on this outcome, indicated an improvement in knowledge from both teaching methods.
Fig 1.
Changes in knowledge, skills, attitude, and behaviour after critical appraisal skills or EBM teaching, grouped by quality of studies. Data presented as 100% stacked bar chart with numbers inside bars indicating number of studies that provided information for a particular outcome (see tables 1 and 2 for details of each study)
Do critical appraisal skills improve?
Nine of the studies assessed critical appraisal skills (fig 1). The only randomised trial that reported this outcome in the standalone group did not find an improvement (R Taylor, personal communication). Of the six non-randomised studies that reported this outcome in the standalone group, three found an improvement. On the other hand, both the studies, including a randomised trial,22 which reported skills as an outcome in the integrated teaching group found an improvement. Therefore, on balance, there is weak evidence that standalone courses improve appraisal skills and good evidence, including evidence from a randomised trial,22 that the integrated approach leads to gains in appraisal skills.
Do attitudes change?
Six studies assessed change in attitudes, three each in both teaching groups (fig 1). In the standalone teaching group the three studies, including a randomised trial (R Taylor, personal communication), did not find a change in attitudes. In the integrated teaching group, however, all studies, including one randomised trial,22 found an improvement in attitudes. Therefore there is compelling evidence that teaching integrated into clinical practice changes attitudes about the role of EBM or critical literature appraisal in medicine, while a standalone approach does not.
Does behaviour change?
Fourteen studies assessed the outcome of behavioural change after EBM or critical appraisal teaching, including four randomised trials, two in each teaching group (fig 1). The two randomised trials (R Taylor, personal communication)5 in the standalone group found no change in behaviour, and both randomised trials in the integrated teaching group observed a change in behaviour.22,23 These findings from the randomised evidence were found to be consistent with the findings of the non-randomised studies, with four of seven studies in the standalone group not showing a change in behaviour and all three non-randomised studies in the integrated teaching group showing benefit. The improvements noted in behaviour included changes in reading habits24 and choice of information resources,25 as well as substantial outcomes such as changes in management of patients23,26 and guidelines.26
Do patients' health outcomes improve?
None of the studies evaluated health outcomes. As the integrated teaching approach showed that it was possible to change behaviour, however, this holds the potential for improving health outcomes. The translation of changes in behaviour into complex outcomes such as better care of patients may not be a linear one, as improving care is likely to be affected by many factors, only one of which may be the practice of EBM. Moreover, such improvements in patients' outcomes are likely to occur over a long period of time and among many other changes, making them difficult to identify in studies of evaluation of teaching or practice of EBM and critical appraisal skills.
Discussion
To our knowledge, a comparison of the effects of standalone versus integrated teaching in critical appraisal skills and EBM has not been done before. In addition to not making the distinction between standalone and integrated courses,2-4 several existing reviews have generally considered undergraduates and postgraduates together. There is empirical evidence, however, that the outcomes of teaching EBM markedly differ between undergraduates and postgraduates, with smaller gains in knowledge among the postgraduates.1 Moreover, adult learning theory suggests that the determinants of learning in the two groups are different, with postgraduate learning tending to be driven by self motivation and relevance to clinical practice, whereas undergraduate learning is generally driven by external factors such as curriculum and examinations.27 This suggests that effectiveness of educational interventions in teaching critical appraisal skills and EBM should be evaluated separately for postgraduate and continuing education, which we have done.
Studies examining the effectiveness of educational interventions may suffer from various weaknesses. Even a randomised controlled study, which is generally regarded as the optimum method for settling questions of effectiveness, is not immune to many of these weaknesses. These weaknesses include difficulty with standardising the educational intervention(s), contamination between the two arms of a study, inability to blind the study participants and the teachers from the educational intervention(s) leading to selective cointervention, and finally difficulty with measuring outcomes due to the lack of valid and reliable assessment tools. Some of these factors make randomised trials unfeasible in educational settings, thus necessitating other designs such as non-randomised controlled and before and after studies. We included all three designs in our review.
We have shown that while standalone teaching and integrated teaching are both effective in improving the knowledge base, it is clinically integrated teaching of EBM that is likely to bring about changes in skills, attitudes, and behaviour (fig 2). Changes in attitudes are likely to be important in bringing about sustained changes in behaviour, which may ultimately benefit care of patients. It is therefore important that teachers of critical literature appraisal and EBM consciously find ways of integrating and incorporating teaching of critical appraisal into routine clinical practice. Where resources and facilities are available, such teaching can form part of a “real time” ward round with the dual purposes of teaching EBM skills and attempting to improve care with best available evidence.28,29 If the provisions for real time teaching are not available, then even traditional teaching settings, such as a journal club,30 can be adapted to be based on real and current clinical problems, thus illustrating that the process is not merely an academic exercise but that it informs care.
Fig 2.
Reasons why integrated teaching may achieve better outcomes than standalone teaching
The purpose of EBM is to integrate best research evidence with clinical skills and patients' values and preferences.31 Teaching EBM should not only equip practitioners with knowledge and skills but also foster their attitudes and encourage the practice of EBM. This is because the ultimate aim of improving care could not be achieved with changes in knowledge and skills alone—it would also require changes in attitudes and behaviour. Critical appraisal and EBM teaching that is integrated into clinical practice seems more effective in improving such substantial outcomes including behavioural changes. Teachers of critical appraisal and EBM should aim to bring teaching out of classrooms into the clinic, but this will require a greater effort. Future studies should focus not only on substantial outcomes such as behaviour and health outcomes but also on longer term outcomes as there is the potential for decay of learning outcomes over time.
Summary points
Critical literature appraisal and evidence based medicine (EBM) can be taught through standalone courses or through instructional methods that incorporate teaching into routine clinical care
Several randomised and non-randomised studies have evaluated the effects of teaching EBM to postgraduates
Both standalone courses and integrated teaching improve knowledge
Improvements in skills, attitudes, and behaviour, however, come about when teaching is integrated into clinical practice; standalone courses bring about no change
It is important to incorporate EBM teaching into clinical practice, but this would require a sustained effort well beyond standalone instruction
Contributors: AC and KSK conceived, conducted, and wrote the review. AC is the guarantor.
Funding: None.
Competing interests: AC and KSK have a grant from West Midlands Deanery to teach EBM to specialist registrars in the region, as well as a European Union Grant (LSE031068WM2) to promote EBM among small to medium size enterprises that supply the NHS.
Ethical approval: Not required.
References
- 1.Norman GR, Shannon SI. Effectiveness of instruction in critical appraisal (evidence-based medicine) skills: a critical appraisal. CMAJ 1998;158: 177-81. [PMC free article] [PubMed] [Google Scholar]
- 2.Parkes J, Hyde C, Deeks J, Milne R. Teaching critical appraisal skills in health care settings. Cochrane Database Syst Rev 2001;(3): CD001270. [DOI] [PubMed]
- 3.Taylor R, Reeves B, Ewings P, Binns S, Keast J, Mears R. A systematic review of the effectiveness of critical appraisal skills training for clinicians. Med Educ 2000;34: 120-5. [DOI] [PubMed] [Google Scholar]
- 4.Coomarasamy A, Taylor R, Khan KS. A systematic review of postgraduate teaching in evidence-based medicine and critical appraisal. Med Teach 2003;25: 77-81. [DOI] [PubMed] [Google Scholar]
- 5.Linzer M, Brown JT, Frazier LM, DeLong ER, Siegel WC. Impact of a medical journal club on house-staff reading habits, knowledge, and critical appraisal skills. A randomized control trial. JAMA 1988;260: 2537-41. [PubMed] [Google Scholar]
- 6.Gehlbach SH, Bobula JA, Dickinson JC. Teaching residents to read the medical literature. J Med Educ 1980;55: 362-5. [DOI] [PubMed] [Google Scholar]
- 7.Kitchens JM, Pfeifer MP. Teaching residents to read the medical literature: a controlled trial of a curriculum in critical appraisal/clinical epidemiology. J Gen Intern Med 1989;4: 384-7. [DOI] [PubMed] [Google Scholar]
- 8.Green ML, Ellis PJ. Impact of an evidence-based medicine curriculum based on adult learning theory. J Gen Intern Med 1997;12: 742-50. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Fu CHY, Hodges B, Regehr G, Goldboom DS, Garfinkel P. Is a journal club effective for teaching critical appraisal skills? A controlled trial with residents in psychiatry. Acad Psychiatry 1999;23: 205-9. [Google Scholar]
- 10.Bazarian JJ, Davis CO, Spillane LL, Blumstein H, Schneider SM. Teaching emergency medicine residents evidence-based critical appraisal skills: a controlled trial. Ann Emerg Med 1999;34: 148-54. [DOI] [PubMed] [Google Scholar]
- 11.Smith CA, Ganschow PS, Reilly BM, Evans AT, McNutt RA, Osei A, et al. Teaching residents evidence-based medicine skills: a controlled trial of effectiveness and assessment of durability. J Gen Intern Med 2000;15: 710-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Ross R, Verdieck A. Introducing an evidence-based medicine curriculum into a family practice residency—is it effective? Acad Med 2003;78: 412-7. [DOI] [PubMed] [Google Scholar]
- 13.Mulvihill MN. Faculty development and resident training in epidemiology and biostatistics. Mt Sinai J Med 1981;48: 350-2. [PubMed] [Google Scholar]
- 14.Seelig CB. Affecting residents' literature reading attitudes, behaviors, and knowledge through a journal club intervention. J Gen Intern Med 1991;6: 330-4. [DOI] [PubMed] [Google Scholar]
- 15.Langkamp DL, Pascoe JM, Nelson DB. The effect of a medical journal club on residents' knowledge of clinical epidemiology and biostatistics. Fam Med 1992;24: 528-30. [PubMed] [Google Scholar]
- 16.Seelig CB. Changes over time in the knowledge acquisition practices of internists. South Med J 1993;86: 780-3. [DOI] [PubMed] [Google Scholar]
- 17.Hillson SD, Schlossberg LA. Effects of journal club on critical review. Clin Res 1993;41: 559A. [Google Scholar]
- 18.Caudill TS, Johnson MMS, Rich EC. The effect of journal club on medicine resident interaction with original research literature. Clin Res 1993;41: 816A. [Google Scholar]
- 19.Ibbotson T, Grimshaw J, Grant A. Evaluation of a programme of workshops for promoting the teaching of critical appraisal skills. Med Educ 1998;32: 486-91. [DOI] [PubMed] [Google Scholar]
- 20.Kellum JA, Rieker JP, Power M, Powner DJ. Teaching critical appraisal during critical care fellowship training: a foundation for evidence-based critical care medicine. Crit Care Med 2000;28: 3067-70. [DOI] [PubMed] [Google Scholar]
- 21.Schoenfeld P, Cruess D, Peterson W. Effect of an evidence-based medicine seminar on participants' interpretations of clinical trials: a pilot study. Acad Med 2000;75: 1212-4. [DOI] [PubMed] [Google Scholar]
- 22.Bradley DR, Rana GK, Martin PW, Schumacher RE. Real-time, evidence-based medicine instruction: a randomized controlled trial in a neonatal intensive care unit. J Med Libr Assoc 2002;90: 194-201. [PMC free article] [PubMed] [Google Scholar]
- 23.McGinn T, Seltz M, Korenstein D. A method for real-time, evidence-based general medical attending rounds. Acad Med 2002;77: 1150-2. [DOI] [PubMed] [Google Scholar]
- 24.Khan KS, Pakkal M, Brace V, Dwarakanath LS, Awonuga A. Postgraduate journal club as a means of promoting evidence-based obstetrics and gynaecology. J Obstet Gynaecol 1999;19: 231-4. [DOI] [PubMed] [Google Scholar]
- 25.Grad R, Macaulay AC, Warner M. Teaching evidence-based medical care: description and evaluation. Fam Med 2001;33: 602-6. [PubMed] [Google Scholar]
- 26.Haines SJ, Nicholas JS. Teaching evidence-based medicine to surgical subspecialty residents. J Am Coll Surg 2003;197: 285-9. [DOI] [PubMed] [Google Scholar]
- 27.Knowles MS, Downie CM, Basford P. Teaching and assessing in clinical practice. London: University of Greenwich, 1998: 23-38.
- 28.Sackett DL, Straus SE. Finding and applying evidence during clinical rounds: the “evidence cart.” JAMA 1998;280: 1336-8. [DOI] [PubMed] [Google Scholar]
- 29.Deshpande N, Publicover M, Gee H, Khan KS. Incorporating the views of obstetric clinicians in implementing evidence-supported labour and delivery suite ward rounds: a case study. Health Info Libr J 2003;20: 86-94. [DOI] [PubMed] [Google Scholar]
- 30.Dwarakanath LS, Khan KS. Modernizing the journal club. Hosp Med 2000;61: 425-7. [DOI] [PubMed] [Google Scholar]
- 31.Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practice and teach EBM. Edinburgh: Churchill Livingstone, 2000.