Effectiveness of an e-learning course in evidence-based medicine for foundation (internship) training

Summary

Aim

To evaluate the educational effectiveness of a clinically integrated e-learning course for teaching basic evidence-based medicine (EBM) among postgraduate medical trainees compared to a traditional lecture-based course of equivalent content.

Methods

We conducted a cluster randomized controlled trial to compare a clinically integrated e-learning EBM course (intervention) to a lecture-based course (control) among postgraduate trainees at foundation or internship level in seven teaching hospitals in the UK West Midlands region. Knowledge gain among participants was measured with a validated instrument using multiple choice questions. Change in knowledge was compared between groups taking into account the cluster design and adjusted for covariates at baseline using generalized estimating equations (GEE) model.

Results

There were seven clusters involving teaching of 237 trainees (122 in the intervention and 115 in the control group). The total number of postgraduate trainees who completed the course was 88 in the intervention group and 72 in the control group. After adjusting for baseline knowledge, there was no difference in the amount of improvement in knowledge of EBM between the two groups. The adjusted post course difference between the intervention group and the control group was only 0.1 scoring points (95% CI −1.2–1.4).

Conclusion

An e-learning course in EBM was as effective in improving knowledge as a standard lecture-based course. The benefits of an e-learning approach need to be considered when planning EBM curricula as it allows standardization of teaching materials and is a potential cost-effective alternative to standard lecture-based teaching.

Aim

Acquiring knowledge, skills and attitudes to practice evidence-based medicine (EBM) is a core competence for all junior doctors.¹ For EBM teaching and learning to be effective it should be clinically integrated.² However, the majority of teaching of EBM and critical appraisal takes place in the classroom away from a clinical setting. Integrating EBM teaching into a clinical setting is not a straightforward task and risks distracting from service delivery as it can be time-consuming. The pool of teachers trained in providing face-to-face teaching on the foot is limited.³

To address the lack of time and capacity to deliver EBM teaching on the go, we developed an e-learning EBM course for just-in-time learning through on-the-job training for postgraduate medical trainees.^4,5 Its feasibility was piloted in different languages and settings across five European countries in a before-and-after study. It showed that knowledge improved significantly and both trainees and tutors found the e-learning course acceptable.⁵ How would such a course fare against delivery of equivalent content in the traditional lecture mode during foundation (internship) training? We determined the effect of the e-EBM course on participant's knowledge of EBM in a randomized controlled trial.

Methods

Design

We conducted a cluster randomized controlled trial⁶ to compare the clinically integrated e-learning EBM course to a lecture-based course between May and September 2007.

Setting

The trial was undertaken in seven teaching hospitals in the UK West Midlands region.

Participants

All of the participants in the trial were foundation year two doctors (interns). All trainees consented to be part of the trial.

Recruitment

The West Midlands Deanery leads the training programmes for all foundation-year doctors in the region. Approval was obtained from the Deanery, who granted us permission to undertake the trial. All of the hospitals in the region who were responsible for postgraduate medical education were recruited for the trial.

Randomization

The teaching hospitals were randomized by means of computer programme into two groups: Group 1 – received the e-learning EBM teaching programme (experimental intervention) (four clusters, 122 participants); and Group 2 – received standard classroom-based standalone EBM teaching sessions of equivalent content (control group) (three clusters, 115 participants). Randomization was performed after recruitment and after consent was obtained.

Intervention and comparator

The teaching sessions (standalone and e-learning) were divided into three separate modules. The module contents covered the topics: asking and formulating clinical questions, searching for research evidence and critical appraisal of systematic reviews and their constituent studies ( Table 1). The development of the curriculum and the contents of the modules have been previously reported.⁴ The learning objectives and material contents for both the standalone and the e-learning group were the same.

Table 1.

An overview of the clinically integrated e-learning course in evidence-based medicine (EBM)

Aim

To familiarize course participants with evidence-based medicine (EBM) basics

Target participants

Foundation (internship) doctors in a clinical teaching setting

Learning objectives

Upon the completion of the course, participants should be competently able to: generate structured questions arising from clinical problems in practice search relevant literature, identifying systematic reviews wherever possible assess the quality (validity) of systematic reviews and primary research included within them

E-learning modules

Three models provide learning materials at www.ebm-unity.org Module 1: Asking clinical questions Module 2: Searching the evidence Module 3: Critical appraisal (of systematic reviews and their constituent studies)

Learning/teaching methods Participants to pursue independent study by using the e-learning modules and to undertake formal assessment

Assessment Multiple choice questions to test knowledge

For the standalone group, the teaching was delivered by a lecturer over one session consisting of three hours in each centre, following the standard format used in the educational programme devised for foundation training. To ensure equivalence of teaching content, we used PowerPoint slides identical to those deployed in the e-EBM course in the same sequence. The e-learning group were granted unlimited access for a period of six weeks to the e-learning materials via a project specific website (www.ebm-unity.org ). Each participant was given their own personal log-in code and their log-in activities were monitored on a weekly basis to ensure compliance with the trial protocol.

Data collection

All participants completed validated multiple choice questions (MCQs)⁷ to assess EBM knowledge before accessing the e-learning materials and prior to the start of the teaching sessions. After completion of each module the participants completed the same MCQs relevant to that module again ( Figure 1). The MCQs were developed previously as part of a study to evaluate the outcomes of the EBM e-learning course in five European countries, which has been published elsewhere.⁵ The MCQs were linked to the learning objectives of the course to assure content validity.

Figure 1.

Flow chart of participants in the trial

Analysis

There was no a priori sample size calculation. However, post hoc statistical power computation shows that a study with seven clusters (mean cluster size of 22) and assuming an intraclass correlation coefficient (ICC) of 0.01, with a standard type I error rate (p 0.05) our sample had 80% power to detect a 3.5-point difference between groups in post course scores. Assuming a common standard deviation of 7 points, this represents a medium-effect size in terms of Cohen's d.⁸ If we assume an ICC of 0.02, the corresponding power would have been 74%.

Responses to the pre and post module MCQs were scored and comparisons between the e-learning and the standalone group were made. Missing data were not imputed and analyses were restricted only to those participants who answered both pre and post module MCQs. Analyses were done in an intention-to-treat basis (all participants were analysed in the group they were randomized in). Between arms comparison was undertaken by means of generalized estimating equations (GEE) model⁹ which permits the specification of a working correlation matrix to account for within-cluster (i.e. hospital) correlation of the outcome. In this case we assume that there is no order for participants within a hospital and then an exchangeable correlation matrix was specified. The model included teaching modality as the independent variable and baseline knowledge as a covariate.

Results

A total of 91 postgraduate trainees completed the pre course MCQs in the e-learning (intervention) group and 79 in the standard classroom teaching (control) group. The number of participants who did not fully complete the course was 3 in the e-learning group and 7 in the standard classroom teaching group. The reasons for not completing included being called out of the teaching session to see patients, being on leave and leaving the training programme. The total number who completed the course and returned both pre and post course assessments was 88 in the e-learning group and 72 in the standard classroom teaching group (Figure 1).

Table 2 shows overall baseline knowledge scores and scores by teaching module for the two arms of the trial. The scores were generally high in both groups. Maximum attainable scores were 13, 7 and 18 points for modules 1, 2 and 3, respectively. Post course scores were improved in both groups compared to baseline (Table 2, Figure 2). After adjusting for baseline knowledge and taking into account the cluster nature of the trial design, there was no difference in the amount of improvement of knowledge between the two groups. The adjusted post course difference between the intervention group and the control group was only 0.1 scoring points (95% CI −1.2–1.4) (Figure 2).

Table 2.

Knowledge (MCQ) scores after course delivery

	Control group Baseline n=72	Control group Post course n=72	e-learning group Baseline n=88	e-learning group Post course n=88	Adjusted post course difference* (95% CI)	P value
Module 1: Framing questions	8.9 (1.8)	10.5 (1.8)	8.4 (3.9)	9.7 (4.3)	−0.3 (−1.1; 0.4)	0.42
Module 2: Searching literature	4.2 (1.4)	6.0 (1.5)	4.7 (1.6)	6.1 (1.9)	−0.2 (−0.3; 0.0)	0.03
Module 3: Critical appraisal	11.7 (2.4)	11.7 (3.8)	9.9 (4.2)	11.1 (4.6)	0.6 (−0.4; 1.7)	0.24
Overall	24.7 (3.9)	28.2 (6.0)	22.9 (7.0)	27.0 (7.5)	0.1 (−1.2; 1.4)	0.89

^*GEE analyses adjusting for baseline score. See methods section for details

Figure 2.

Comparison of knowledge scores achieved through standard teaching versus e-learning (dots show data for individual participants; boxes with vertical whiskers show means and SD)

Discussion

Main findings

The trial demonstrated that both the e-learning and the standard classroom-based teaching approaches lead to an overall improvement in EBM knowledge. The e-learning group achieved similar scores for overall knowledge gain compared to the control group. This raises the possibility of delivering this element of foundation (internship) curriculum through e-learning, replacing the traditional lecture-based method without loss of knowledge.

Comparison with other studies

There have been several other studies that have examined the effectiveness of computer-based teaching of EBM compared to standard lecture-based approach^10–14 and our study also concurs with their findings. Our trial merits consideration as we undertook a randomized design to provide scientifically sound results. One of the strengths of the trial was that we were able to recruit more participants than previous trials^10–12 and this increases the precision of our findings. Furthermore, a cluster randomized trial design was chosen to reduce the effect of contamination.¹⁵ This is a particular concern with educational research. Additionally, cross-over of participants and an inability to blind participants and the teachers can cause concern in educational research.¹¹ These potential methodological flaws are addressed better with cluster randomized trials.¹⁶ Concerning randomization we ensured allocation concealment. Our statistical analysis of the data took into consideration the cluster design.⁹ Our results concur with a previous study examining the effectiveness of teaching EBM to postgraduate medical doctors in the obstetrics and gynaecology specialty by e-learning, which was conducted in the Netherlands and the UK.¹²

Methodological limitations

There are several limitations of our trial that warrant consideration in the interpretation of our findings. As previously found,¹² trainees in both groups showed high baseline knowledge for all modules prior to the EBM teaching and this left little margin for improvement in the post course scores. This may be attributed to previous training in EBM within the deanery. Several questionnaire surveys to address junior doctors' knowledge and beliefs concerning EBM have been undertaken in the deanery^17,18 which has fostered an educational environment that promotes the use of EBM in clinical practice. The participants may have been more aware and self-motivated than trainees generally. Furthermore, a small proportion of participants in both groups had lower scores post course compared to pre course. This has also been found in other studies that have used knowledge assessments and has been associated with cognitive test anxiety.^12,19,20 Clinically integrated teaching of EBM can be expected to bring about changes in behaviour, practice and outcomes, but we were unable to measure these.

Implications for practice

As postgraduate medical training in the UK is currently evolving to meet future healthcare needs, it is imperative that it encompasses the challenges of teaching doctors in changing environments.²¹ The benefits of an e-learning approach to teaching should be considered as a viable way in meeting these challenges, as it can support a wide range of learning activities, which are readily accessible and can be tailor-made to meet specific learning objectives. As our trial implies, e-learning teaching provides knowledge gains equivalent to that of standard classroom-based teaching. Particularly, the benefits should be considered when planning EBM curricula as it allows standardization of teaching materials and is a potential cost-effective alternative to standard lecture-based sessions.

Footnotes

DECLARATIONS —

Competing interests None declared

Funding The project was funded by the European Union Leonardo da Vinci project (grant number UK/05/B/F/PP-162_349)

Ethical approval Not applicable

Guarantor JH

Contributorship The trial was conceived by KSK and the Leonardo project team. JH coordinated the project, delivered the teaching sessions and conducted the trial. GZ provided Internet-based data collection for the e-learning group. JZ performed the statistical analysis. All authors commented on the manuscript

Reviewer Elizabeth Murray