Evaluating the teaching of evidence based medicine: conceptual framework

Short abstract

Although evidence for the effectiveness of evidence based medicine has accumulated, there is still little evidence on what are the most effective methods of teaching it.

Interest in evidence based medicine (EBM) has grown exponentially, and professional organisations and training programmes have shifted their agenda from whether to teach EBM to how to teach it. However, there is little evidence about the effectiveness of different methods,¹ and this may be related to the lack of a conceptual framework within which to structure evaluation strategies. In this article we propose a potential framework for evaluating methods of teaching EBM. Showing the effectiveness of such teaching methods relies both on psychometrically strong measurements and methodologically rigorous and appropriate study designs, and our framework addresses the former.

This effort was initiated by the Society of General Internal Medicine Evidence-Based Medicine Task Force.² In an attempt to tackle the challenges in designing and evaluating a series of teaching workshops on EBM for busy practising clinicians, the task force created a conceptual framework for evaluating teaching methods. This was done by a working group of clinicians interested in the subject. They completed a literature review of instruments used for evaluating teaching of EBM (manuscript in preparation), and two members of the task force used the information to draft a conceptual framework. This framework and relevant background materials were discussed and revised at a consensus conference including 10 physicians interested in EBM, evaluation of education methods, or programme development. We then sent a revised framework to all members of the task force and six other international colleagues interested in the subject. We incorporated their suggestions into the framework presented in this article.

When formulating clinical questions, advocates of EBM suggest using the “PICO” approach—defining the patient, intervention, comparison intervention, and outcome.³ We used this approach to provide a framework for the evaluation matrix, specifically:

• Who is the learner?

• What is the intervention?

• What is the outcome?

The answers to these three questions form the structure of our conceptual model.

Who is the learner?

Learners can be doctors, patients, policy makers, or managers. This article focuses on doctors, but our evaluation framework could be applied to other audiences.

Figure 1.

Credit: PHILIP SIMPSON/PHOTONICA

Not all doctors want or need to learn how to practise all five steps of EBM (asking, acquiring, appraising, applying, assessing).⁴,⁵ Indeed, most doctors consider themselves users of EBM, and surveys of clinicians show that only about 5% believe that learning all these five steps is the most appropriate way of moving from opinion based to evidence based medicine.⁴

Doctors can incorporate evidence into their practice in three ways.³,⁶ In a clinical situation, the extent to which each step of EBM is performed depends on the nature of the encountered condition, time constraints, and level of expertise with each of the steps. For frequently encountered conditions (such as unstable angina) and with minimal time constraints, we operate in the “doing” mode, in which at least the first four steps are completed. For less common conditions (such as aspirin overdose) or for more rushed clinical situations, we eliminate the critical appraisal step and operate in the “using” mode, conserving our time by restricting our search to rigorously preappraised resources (such as Clinical Evidence). Finally, in the “replicating” mode we trust and directly follow the recommendations of respected EBM leaders (abandoning at least the search for evidence and its detailed appraisal). Doctors may practise in any of these modes at various times, but their activity will probably fall predominantly into one category.

The various methods of teaching EBM must therefore address the needs of these different learners. One size cannot fit all. Similarly, if a formal evaluation of the educational activity is required, the evaluation method should reflect the different learners' goals. Although several questionnaires have been shown to be useful in assessing the knowledge and skills needed for EBM,⁷,⁸ we must remember that learners' knowledge and skills targeted by these tools may not be similar to our own. The careful identification of our learners (their needs and learning styles) forms the first dimension of the evaluation framework that we are proposing.

What is the intervention?

The five steps of practising EBM form the second dimension of our evaluation framework. But what is the appropriate dose and formulation? If our learners are interested in practising in the “using” mode, our teaching should focus on formulating questions, searching for evidence already appraised, and applying that evidence. Evaluation of the effectiveness of the teaching should exclusively assess these steps. In contrast, doctors interested in practising in the “doing” mode would receive training in all five steps of practising EBM, and the evaluation of the training should reflect this.

Published evaluation studies of teaching EBM show the diversity of existing teaching methods. Some evaluation studies use an approach to clinical practice, whereas others use training in one of the skills of EBM such as searching Medline⁹ or critical appraisal.¹⁰ Indeed, one review of 18 reports of graduate medical education in EBM found that the courses most commonly focused on critical appraisal skills, in many cases to the exclusion of other necessary skills.¹¹ Some studies have looked at 90 minute workshops whereas others included courses that were held over several weeks to months, thereby increasing the “dose” of teaching. Evaluation instruments should be tailored to the dose and delivery method, thereby assessing outcomes and behaviours that are congruent with the intended objectives.

What are the outcomes?

Effective teaching of EBM will produce a wide range of outcomes. Various levels of educational outcomes could be considered, including attitudes, knowledge, skills, behaviours, and clinical outcomes. The outcome level (the third dimension of the conceptual framework) reflects Miller's pyramid for evaluating clinical competence¹² and builds on the competency grid for evidence based health care proposed by Greenhalgh.¹³ Changes in doctors' knowledge and skills are relatively easy to detect, and several instruments have been evaluated for this purpose.⁷,⁸ However, many of these instruments primarily evaluate critical appraisal skills, focusing on the role of “doer” rather than “user.” A Cochrane review of critical appraisal teaching found one study that met the authors' inclusion criteria and that the course studied increased knowledge of critical appraisal.¹⁰ With our proposed framework, evaluation of this teaching course falls into the learner domain of “doing,” the intervention domain of “appraisal,” and the outcome domain of “knowledge.”

Changes in behaviours and clinical outcomes are more difficult to measure because they require assessment in the practice setting. For example, in a study evaluating a family medicine training programme, doctor-patient interactions were videotaped and analysed for EBM content.¹⁴ A recent before and after study has shown that a multi-component intervention including teaching EBM skills and providing electronic resources to consultants and house officers significantly improved their evidence based practice (Straus SE et al, unpublished data). With our proposed framework, evaluation of this latter teaching intervention would be categorised into the learner domain of “doing.” The intervention domains include all five steps of EBM, and the outcome domain would be “doctor behaviour.”

Implementing the evaluation framework

The EBM task force developed teaching workshops for practising doctors that focused on formulating questions and searching for and applying preappraised evidence. Because these workshops were unlike traditional workshops that focused on the five steps of practising EBM,¹⁵ we concluded that evaluation of these workshops must be different. We created an evaluation instrument to detect an effect on learners' EBM knowledge, attitudes, and skills.

When we applied the evaluation framework to our evaluation instrument we found that our learners' goals were different from what we were assessing (table 1). We found that we placed greater emphasis on the skills necessary for practising in the “doing” mode than those required in the “using” mode, whereas the intervention was targeted to improve “user” behaviour. Moreover, the assessment mirrored traditional evaluation methods, focusing on appraisal skills, with little attention paid to question formulation. Finally, we saw that our evaluation predominantly measured skills rather than behaviour. This reflection led us to redesign our evaluation instrument to more closely reflect the learning objectives. We also attempted to show how the evaluation framework could be used—how to move from a concept to actual use (table 2).

Table 1.

Application of evaluation framework to SGIM EBM Task Force evaluation tool

		Intervention*
Outcome	Learner	Ask	Acquire	Appraise	Apply	Assess
Attitudes	Replicator	1
	User	1
	Doer	1
Knowledge	Replicator
	User		10, 12
	Doer		10, 12
Skills	Replicator	3, 5	5	5	5
	User	3, 5	5	5	5, 16ii
	Doer	3, 5	5	5, 8, 9, 11, 15, 16i	5, 13, 14, 16ii
Behaviour	Replicator	3	1, 2
	User	3	1, 2
	Doer	3	1, 2
Clinical outcomes	Replicator
	User
	Doer

SGIM EBM Task Force=Society of General Internal Medicine Evidence-Based Medicine Task Force.

^*Numbers refer to questions on the evaluation tool (see sample questions from evaluation tool on bmj.com).

Table 2.

Application of the conceptual framework for formulating clinical questions

Outcome	Replicator	User	Doer
Attitudes	• Recognise the importance of identifying knowledge gaps	• Replicator objectives and	• User objectives
	• Recognise that converting the gap into a focused clinical question is important	• Recognise that multiple knowledge deficits commonly exist in clinical situations
	• Be open to new knowledge and to seeking new knowledge
Knowledge	• List and understand crucial, relevant components of a focused clinical question	• List and understand all relevant components of a focused clinical question	• User objectives
Skill	• Construct a focused clinical question that contains relevant components	• Be able to ask a focused clinical question containing all relevant components for each type of clinical question that arises	• User objectives
		• Be able to select the appropriate question(s) to pursue from the list based on importance to user's and patient's needs
Behaviour	• Occasionally ask appropriate colleagues focused clinical questions containing relevant components	• Frequently use appropriate, focused clinical questions relevant to clinical patients in order to seek new knowledge about the care of these patients	• User objectives and
			• Often record the focused clinical questions that arise and those questions that have been answered
Clinical outcomes	• Use clinical questions to identify gaps in practice and to change practice accordingly	• Replicator objectives	• Replicator objectives

Limitations of this framework

Our model requires that teachers work with learners to understand their goals, to identify in what mode of practice they want to enhance their expertise, and to determine their preferred learning style. This simple model could be expanded to include other dimensions, including the role of the teacher and the “dose” and “formulation” of what is taught. However, our primary goal was to develop a matrix that was easy to use. Although we have applied this framework to several of the published evaluation instruments and have found it to be useful, others may find that it does not meet all of their requirements.

What's next?

While EBM teachers struggle with developing innovative course materials and evaluation tools, we propose a coordinated sharing of these materials in order to minimise duplication of effort. Using the proposed framework as a categorisation scheme, the task force is establishing an online clearinghouse to serve as a repository for evaluations of methods of teaching EBM including details on their measurement properties.² Teachers will be able to identify evaluation tools that might be useful in their own setting, using the framework to target their needs.

There is still little evidence about the effectiveness of different teaching methods,¹ and attempting to evaluate such teaching is challenging given the complexity of the learners, the interventions, and the outcomes. One way to help meet these challenges is to develop a collaborative research network to conduct multicentre, randomised trials of educational interventions. We invite interested colleagues to join us in developing this initiative and to create the clearinghouse for evaluation tools (www.sgim.org/ebm.cfm).

Summary points

There is little evidence about the effectiveness of different methods of teaching evidence based medicine

Doctors can practise evidence based medicine in one of three modes—as a doer, a user, or a replicator

Instruments for evaluating different methods of teaching evidence based medicine must reflect the different learners (their learning styles and needs), interventions (including the dose and formulation), and outcomes that can be assessed

Our framework provides only one way to conceptualise the evaluation of teaching EBM; many others could be offered. We hope that our model serves as an initial step towards discussion and that others will offer their suggestions so that we may work together towards improved understanding of the evaluation process and promote more rigorous research on the evaluation of teaching EBM.