Skip to main content
The BMJ logoLink to The BMJ
. 2002 Jan 19;324(7330):153–156. doi: 10.1136/bmj.324.7330.153

Problem based learning in continuing medical education: a review of controlled evaluation studies

P B A Smits a, J H A M Verbeek b, C D de Buisonjé a
PMCID: PMC64518  PMID: 11799034

Problem based learning is one of the best described methods of interactive learning, and many claim it is more effective than traditional methods in terms of lifelong learning skills, and is more fun.1 In the early 1990s, four systematic reviews of undergraduate medical education cautiously supported the short term and long term outcomes of problem based learning compared with traditional learning.25 Since then, many medical curricula have changed to problem based learning, but a recent review has questioned the value of problem based learning in undergraduate medical education.6

Postgraduate and continuing medical education differ from undergraduate education in that they go beyond increasing knowledge and skills to improving physician competence and performance in practice, ultimately leading to better patient health.7 Problem based learning may also be effective in this context.8 There is some evidence that interactive sessions can change professional practice, but there have been few well conducted trials.9,10

We could find no reviews of the effectiveness of problem based learning in continuing medical education. Controlled evaluation studies provide the best evidence of effectiveness of educational methods, in line with the movement of best evidence medical education.11 We therefore conducted a systematic review of the literature to find out if there is evidence that problem based learning in continuing medical education is effective.

Summary points

  • Reviews of undergraduate medical education cautiously support the short term and long term outcomes of problem based learning compared with traditional learning

  • The effectiveness of problem based learning in continuing medical education, however, has not been reviewed

  • This review of controlled evaluation studies found limited evidence that problem based learning in continuing medical education increased participants' knowledge and performance and patients' health

  • There was moderate evidence that doctors are more satisfied with problem based learning

Methods

Literature search

We searched the databases Medline, Embase, Psyclit, the Educational Resources Information Centre (ERIC), the Cochrane Library, and the Research and Development Resource Base in CME on the Internet (RDRBWEB) from 1974 (the year Neufeld and Barrows published their new approach to medical education) to August 2000. We searched for studies with the keywords “problem-based (PBL),” “practice-based,” “self-directed,” “learner centred,” and “active learning.” We combined the search results with another search using the keywords “continuing medical education (CME),” “continuing professional development (CPD),” “post-professional,” “postgraduate,” and “adult learning.” Finally, we conducted a manual search of relevant references in the included studies.

Inclusion criteria

We included studies in which the author(s) had indicated that the educational intervention was problem based and in which the learning process in essence resembled the methods used at McMaster University or the University of Maastricht.12,13 This consists of a tutor facilitated, problem based learning session in which a small, self directed group starts with a brainstorming session. A problem is posed that challenges their knowledge and experience. Learning goals are formulated by consensus, and new information is learnt by self directed study. It ends with a group discussion and evaluation.

For this review, we included keywords to find relevant educational articles in the total domain of postgraduate and continuing medical education and continuing professional development. We scanned all the studies collected for controlled trials with a pretest/post-test design. Because of the small number of randomised trials, we did not exclude other types of controlled trial. With this strategy, we hoped to find all relevant controlled studies on problem based learning in continuing medical education.

Review method of selected studies

Two reviewers (PBAS and JHAMV) independently assessed the quality of the studies using five quality criteria. Each criterion was allotted a maximum of 10 points, making a maximum possible score of 50 points (see appendix on bmj.com for more details). We discarded a sixth possible criterion, that groups should be treated equally, with the exception of experimental education.14 Many factors may influence the outcome of education (tutor, educational materials, lecture rooms, etc), and it was not possible to extract this information from the studies: we therefore could not assess equal treatment of groups.

Studies with a total score of ⩾25 points were considered to be of high quality, and those with <25 points were of low quality. We distinguished two different categories of study by the control groups: in one category problem based learning was compared with a more lecture based programme, while in the other it was compared with no educational intervention.

Outcome variables

For each study, we looked for four outcome variables— participants' knowledge, performance, and satisfaction and patients' health—and assessed the level of evidence on these. We graded the evidence for the effectiveness of problem based learning as strong if there was a positive outcome in two high quality studies, as moderate if there was a positive outcome in one high quality and one low quality study, as limited if there was a positive outcome in one high quality study or one or more low quality study, and none if there was a contradictory outcome or no outcome.

Results of literature search

Six controlled trials met our inclusion criteria.1520 A manual search of references from these studies did not yield any new trials that met our criteria. Five of the studies assessed the effect of continuing medical education on general practitioners. Four studies contained over 50 participants,1618,20 and two had fewer than 20.15,19

Table 1 shows the results of our quality assessment of the six studies. Two studies were of “high” quality,15,17 and the others were “low.” Two of the trials were randomised trials.15,16 In Heale et al's study, however, the group of randomly allocated doctors was combined with a group who did not want to participate in the entire study.16 Whether the randomisation is valid in terms of equality of groups is unclear.

Table 1.

Quality assessment of six studies evaluating effectiveness of problem based learning in continuing medical education

Study Scores for quality criteria*
Total score Quality of study
Randomised assignment to group? Follow up sufficiently long and complete? Intention to treat analysis? Participants and observers blinded? Groups similar at start?
Chan et al 199915 10 5 10 0 10 35 High
Heale et al 198816 5 8 0 0 0 13 Low
Doucet et al 199817 5 8 10 0 5 28 High
Benjamin et al 199918 5 8 10 0 0 23 Low
Moran et al 199619 0 8 10 0 0 18 Low
Premi et al 199420 0 8 10 0 0 18 Low
*

Each criterion has maximum score of 10, and maximum total score is 50. 

Total scores ⩾25 indicate high quality, and scores <25 indicate low quality . 

Results of studies

Table 2 shows the results of the six studies. Outcome measurement was often restricted to only one variable. No study measured both the preferred outcome variables—participants' performance and patients' health. In one of the high quality studies—problem based learning via email versus use of internet resources—neither educational programme increased participants' knowledge, but group size was small.15 The other high quality study—problem based versus lecture based learning—showed positive results for problem based learning in terms of participants' knowledge, clinical reasoning, and satisfaction.17 It is unclear whether these effects can be attributed to the problem based learning format, however, because of differing periods of educational exposure.

Table 2.

Results of six studies evaluating effectiveness of problem based learning in continuing medical education

Study Participants Study design Educational intervention
No of participants
Outcome variables (PBL v control)
PBL Control PBL Control Participants' knowledge Participants' performance Participants' satisfaction Patients' health Follow up
Chan et al 199915 General practitioners RCT Email PBL Internet resources  8 11 No effect NA NA NA NA
Heale et al 198816 General practitioners RCT Small group PBL Large group discussion, didactic lecture  22*  39* No effect No effect Higher NA No effect
Doucet et al 199817 General practitioners CT PBL Lecture format 34 29 Positive Positive Higher NA NA
Benjamin et al 199918 Staff of outpatient clinic CT PBL None ∼50§ ∼50§ NA NA NA Positive Positive
Moran et al 199619 General practitioners CT Learner centred small group None  5 10 NA Positive** NA NA Positive**
Premi et al 199420 General practitioners CT Practice based small group None 78 46 Positive** Positive‡‡ High score†† NA NA

RCT=Randomised controlled trial. CT=Controlled trial. PBL=Problem based learning. NA=Not assessed. 

*

No information on number who completed knowledge test at end of trial. 

P value not given in article. 

Only participants' knowledge tested on follow up. 

§

Information from author, not given in article. 

Only participants' performance tested on follow up. 

**

PBL group's post-intervention scores compared with own pre-intervention scores. 

‡‡

75/78 participants reported 127 practice changes as result of PBL. 

††

Mean (SD) score of 5.8 (0.9) on 7 point scale for PBL group. 

Table 3 shows the level of evidence we found for the outcome variables. With the three studies that compared problem based learning with another educational format, we found no evidence that problem based learning affected participants' knowledge and performance and moderate evidence that it increased participants' satisfaction. None of the studies measured patients' health. The other three studies compared problem based learning with no educational intervention and were of low quality. They show limited evidence that problem based learning was effective in improving participants' knowledge and performance and patients' health (table 3). Differing degrees of satisfaction cannot be compared in this study design.

Table 3.

Level of evidence on outcome variables measured in six studies evaluating effectiveness of problem based learning in continuing medical education

Outcome variable High quality studies
Low quality studies
Level of evidence*
Positive result Negative result Positive result Negative result
Problem based learning v other educational interventions
 Participants' knowledge 1 1 0 1 None
Participants' performance 1 0 0 1 None
Participants' satisfaction 1 0 1 0 Moderate
Patients' health 0 0 0 0 None
Problem based learning v no educational intervention
 Participants' knowledge 0 0 1 0 Limited
Participants' performance 0 0 3 0 Limited
Patients' health 0 0 1 0 Limited
*

Possible levels of evidence: strong, moderate, limited, or none (see Methods for details). 

Conclusions

We found few relevant studies, of varying quality. There is no consistent evidence that problem based learning in continuing medical education was superior to other educational strategies in increasing doctors' knowledge and performance but moderate evidence that it led to higher satisfaction. There is limited evidence that problem based learning increased doctors' knowledge and performance and patients' health more than no educational intervention at all.

However, the studies in which the control group received no educational intervention can give information only on the effects of receiving education, not of the specific educational method. With the studies that compared problem based learning with another method, in order to deduce that one educational intervention is more effective, the content, process, and influencing variables in both interventions must be clearly stated. The information on the educational interventions given in the three studies can be rated as completely absent,16 poor,15 and reasonable.17

In studies not restricted to problem based learning, there is some evidence that interactive educational methods in continuing medical education are more effective in changing doctors' performance and patients' health.9 The results of this literature study on problem based learning in continuing medical education seem to be comparable with those on problem based learning in undergraduate medical education.24

Studying the effectiveness of education is complex,21,22 but we should be able to perform studies of higher quality than those reviewed here, especially when comparing educational methods. As our review found, it is apparently not impossible to randomise participants to different educational methods. We have to do better in defining an educational method and controlling what actually happens in educational practice. We also have to clarify the aims of our education. Is our objective to increase knowledge, change attitudes, or improve health care? Outcome variables should correspond with our objectives, and preferably several different variables should be measured, including participants' performance and patients' health.9 There seems to be agreement that a small significant effect found is evidence of effectiveness.21,23 Evaluation is further complicated by professional and social context, as is shown in research on implementation of guidelines.24 This calls for randomisation, because observational studies can easily be biased by these factors.

Supplementary Material

[extra: Details of criteria used]

Footnotes

Editorials by Prideaux and Goldbeck-Wood and Peile

Funding: This study was supported by the Netherlands Organisation of Scientific Research (NWO) as part of the priority programme on fatigue and work and by the Netherlands School of Occupational Health.

Competing interests: None declared.

Details of criteria used to assess quality of studies appear on bmj.com

References

  • 1.Dolmans D, Schmidt HG. The advantages of problem-based curricula. Postgrad Med J. 1996;72:535–538. doi: 10.1136/pgmj.72.851.535. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Albanese MA, Mitchell S. Problem based learning: a review of literature on its outcomes and implementation issues. Acad Med. 1993;68:52–81. doi: 10.1097/00001888-199301000-00012. [DOI] [PubMed] [Google Scholar]
  • 3.Vernon DT, Blake RL. Does problem based learning work? A meta analysis of evaluation research. Acad Med. 1993;68:550–563. doi: 10.1097/00001888-199307000-00015. [DOI] [PubMed] [Google Scholar]
  • 4.Berkson L. Problem based learning: have the expectations been met? Acad Med. 1993;68:579–588. doi: 10.1097/00001888-199310000-00053. [DOI] [PubMed] [Google Scholar]
  • 5.Norman GR, Schmidt HG. The psychological basis of problem-based learning: a review of evidence. Acad Med. 1993;67:557–565. doi: 10.1097/00001888-199209000-00002. [DOI] [PubMed] [Google Scholar]
  • 6.Colliver JA. Effectiveness of problem-based learning curricula: research and theory. Acad Med. 2000;75:259–266. doi: 10.1097/00001888-200003000-00017. [DOI] [PubMed] [Google Scholar]
  • 7.Bennett NL, Davis DA, Easterling WE, Friedmann P, Green JS, Koeppen BM, et al. Continuing medical education: a new vision of the professional development of physicians. Acad Med. 2000;75:1167–1172. doi: 10.1097/00001888-200012000-00007. [DOI] [PubMed] [Google Scholar]
  • 8.David TJ, Dolmans DHJM, Patel L, van der Vleuten CPM. Problem based learning as an alternative to lecture-based continuing medical education. J R Soc Med. 1998;91:626–630. doi: 10.1177/014107689809101204. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Davis D, O'Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education. Do conferences, workshops, rounds, and other traditional continuing education activities change physician behaviour or health care outcomes? JAMA. 1999;282:867–874. doi: 10.1001/jama.282.9.867. [DOI] [PubMed] [Google Scholar]
  • 10.Sanci LA, Coffey CMM, Veit FCM, Carr-Gregg M, Patton GC, Day N, et al. Evaluation of the effectiveness of an educational intervention for general practitioners in adolescent health care: randomised controlled trial. BMJ. 2000;320:224–230. doi: 10.1136/bmj.320.7229.224. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Wolf FM, Shea JA, Albanese MA. Towards setting a research agenda for systematic reviews of evidence of the effects of medical education. Teach Learn Med. 2001;13:54–60. doi: 10.1207/s15328015tlm1301_11. [DOI] [PubMed] [Google Scholar]
  • 12.Maudsley G. Do we all mean the same thing by problem-based learning? A review of the concepts and formulation of the ground rules. Acad Med. 1999;74:178–185. doi: 10.1097/00001888-199902000-00016. [DOI] [PubMed] [Google Scholar]
  • 13.Barrows HS. A taxonomy of problem-based learning methods. Med Educ. 1986;20:481–486. doi: 10.1111/j.1365-2923.1986.tb01386.x. [DOI] [PubMed] [Google Scholar]
  • 14.Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practice and teach EBM. 2nd ed. London: Churchill Livingstone; 2000. [Google Scholar]
  • 15.Chan DH, Leclair K, Kaczorowski J. Problem-based small-group learning via the internet among community family physicians: a randomised controlled trial. MD Comput. 1999;16:54–58. [PubMed] [Google Scholar]
  • 16.Heale J, Davis D, Norman G, Woodward C, Neufeld V, Dodd P. A randomized controlled trial assessing the impact of problem-based versus didactic teaching methods in CME. Proc Annu Conf Res Med Educ. 1988;27:72–77. [PubMed] [Google Scholar]
  • 17.Doucet MD, Purdy RA, Kaufman DM, Langille DB. Comparison of problem-based learning and lecture format in continuing medical education on headache diagnosis and management. Med Educ. 1998;32:590–596. doi: 10.1046/j.1365-2923.1998.00277.x. [DOI] [PubMed] [Google Scholar]
  • 18.Benjamin EM, Schneider MS, Hinchey KT. Implementing practical guidelines for diabetes care using problem-based learning. A prospective controlled trial using firm systems. Diabetes Care. 1999;22:1672–1678. doi: 10.2337/diacare.22.10.1672. [DOI] [PubMed] [Google Scholar]
  • 19.Moran JA, Kirk P, Kopelow M. Measuring the effectiveness of a pilot continuing medical education program. Can Fam Physician. 1996;42:272–276. [PMC free article] [PubMed] [Google Scholar]
  • 20.Premi J, Shannon S, Hartwick K, Lamb S, Wakefield J, Williams J. Practice-based small-group CME. Acad Med. 1994;69:800–802. doi: 10.1097/00001888-199410000-00002. [DOI] [PubMed] [Google Scholar]
  • 21.Norman GR, Schmidt HG. Effectiveness of problem-based curricula: theory, practice and paper darts. Med Educ. 2000;34:721–728. doi: 10.1046/j.1365-2923.2000.00749.x. [DOI] [PubMed] [Google Scholar]
  • 22.Albanese MA. Problem-based learning: why curricula are likely to show little effect on knowledge and clinical skills. Med Educ. 2000;34:729–738. doi: 10.1046/j.1365-2923.2000.00753.x. [DOI] [PubMed] [Google Scholar]
  • 23.Cate ThJ ten. What happens to the student? The neglected variable in educational outcome research. Adv Health Sci Edu Theory Pract. 2001;6:81–88. doi: 10.1023/a:1009874100973. [DOI] [PubMed] [Google Scholar]
  • 24.Grol R, Jones R. Twenty years of implementation research. Fam Pract. 2000;17:S32–S35. doi: 10.1093/fampra/17.suppl_1.s32. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

[extra: Details of criteria used]

Articles from BMJ : British Medical Journal are provided here courtesy of BMJ Publishing Group

RESOURCES