Research in medical education: three decades of progress

The specialty of research in medical education began just over three decades ago with a small group of clinicians and educational researchers at the medical school in Buffalo, New York. Since that time it has expanded worldwide. This paper is a personal reflection on how this research has informed our understanding of learning, teaching, and assessment in medicine.

Summary points

• Research in medical education has contributed substantially to understanding the learning process

• The educational community is becoming aware of the importance of evidence in educational decision making

• Areas of major development include basic research on the nature of medical expertise, problem based learning, performance assessment, and continuing education and assessment of practising physicians

Measuring progress

In medicine, indicators of scientific progress might be measured by objective indicators such as death from cardiovascular disease. In education such “hard” evidence may be lacking for several reasons. Firstly, paradoxically, real differences in educational strategies may not be reflected in outcomes, such as licensing examination performance, simply because students are highly motivated and are not blinded to the intervention, so will compensate for any defects in the curriculum.^1–5 Secondly, a curriculum is not like a drug, which can be given at standard doses, but instead contains many components, delivered with variable quality by different teachers. Finally, the time between learning and important outcomes may be so long that the effects of the curriculum are obscured—although not always.⁶

Use of evidence in educational decision making

Perhaps the most important evidence of progress in the discipline is that we are now more likely than before to demand evidence to guide educational decision making. Before 1970 important educational advances were largely adopted by persuasion and politics; since that time changes are more likely to be initiated or accompanied by evidence. Although this may sound circular, it represents tangible recognition of the contributions that research can make to the practice of education.

Specific areas of progress

Beyond this cultural change, the following broad domains have seen real progress: basic research in the acquisition of expertise, problem based learning, advances in assessment methods, and continuing education, recertification, and relicensure.

Basic research in the acquisition of expertise

In the early 1970s basic research into the nature of clinical reasoning pursued the hypothesis that expert clinicians were distinguished by the possession of general “clinical problem solving” skills. This was wrong; what emerged was that expertise lay predominantly in the knowledge, both formal and experiential, that the expert brought to the problem.⁷,⁸

This finding resulted in a new direction of inquiry, and a new generation of researchers attempted to uncover the ways that expert clinicians organise medical knowledge in their minds, using research strategies derived from cognitive psychology.⁹,¹⁰ Although the fruits of these labours are not yet ripe, the research has moved from purely descriptive research to experimental studies directed at a better understanding of the process and theory based interventions that promise to improve the effectiveness of instruction.¹¹

Problem based learning

Problem based learning developed at McMaster University in the late 1960s, driven by a desire to construct a medical school that was more humane than one that used the traditional, lecture based approach. Since that time an extensive body of evidence has emerged about its effectiveness.^2–5,¹² If the evaluation is restricted to the central educational outcomes such as performance on licensing examinations, few differences are found.²,³ This should not be a surprise—most students will do whatever is necessary to compensate for any perceived weakness in a curriculum.¹³ However, in terms of the original goal of creating a humane learning environment, problem based learning is an unqualified success.³,⁴

Advances in assessment methods

Arguably the most dramatic advances have occurred in approaches to assessment. Thirty years ago assessment was dominated by written tests—essays in Europe and multiple choice questions in North America. When performance assessment did occur, as in specialty certification, the traditional viva dominated. Moreover, there was a paucity of evidence about the strengths and weaknesses of various approaches.

That has all changed. In the 1970s there was a proliferation of simulation approaches to assessing higher skills like problem solving. These developments were accompanied by careful research, largely stimulated by the licensing and certification bodies in the United States. Content specificity has been a major finding of this research—the correlation of the various measures of skills across problems was typically in the range of 0.1 to 0.3,¹⁴ so many samples of behaviour were necessary to obtain stable, thus reliable, measurement. It is a direct consequence of content specificity that the objective structured clinical examination,¹⁵ with its multiple samples of performance, has come to dominate performance assessment and has led to an extensive literature regarding the impact of various elements such as ratings of simulated patients versus physician observers.

Additional educational resources

Useful websites

• Queen's University, Ontario, Canada (http://meds.queensu.ca/medicine/pbl/pblhome.htm)—problem based learning home page from Queen's University's School of Medicine

• Clerkship Directors in Internal Medicine Task Force Subgroup Report (http://www.im.org/cdim/5educate/eval/clinical.html)—evaluation of clinical competence

• National Board of Medical Examiners (http://www.nbme.org/about/itemwriting.asp)—constructing written test questions for the basic and clinical sciences

Useful publications

• Norman GR, Schmidt HG. The psychological basis of problem-based learning: a review of evidence. Acad Med 1992;67:557-65.

• Norman GR, van der Vleuten CPM, Newble DI. International Handbook of Research in Medical Education. Dordrecht: Kluwer, 2002.

• Swanson DB, Norman GR, Linn RL. Performance based assessment: lessons from the health professions. Educ Res 1995;24:5-11.

This research has led to major advances in performance assessment—for example, the Medical Council of Canada now administers a performance examination to 1800 licensure candidates each year.¹⁶ Changes in assessment methods at the school level have, however, been much slower in coming.¹⁷

Research foci and major findings

Basic research on reasoning

Generic reasoning skills are non-existent

Knowledge (formal and experiential) is a critical determinant of reasoning

Problem based learning

Self directed learning does not result in lower knowledge

Students and teachers are happier and more satisfied

Performance assessment

Multiple sampling strategies are crucial to reliable, valid assessment

Performance can be assessed as well as knowledge

Continuing education

Improved reliable methods for performance assessment in practice

Systematic approaches to relicensure

Continuing education, recertification, and relicensure

While continuing education of health professionals remains dominated by the “day in medicine,” when physicians assemble and hear a full day of lectures on a particular topic from academic specialists, a substantial body of evidence has emerged pointing to the deficiencies of this approach, and, more importantly, identifying alternative methods that are effective.¹⁸

These changes have paralleled dramatic changes in the assessment of practising physicians. Society is challenging the presumed right of independent practice conferred on the physician at the time of licensure, and the medical education community has responded by devising and implementing several defensible strategies for reassessment. One highly innovative approach is the use of unidentified standardised patients who enter physicians' practices undetected.¹⁹,²⁰ Another is the serious attempt to identify and deal with incompetent physicians through formal performance assessment both in Canada and in the United Kingdom.²¹,²²

The outcome of medical education

Research in medical education is no longer in its infancy (see box). While it remains multifocal, with nearly as many research methods as researchers, there is a growing body of knowledge about the process and outcome of medical education.

In my view there has been insufficient attention to the interface between our understanding of clinical expertise and the application of this knowledge to improve instruction and assessment. Studies at the curriculum level of complex “treatments” like problem based learning are unlikely to reveal a deeper understanding of the interface between instruction and learning. Real improvement in education, just like real improvements in medical treatments, will only result when we combine better the understanding of basic science with the experimental interventions.

Figure.

TOPHAM PICTUREPOINT

Medical education, 1960s-style