Medical Education Curricula

Education, n. One of the few things a fellow is willing to pay for and not get.

—Leonard L. Levinson

The paper by Kollas¹ in this issue of JGIM describes a curriculum for legal medicine and thus is an example of many curricula that address practical and vital educational needs for practicing physicians. It is typical of the renaissance of modern medical education. A changing health care system, burgeoning biomedical knowledge, and an aging population are only a few of the forces driving the expansion of competencies sought by our students and residents. In the last 3 years, new learning objectives have been published for students by the Association of American Medical Colleges as the Medical School Objectives Project,² for core clerkships in internal medicine by the Society of General Internal Medicine and Clerkship Directors in Internal Medicine,³ and for internal medicine residency programs by the Federated Council for Internal Medicine.⁴ New technologies, such as the virtual reality of multimedia, medical informatics, and distance education, are also revolutionizing instructional designs. Any professional meeting of medical educators these days is energized by discussion of innovations and educational strategies being tested in their institutions.

Paradoxically, the extent of this curricular activity is not reflected in the peer-reviewed literature, the classic method for disseminating new ideas. Even JGIM, despite a call for curricular papers,⁵ has published few articles in this area. The lack of publication of new curricula has become a barrier itself to further curricular development. Faculty participants in our annual curriculum development workshop rarely find published curricula in their area of interest and are usually left to develop curricula de novo. Because educational programs share many of the same goals and aims, I imagine that many groups of faculty are concurrently designing new curricula, unaware of one another's efforts.

One explanation for this situation is that peer-reviewed journals such as JGIM have an obligation, not only to disseminate, but also to advance a field like medical education by publishing only those educational interventions that have met criteria for having proven significant results. Faced with a similar conundrum, the British Medical Journal recently published guidelines for evaluating papers on educational interventions.⁶ Examples of criteria used to critique an educational evaluation are shown in Table 1. These criteria, like criteria for other types of research, address the issues of internal and external validity of the results; that is, did we measure what we thought we measured?

Table 1.

Criteria For Reviewing Evaluations of Curricula^*

External validity (generalizability)

Are the methods described in sufficient detail so that the evaluation is replicable?

Is the evaluation question clear? Are independent and dependent variables clearly defined?

Are dependent variables meaningful and congruent with the rationale of the curriculum? (e.g., Is performance measured instead of competence, skill instead of knowledge, when those are the desired or most meaningful effects?)

Are the measurement instruments described or displayed in sufficient detail?

Do the measurement instruments seem to possess face validity? Are they congruent with the evaluation question?

Internal validity

Are raters blinded to the status of learners? Have interrater and intrarater reliability been assessed?

What is the likelihood that the evaluation would detect an effect of the desired magnitude?

Are statistical methods appropriate for the type of data collected?

^*Adapted from Table 5.3 of Kern et al.⁷

Unfortunately, most curricular papers fail to meet such rigorous standards of scholarship in their evaluation designs. The reasons are understandable. Most efforts to develop curricula are underfunded, and faculty rarely have enough resources for both the educational intervention and a sophisticated evaluation. As a result, true experimental designs in medical education are rarely used. Even nonrandomized controlled evaluations of educational interventions are difficult to perform because the numbers of learners may be small and the interventions may be implemented over time periods long enough to introduce confounding by maturation or contamination by other learning experiences. Medical educators charged with curriculum development may be well trained in content, but unskilled in program evaluation.

The very nature of curriculum development further confounds attempts at curricular evaluation because curriculum development is a uniquely dynamic enterprise when compared with medical research. The most effective curricula are those that are learner centered, not teacher or “curriculum centered,” and those that change learning environments and the learners' experiences. Experiential learning may be the most effective method for adult learners, but it is also the most difficult to quantitate and control. Most medical education curricula are iterative and change with each cohort of learners who experience the curriculum. The fluid nature of well-designed curricula makes evaluation problematic.

Despite these challenges, medical educators are not exempt from evaluations of their curricula. To justify the cost of change, learners, patients, and society expect that our interventions will improve education and help learners achieve the competencies pertinent to their profession. We have not always responded well. For example, basic tenets of modern medical education, such as the need to shift to ambulatory training sites, and the importance of early clinical experiences, owe their prominence more to enthusiasm than to unbiased evaluations. Before we commit to such changes in our training programs, we should document the effectiveness of those changes.

In fact, many barriers to effective evaluation can be addressed with advanced planning, such as searching for previously validated measurement instruments and working with pre-experimental designs.⁸ Use of multiple measurement methods over time is another technique to improve the quality of an educational evaluation. Although few curricula will meet all of the criteria listed in Table 1, curriculum developers should at least attempt to address some of these criteria when developing their evaluation plans and anticipate any threats to their evaluations.

Among the most powerful tools medical educators have to improve the quality of their evaluations are their professional collaborations. The multi-institutional curricular effort can be used to increase learner numbers, increase the power to measure an effect, and address issues of generalizability. Individual institutions should be considered pilot laboratories for curricula, which can then be shared with collaborators and disseminated only after the principal evaluation question has been answered, “Have our learners achieved the objectives of the curriculum?”