Developing Comprehensive Strategies to Evaluate Medical School Curricula

Abstract

Evaluation of medical school curriculum is important to document outcomes, effectiveness of learning, engagement in quality improvement, and to meet accreditation compliance. This monograph provides a roadmap and resource for medical schools to meaningfully evaluate their curriculum based on specific metrics. The method of evaluation includes an examination of Kirkpatrick’s levels of outcomes including reactions, learning, behavior, and impact. It is important that student outcomes are mapped in relation to curricular objectives. There are specific outcomes that may be utilized to determine if the curriculum has met the institution’s goals. The first is comparison to national metrics (United States Medical Licensing Examinations and American Association of Medical Colleges Graduation Questionnaire). Second, medical schools collect internal program metrics, which include specific student performance metrics, such as number of students graduating, attrition, and matching to specialty. Further, schools may examine student performance and surveys in the preclerkship and clinical phases (e.g., grades, failing courses, survey responses about the curriculum), including qualitative responses on surveys or focus groups. As the learning environment is critical to learning, a deep dive to understand the environment and mistreatment may be important for program evaluation. This may be performed by specifically examining the Graduation Questionnaire, internal surveys, and mistreatment reporting. Finally, there are numerous attitudinal instruments that may help medical schools understand their students’ development at one point or over time. These include measurements of stress, wellness, burnout, lifelong learning, and attitudes toward patient safety. Together, examining the composite of outcomes helps to understand and improve the medical school curriculum.

Background

Medical school curricular evaluation is important to document outcomes and determine effectiveness for the purpose of engaging in quality improvement of educational programs. In addition, evaluation is required for compliance with and accreditation for the Liaison Committee on Medical Education (LCME). As undergraduate medical education (UME) curricula have increased in complexity over the past decade, and the majority of medical schools are undergoing both small and large curricular changes, there is a need for careful evaluation of program outcomes [1]. Goal-directed evaluation is imperative to ensure that medical education program quality is maintained and that students achieve all medical education program objectives and are prepared to enter the health care system as residents while maintaining the LCME accreditation obligations.

The difficulty in determining and measuring outcomes that are most important poses a challenge. According to LCME, standard 8.4 Program Evaluation states, “A medical school collects and uses a variety of outcome data, including national norms of accomplishment, to demonstrate the extent to which medical students are achieving medical education program objectives and to enhance medical education program quality. These data are collected during program enrollment and after program completion [2].” While this LCME standard is critical and provides a sampling of common outcome measures to be considered, it does not provide guidance on the program objectives or specific outcomes to be evaluated. This standard enhances the challenge of determining the most relevant outcome measurements for a specific program’s evaluation.

Recognizing that evaluation of UME program objectives and outcomes is important, the purpose of this monograph is to provide a resource for medical schools to meaningfully evaluate their curriculum based on specific outcomes that help address LCME standard 8.4 and to provide meaningful feedback in support of curriculum planning decisions. It is important to remember that one main purpose for the evaluation is to demonstrate that medical students are achieving the education program objectives.

Evaluation Approach and Process

Evaluation is used to determine the effectiveness of a training program. In order to effectively assess the educational program through practical application, it is central to start with the end in mind and to be focused on the goals, objectives, and outcomes of the curriculum [3]. According to Kirkpatrick [4], there are three reasons to conduct an evaluation: (1) “to justify the existence and budget of the training program by showing how it contributes to the objectives and goals,” (2) “to decide whether to continue or discontinue training programs,” and (3) “to gain information on how to improve future training programs.”

Kirkpatrick described four levels (reactions, learning, behavior, impact) for evaluating medical education [4, 5]. These levels map directly to several LCME elements (Table 1). Participation includes the learners’ perspectives on the learning experience. Did the students find the educational program effective? Were they satisfied? Did they “like it”? Programs can also be evaluated through changes in attitudes toward the intervention. What was the students’ view of the learning environment? Knowledge relates to the acquisition of concepts, procedures, and principles, while skills relate to the acquisition of thinking, problem solving, psychomotor, and social skills. Behavior change documents transferable skills from the learning environment to the workplace or the willingness of learners to apply new knowledge and skills. Newer evaluation frameworks include the program’s impact on the system, patients’ health, and health system [3], but they are less common in medical school evaluation.

Table 1.

Kirkpatrick evaluation matched to LCME standards [2, 4]

Kirkpatrick level	LCME standard
Reaction	8.5 Medical student feedback
Learning	8.6 Monitoring of required clinical experiences Std 9 Supporting data on methods of assessments 9.5 Narrative assessment 9.7 Formative assessment and feedback
Behavior	9.4 Assessment system (direct observation/OSCE) 3.4 Learning environment/professionalism
Results	8.4 Program evaluation (especially measure obtained after program completion) 3.4 Learning environment/professionalism

As medical educators evaluate programs, it is essential to recognize that the education exists in and interacts with a system of education as well as a system of health care delivery [6]. Further, these systems are dynamic, requiring evaluators to consider the changes in performance requirements [7]. In a system-based evaluation, there is feedback at multiple levels, including the individual course, the program, and then the system-level outcomes (Table 2).

Table 2.

System-based evaluation framework for medical education [6]

Level of evaluation	Evaluation questions	Evaluation feedback
Level 1: Provides feedback about individual curriculum activities or stakeholders	Does the “glands and guts” course meet its stated learning objectives? Do an appropriate portion of our students match into their first residency?	Timely and specific feedback actionable at the course level
Level 2: Program performance	What is the relationship between the learning environment, performance, and well-being? What is the effectiveness of a new longitudinal patient safety course?	Identify root cause for variability in student performance and experiences in the program
Level 3: System readiness for future	How well are students prepared for internships? Do school objectives predict career success?	Evaluate longitudinal trends and alignment with national regulatory and best practices

The system-based learning evaluation model provides comprehensive and dynamic feedback to enable curriculum planners to make timely improvements to specific educational activities using level 1 evaluation, plan programmatic improvements which require more complex interventions using level 2 evaluation, and better anticipate attainment of long range programmatic and community health goals using level 3 evaluation

Outcomes

The key component for evaluation of a curriculum is the data that are used to measure or represent the outcomes; specifically, that students are achieving the educational objectives. To best evaluate a program, outcomes of a curriculum should be examined from multiple perspectives. Evaluation should address areas related to achievement of school curriculum program objectives, medical school learning environment, student well-being, and LCME accreditation elements. Data sources can also be analyzed in multiple ways to make different inferences about the program. Areas to consider include comparison to national norms, internal program metrics, specifically looking at learning environment, mistreatment, and student performance in the preclerkship phase and clinical phase. Methods of evaluation should be reliable, valid, feasible, and acceptable to appropriately measure the outcome of interest [8].

Utilization of both quantitative and qualitative data sources for program evaluation, such as surveys, focus groups or interviews, and longitudinal databases, provides a better understanding of how programs interact with individual differences to achieve outcomes. In addition, external data sources such as American Association of Medical College (AAMC)-sponsored surveys (e.g., Graduation Questionnaire), licensing exams, board certification information, and local organizational performance data can be used for evaluation of the curricula and identifying areas of strength and improvement. Data sources are used in combination to provide feedback for Kirkpatrick levels of evaluation outcomes [4, 5] and systems level 1, level 2, and level 3 evaluation [6] questions. This section will provide an extensive list of potential outcomes for program evaluation (Table 3).

Table 3.

List of potential outcome measurements

Category	Source	Measure
Comparison to national norms	USMLE Step 1 USMLE Step 2 CK USMLE Step 2 CS	Mean, standard deviation, and first-time pass rate Each has more detailed results by domains (e.g., communication on Step 2 CS)
	AAMC	Graduation Questionnaire (GQ) Second-year survey (Y2Q)
Internal metrics	Internal metrics	● Matriculated students who graduate ● Matriculated students who graduate in 4 years ● Attrition rate ● Number of students going on leave of absence ● Failure of courses (or incomplete) ● Students repeating courses (students repeating years) ● Dismissal or withdrawal ● Professional concerns ● Matching/not matching (SOAP) ● Matching to top 3 programs
	Admission metrics	● MCAT and GPA ● In-state residency ● Underrepresented in medicine ● Additional degrees ● Number of students applying/interviewing, offer admission, and accepting admission (yield)
Learning environment and mistreatment	Various measures Clerkship evaluation Graduation Questionnaire (GQ)	● Medical student learning environment survey (supportiveness) ● Burnout ● Wellness or distress instruments ● Emotional climate ● Student-faculty interaction ● Official mistreatment reporting ● GQ mistreatment reports ● Internal mistreatment surveys
Internal performance	Student performance	● Grades or scores in courses or clerkships ● Evaluation of graduates from interns and program directors ● Number of grades in each category, number of students failing clinical course of clerkship
Internal evaluation	Student surveys	Student course evaluations Evaluations of faculty Annual surveys

Comparison to National Norms

Utilizing data from national norms or benchmarks helps medical educators compare their students’ performance to students from other schools. These comparisons fall into two categories: (1) USMLE (United States Medical Licensing) examinations and (2) the AAMC Graduation Questionnaire. Standardized examinations such as USMLE Step 1, Step 2 CK, and CS (Clinical Skills) can be useful to determine performance. The mean, standard deviation, and first-time pass rate are important metrics to understand relative performance of a cohort and the overall level of success in passing. The summary reports from the National Board of Medical Examiners (NBME) also provide figures that indicate which areas of the exam the students were above or below the national mean, including the standard deviation intervals. Therefore, a school may look at the Step 1 result figure and note that the students were well above the mean in the nervous system but below in the cardiovascular system. Likewise, with USMLE Step 2 CS, the school is provided a figure indicating the comparison of the students’ aggregate performance in Integrated Clinical Encounter and Communication and Interpersonal Skills. These types of comparisons help to identify areas of strength and weakness of the curricula. Along with the USMLE Step 1 and 2 scores, schools may use NBME basic science or clinical (shelf) subject examinations or comprehensive examinations such as the Comprehensive Basic Science Self-Assessment or the Comprehensive Basic Science Exam. Examining these scores as part of program evaluation is useful to know if the UME program meets this metric.

The second, common national benchmark is the Association of American Medical Colleges Graduation Questionnaire (GQ) and Y2Q (second-year survey) [9]. These documents provide not only the trends of school data, but also provide benchmarking data to compare your school to other AAMC-member schools. The difficulty with using these surveys for program evaluation is that the GQ report is over 50 pages long and contains 60 questions, most with several parts, making it cumbersome to examine the entire GQ. Starting in 2016, AAMC has published a GQ Supplementary Benchmarking Report which provides percentile benchmarking for 17 of the highest profile questions, providing schools with more information to learn where they fall on the continuum of responses. Using the benchmarked data can identify areas of concern where school numbers were low compared to other schools or compared to internal school areas of focus such as inter-professional education, service learning, or basic science integration. Analyzing this national benchmarked data can also flag items which may indicate an area of potential LCME non-compliance.

Internal Program Metrics

Internal program metrics are vital outcomes to be examined. Some examples are percentage of matriculated students who graduate at all or in 4 years, attrition rate, number of students going on leave of absence, repeating courses/clerkship/academic year, dismissed, and receiving professionalism concerns. Some schools evaluate their programs with admission metrics such as MCAT, premedical grade point average, in-state residency, underrepresented in medicine, and additional degrees, as well as number of students that apply and interview, are offered and accept admission (yield). Other outcomes include students not matching either before or after the Supplemental Offer and Acceptance Program and students’ report of matching within their first three choices. Another measurement of a curriculum is asking interns and their program directors about the performance of the graduates.

Qualitative Data

Qualitative data includes open-ended question responses to provide insight into the views of the students. It is common practice to include various open-ended questions on internal surveys. Further, the GQ includes a report of comments to open-response items. These can be reviewed verbatim or aggregated into summary themes. Additional information may be collected with student interviews or focus groups using a specific structured methodology [10].

Learning Environment and Mistreatment

Partly in response to the LCME attention, but also with the recognition that the learning environment is important, there is increased scrutiny on these metrics as part of evaluation [11]. Measurement of the learning environment can include clerkship evaluations completed by students, end of year surveys, the GQ, and the Y2Q. In addition, there may be other measurements, such as official mistreatment reporting or other surveys of the learning environment, such as the Medical Student Learning Environment Survey [12, 13].

Preclerkship Outcomes

There are specific measurements that can be looked at from the preclerkship phase. These include course performance means, number of students failing each course, student course evaluations, and course reviews on the part of a curriculum committee. For schools that have grades, metrics might include the proportion of grades. Some schools will have annual surveys of the students asking them to evaluate the program with questions like, “How well did your preclerkship curriculum prepare you for clerkships?” When using these measures to evaluate the preclinical phase, it may be helpful to compare the GQ data while recognizing that students are looking back 2–4 years to their preclerkship phase. In addition, the AAMC launched the Y2Q survey to collect information about the preclinical phase, but for many schools, the response rate limits the utility of this report.

Clinical Outcomes

Similarly, there are specific measurements for the clinical phase, including clerkships and the fourth year rotations. Clerkship, acting internships, and other course performance may be reported. This might include the numbers of grades in each category (honors/high pass/pass/fail or A/B/C/D/F). In addition, metrics should include number of students failing clerkships or other sub-internships or electives. Other measures of performance, such as direct observation of clinical skills, chart reviews, and standardized patient cases, may also provide behavioral outcomes. Other metrics include student reactions to the curriculum including student course evaluations, evaluations of faculty, and annual surveys. Likewise, the GQ and the national benchmark of NBME shelf examinations are often useful.

Attitudinal Scales

There are a variety of scales that measure individual differences related to medical student outcomes (Table 4). Some have not been used in the medical student population and need additional validity evidence, while others have reasonable validity evidence. When choosing a scale, as with other outcomes, it is important to align the construct of the scale with the outcome of the curriculum. Some examples of possible scales for our curriculum are provided below. Administrators should keep in mind that some instruments are proprietary and require payment while others do not. Most importantly, ensure the scales reflect the attitudes supported by the curricular goals, objectives, and outcomes.

Table 4.

Possible attitudinal instruments

Attitudinal survey/questionnaire instruments
Measure	Description
World Health Organization Well-being Index (WHO-5)	5 items that measure well-being and asks respondents how they felt over the past 2 weeks [14]
Perceived Stress Scale (PSS)	10 items that assess the degree to which situations are appraised by individuals as stressful [15]
Emotion Regulation Questionnaire (ERQ)	10 items that measure an individual’s strategies for regulating emotions across two subscales: cognitive reappraisal and expressive suppression [16]
Medical Student Well-being Index (MSWBI)	7 items used to identify medical students who may be experiencing severe psychological distress [17]
Maslach Burnout Inventory (MBI)	22 items that measure burnout through 3 subscales: emotional exhaustion, personal accomplishment, and depersonalization [18]
Brief Resilience Scale (BRS)	6 items that measure the ability to bounce back from stressful situations [19]
Connor Davidson Resilience Scale (CD-RISC)	25 items that measur resilience, and a score is generated. Scores range from 0 to 40 [20]
Medical School Learning Environment Scale (MSLES)	17 items that capture the learning environment across 6 subscales: breadth of interest, student-student interactions, supportiveness, meaningful learning experiences, organization, and vertical integration. Rosenbaum et al. (2007) used MSLES specifically for medical students [12, 13]
Attitudes Toward Patient Safety Questionnaire (APSQ)	26 items that measure attitudes around patient safety across 9 subscales: Patient safety training received, error reporting confidence, working hours as error cause, error inevitability, professional incompetence as cause, disclosure responsibility, team functioning, patient involvement, and importance of patient safety in curriculum [21, 22]
Jefferson Scale for Lifelong Learning (JeffSPL)	14 items that measure attitudes toward lifelong learning across 3 subscales: learning beliefs and motivation, lifelong learning skills, and attention to learning opportunities. The scale was originally designed for physicians [23], but was adapted for use among medical students (JeffSPL-MS) [24]
Jefferson Scale of Empathy (JSE)	20 items designed to assess empathy in patient-care and medical school contexts across 3 subscales: perspective taking, compassionate care, and walking in a patient’s shoes [25]
Tolerance for Ambiguity (TFA)	7 items that measure an individual’s ability to deal with uncertainty in learning or performance contexts [26]
NEO Five-Factor Inventory (NEO-FFI)	A shortened version of the total 5-factor personality inventory looking at 5 core personality traits: extraversion, neuroticism, openness to experience, agreeability, and conscientiousness. The shortened version contains 60 items, while the full personality inventory (NEO-PI) contains 240 items [27]

Student well-being is a very important metric given the high rates of burnout in the medical profession [28]. The most common instrument is the Maslach Burnout Inventory [18]. The World Health Organization’s Well-being Index [14] is used in general populations globally [29]. Another measurement of distress is the Medical Student Well-being Index [17, 30], which looks at identifying medical students experiencing severe distress and includes items from emotional exhaustion burnout [18], depersonalization burnout [18], depression [31], mental quality of life (QOL) [32], physical QOL [32], stress [15], and fatigue [33]. The Perceived Stress Scale [15] measures individual stress levels, while the Emotion Regulation Questionnaire [16] assesses individual strategies to regulate their emotions. Other scales measure resilience, such as the Brief Resilience Scale [19] and the Connor Davidson Resilience Scale [20].

There are also various measures that assess medical student attitudes or perceptions around different domains, such as patient safety, the learning environment, and lifelong learning. The Attitudes toward Patient Safety Questionnaire [21, 22] is a possible measure if patient safety is an important outcome of your curriculum. The Medical School Learning Environment Scale [12, 13] is used to measure students’ perceptions of the learning environment throughout different phases of the curriculum. The Jefferson Scale for Physician Lifelong Learning [23] is used to measure attitudes around lifelong learning among physicians, and a student version has been developed [24]. The Jefferson Scale of Physician Empathy measures empathy and has also been modified for use among medical students [25].

Some medical schools are interested in personality or other student traits. Instruments might include the NEO personality inventory (NEO-PI) or the shortened NEO Five-Factor Model (NEO-FFI) [27], and Tolerance for Ambiguity [26].

Aligning Outcomes with Program Objectives

The preceding section described numerous potential outcome measurements for UME program evaluation. First, it is important to map curricular objectives and assessments to the outcomes of the curriculum. For example, one objective might be to graduate students who are excellent in functioning in teams. Thus, the school may use Entrustable Professional Activity (EPA) 13, “Collaborate as a member of an inter-professional team,” as an acceptable outcome of the program. In this case, program evaluation will examine the outcomes related to this objective, including assessments of students, the GQ, as well as students’ attitudes and reactions on evaluation surveys. Each school has specific educational objectives that students must meet, and through careful alignment and examination of multiple data sources, program evaluators can determine if the program objectives are met.

Another key process during evaluation is moving from data (numbers) to meaning. Haeckel’s hierarchy describes relationships between facts/data, which are simply observations, to information which provides context to the data [34]. Then, as information is aggregated, analyzed, and used to synthesize theories, data becomes knowledge and wisdom. Similarly, the data in program evaluation become meaningful as relationships are examined. For example, the data of students’ scores on USMLE Step 1 become meaningful when examining the relationship between MCATs, preclerkship examination scores, Step 1 study time, and student burnout. Therefore, the role of program evaluation is to align data and information with objectives to create knowledge of the strengths and weakness of the program.

The Systematic Program Evaluation Process

Once all of the data is gathered, there needs to be a systematic method for interpretation, analysis, decision making, and action items. We describe the program evaluation retreat at Virginia Commonwealth University School of Medicine as one example. At this yearly retreat, key members from the curriculum council, curriculum office, and the office of assessment, evaluation, and scholarship gather to review all available data, which is organized into preclinical and clinical curriculum outcomes. The clinical and preclinical groups specifically examine outcomes, as mapped to program objectives, along with attitudinal measures, external benchmarked data, and internal program metrics. For each phase of the curriculum, a strengths, weaknesses, opportunities, and threats (SWOT) analysis is completed [35]. By reviewing each phase in relation to overall program objectives, a SWOT analysis for the overall curriculum was thus developed. From this, we develop an action plan for curriculum improvement with prioritization of short-term, intermediate, and long-term goals. This is presented to the full curriculum council, the executive committee of the school of medicine, and a faculty meeting. Most importantly, we track the action plan metrics throughout the year to ensure that issues are addressed. An example of an identified opportunity for improvement was when three course directors were mistakenly under the impression the other course directors were covering cholesterol metabolism, a gap that occurred due to the integrated nature of the curriculum. This gap was identified in the review and placed in the proper order in the curriculum.

Program evaluation of medical school curricula is crucial, but it can be a daunting challenge. This monograph offers a variety of approaches, frameworks, methods, instruments, and metrics that can be utilized to address LCME element 8.4. First, choose the most important outcomes and ensure that they are aligned with program objectives. This should be completed by a curriculum committee or other organizing body with responsibilities for developing and meeting the goals of the curriculum. Then, a strategy for analyzing and reviewing data sources should be developed that aligns each analysis with specific interpretations that can be made about the curriculum. For example, a longitudinal analyses of individual differences on a patient safety attitude questionnaire may be interpreted to indicate whether the curriculum is strengthening patient safety attitudes or knowledge over time, while a comparison to national benchmarks on patient safety attitudes would be interpreted as the relative success of students in that domain overall compared to other schools.

This monograph outlines examples of some metrics and data sources that can be used to represent and evaluate curriculum. Finding adequate data sources to represent program objectives is only the first step in obtaining meaningful interpretation and insight about the effectiveness of the curriculum. It will be imperative for medical school program evaluation to move from reporting summary data toward elucidating more meaningful relationships between the curriculum and individual learners so that interventions for improvement can be planned.

The amount of data available and complexity of analyses used to interpret data will continue to grow with time, making the resources and expertise needed to extract meaningful interpretation ever more important for medical schools. Consultation with educational theorists, program evaluators, and experts in data analysis will be important to realize the value of the data for improving the quality of medical school curricula.

Conclusion

Curricular evaluation and improvement cannot be done overnight. Using the various strategies highlighted in this monograph may help in implementing proactive processes for evaluation of your medical school curriculum and taking steps to improve it. This monograph outlines many important considerations when designing your school’s program evaluation, grounded in Kirkpatrick’s levels, and meeting the needs of LCME accreditation elements.

Funding Information

Dr. Santen receives funding for the evaluation work related to the Accelerating Change in Medical Education Grant from the American Medical Association.

Compliance with Ethical Standards

Conflict of Interest

The author declares that they have no conflict of interest.