Impact of the Foundations of Clinical Medicine Course on USMLE Scores

Abstract

BACKGROUND

The synthesis of basic and clinical science knowledge during the clerkship years has failed to meet educational expectations.

OBJECTIVES

We hypothesized that a small-group course emphasizing the basic science underpinnings of disease, Foundations of Clinical Medicine (FCM), could be integrated into third year clerkships and would not negatively impact the United States Medical Licensure Examination (USMLE) step 2 scores.

DESIGN

In 2001–2002, all third year students met weekly in groups of 8–12 clustered within clerkships to discuss the clinical and basic science aspects of prescribed, discipline-specific cases.

PARTICIPANTS

Students completing USMLE step 2 between 1999 and 2004 (n = 743).

MEASUREMENTS

Course evaluations were compared with the overall institutional average. Bivariate analyses compared the mean USMLE steps 1 and 2 scores across pre- and post-FCM student cohorts. We used multiple linear regression to assess the association between USMLE step 2 scores and FCM cohort controlling for potential confounders.

RESULTS

Students’ average course evaluation score rose from 66 to 77 (2001–2004) compared to an institutional average of 73. The unadjusted mean USMLE step 1 score was higher for the post-FCM cohort (212.9 vs 207.5, respectively, p < .001) and associated with step 2 scores (estimated coefficient = 0.70, p < .001). Post-FCM cohort (2002–2004; n = 361) mean step 2 scores topped pre-FCM (1999–2001; n = 382) scores (215.9 vs 207.7, respectively, p < .001). FCM cohort remained a significant predictor of higher step 2 scores after adjustment for USMLE step 1 and demographic characteristics (estimated coefficient = 4.3, p = .002).

CONCLUSIONS

A curriculum integrating clinical and basic sciences during third year clerkships is feasible and associated with improvement in standardized testing.

Electronic supplementary material

The online version of this article (doi: 10.1007/s11606-008-0631-z) contains supplementary material, which is available to authorized users.

INTRODUCTION

In 1989, the Robert Wood Johnson Foundation summarized expert observations on the state of US medical education highlighting, among other problems, the lack of integration of basic science knowledge into the third and fourth years of medical school. They designed “Preparing Physicians for the Future: A Program in Medical Education” to drive undergraduate curriculum change and issued an update report in 2002 outlining the results of grants made to facilitate this process. The report notes that the greatest successes occurred in the preclinical years with the introduction of clinical material into the classroom, but that changes to integrate basic science concepts into the clinical curriculum failed to meet expectations.¹

In 2001, after a decade of curricular reform in many medical schools, the AAMC Medical Objectives Project issued its fourth report on Contemporary Issues in Medicine highlighting the concerns of basic science educators. The authors noted that there were both fewer curricular hours being devoted to basic science education and less involvement of basic science departments in curricular design and implementation during a time in which new knowledge increased at an unprecedented rate. The panel emphasized “there has been insufficient attention paid to the integration of basic science content into the third and fourth years of the curriculum.”² Although the role of basic science knowledge in clinical skill development is not entirely understood, it is a subject of much interest and has been investigated by several authors.^3–5

There have been few successful examples of the reintroduction of the basic sciences during the clinical years. Croen et al. report on a well-received 2-month program for third year Yeshiva University medical students who had completed a year of clinical training, involving basic science and clinical faculty, designed to demonstrate the clinical perspective on the latest medical research.⁶ The University of Michigan Health System developed a fourth year medical student elective integrating clinical anatomy and women’s health with improved postintervention written test scores.⁷ Rudich and Bashan report on the success with a required selective for senior medical students in which students complete an extensive investigation of a clinical issue or question.⁸ O’Neill reports that problem-based learning students at the University of Manchester continue to set basic science learning objectives in their clinical years.⁹ Other authors have recently published descriptions of novel approaches to this problem including Fenton et al.¹⁰ at the University of California, San Francisco and Ogur et al.¹¹ for the Harvard Medical School, Cambridge integrated clerkship. However, these courses do not simultaneously integrate basic science concepts relevant to the clinical knowledge into required third year clerkships, nor do they assess the impact on standardized examinations such as the United States Medical Licensure Examination (USMLE) step 2.

To address this gap, we developed a new course, Foundations of Clinical Medicine (FCM), for all third year medical students with an overall goal of incorporating basic science concepts into the clinical clerkships through a series of case-based, student-led, small-group sessions with oversight by an attending within the clerkship rotation. We measured the acceptability of the course to students through course evaluations and hypothesized that the introduction of this course, in spite of the resulting substitution of lecture or clinical time for its completion, would not reduce students’ performance on the USMLE step 2 exam.

METHODS

Foundations of Clinical Medicine

FCM was developed by the faculty at the Medical University of South Carolina (MUSC) at the behest of the Dean’s Curriculum Coordinating Committee with the goal of integrating the basic and clinical sciences in a clinically relevant fashion while simultaneously fostering students’ continued self-directed learning and reasoning skills. All required third year clerkships participated with the exception of the Family Medicine/Rural clerkship. During the Family Medicine/Rural clerkship, students are off-campus working with family physician volunteer preceptors across the state and, for logistical reasons, a coordinated student small-group learning activity with the preceptors would be challenging. Apart from the Family Medicine/Rural clerkship, students rotate with on-campus faculty at the Medical University Hospital or affiliated Veteran’s Administration Medical Center during their required third year clerkships.

After approximately 18 months of development, FCM was introduced at the beginning of the 2001–2002 academic year and incorporated as an additional required learning activity into the clerkships. Over the subsequent years, the clerkships maintained their scheduled didactic activities, although some clerkships eventually reduced the number of faculty lecture hours and substituted the FCM course material as a portion of their didactic program. For the FCM course, students on each clerkship were divided into small groups of 8–12 and assigned to a specialty-specific preceptor for each rotation with whom they met for 60–90 minutes once per week. Over the course of a year, students spent between 38 and 57 hours (60 or 90 minutes × number of weeks per rotation × 6 rotations) participating in the new curriculum. Faculty preceptors were selected by the clerkship directors who gave them a brief course orientation. These preceptors included ward and clinic preceptors, clerkship directors, generalists, and subspecialists. No formal faculty development was conducted specifically for the FCM course.

A series of clerkship specific cases was developed by the FCM course directors and clerkship directors with each case followed by a series of discussion areas that included basic science and clinical topics. A syllabus was distributed to students with learning goals, references, and clinical cases relevant to the clerkship at hand (Online Appendix 1). Students were expected to prepare the cases and present a discussion during the small-group sessions and were graded pass/fail by their small-group preceptor based on attendance, preparation, contribution, thoughtfulness, and demonstration of appropriate level of knowledge (Online Appendix 2). No students failed the course in any of the study years.

Data Collection and Analysis

FCM course evaluations were distributed to students at the end of each clerkship. Course evaluation items consisted of 10 items standard to all institutional small-group courses. Students were asked to rate their level of agreement on a 5-point Likert scale for the following 10 items: (1) course objectives stated clearly, (2) the course was well-organized, (3) the syllabus was a valuable resource, (4) assignments were related to course goals and objectives, (5) group discussions were intellectually challenging, (6) assignments contributed significantly to learning, (7) grading system was adequately explained, (8) evaluation methods fairly reflected the course objectives and content, (9) course atmosphere encouraged student–faculty interaction, and (10) course format was appropriate.

To determine an overall course score, the mean score for each the 10 items is determined and are added together, then the total is multiplied by 2. The highest possible course score is 100. This process is standard across institutional courses and, therefore, provides a mechanism to compare 1 course performance to another. For the purposes of this study, course evaluation data were collected for the years 2001–2004 and compared with the overall institutional average.

To assess the impact of the course on USMLE step 2 scores, all MUSC medical students completing the USMLE step 2 between 1999 and 2004 were included in the study and were divided into pre-FCM (1999–2001; n = 382) and post-FCM (2002–2004; n = 361) cohorts. Demographic data and scores for the USMLE steps 1 and 2 tests were collected from each student’s record. Passing USMLE step 2 is required for graduation from MUSC, and all students take the exam during their fourth year of medical school. MUSC student pass rates for step 2 were also tracked across the years of each cohort.

Descriptive statistics, bivariate analyses, and multivariable linear regression were performed using SAS 9.1. Hypothesis testing was 1-sided for our unadjusted comparison of USLME steps 1 and 2 scores across FCM cohorts and alpha set at 0.025. One-sided hypothesis testing was chosen because of the noninferiority nature of our hypothesis. For variables with skewed distributions, median and interquartile range (IQR) instead of mean and standard deviation (SD) are reported. Student’s t test was used to compare the mean USMLE steps 1 and 2 scores across the 2 cohorts of students, pre- and post-FCM. To control for potential confounding, multiple linear regression was done to assess the association between USMLE step 2 and FCM cohort (pre- or post-FCM) controlling for known predictors of USMLE step 2 scores: USMLE step 1 score, age, race/ethnicity, and gender. This project was approved by the Medical University of South Carolina Institutional Review Board.

RESULTS

Students’ Characteristics

Table 1 presents the characteristics of the students in the pre- and post-FCM cohorts. The cohorts were similar with regards to demographic characteristics and mean USMLE step 1 scores.

Table 1 Description of Pre- and Post-FCM cohorts

Variable	Pre-FCM cohort 1999–2001 (n = 382)	Post-FCM cohort 2002–2004 (n = 358)	P value
Median age, years (IQR)	27 (25–29)	26 (25–29)	.32*
Gender, n (%)			.3†
Male	225 (59)	199 (56)
Female	157 (41)	159 (44)
Race, n (%)‡			.66§
Non-Hispanic, White	299 (78)	291 (81)
African-American	51 (13)	35 (10)
Hispanic	6 (1.6)	6 (1.7)
Asian||	25 (6.5)	25 (7)
Other¶	1 (0.3)	1 (0.3)
Mean USMLE step 1 score (SD)	207.5 (20.6)	212.9 (23.0)	<.001#
Mean USMLE step 2 score (SD)	207.7 (22.1)	215.9 (21.9)	<.001#

*Wilcoxon rank sum test was used to compare median values across nonnormally distributed variables with 2 categories.

†Chi square analysis was used to compare frequencies across categorical variables.

‡Percentages may not total 100% because of rounding.

^§Fisher’s exact test was used to compare small frequencies across categorical variables.

^¶Includes unknown and nonresident aliens. (Other)

||Includes Asian, Pacific Islander, and American Indians. (Asian)

^#One-tailedttests were used to compare means across cohorts with alpha set at 0.025.

FCM Course Evaluations

Students’ evaluations of FCM improved from an overall average score of 66 (2001–2002) to 71 (2002–2003) and 77 (2003–2004) over the study period. In comparison to the institutional small-group average course score of 73, course evaluation data indicates that by the end of our study period, the FCM course was well-received by students in comparison to similar small-group courses in the curriculum.

Student comments on the first set of evaluations could be grouped into 3 major themes: cases should be developed by the clerkships; cases should be more clinically relevant; and the basic science faculty members often contributed little to the discussion or lectured on material which was not germane. The course underwent revisions after the first year to address these concerns. Administration of the course was moved from the Dean’s office to the clerkships, and cases were revised by clinical and/or basic science faculty members within participating departments with oversight by the FCM course directors. Basic scientists attending sessions were recruited by each clerkship and, in many cases, were personally invited to participate by clinical faculty members with whom they had a previous relationship.

USMLE Step 2

All MUSC students completing USMLE step 2 between 1999 and 2004 were included in the analysis (n = 743). Students were divided into pre-FCM (n = 382) and post-FCM (n = 361) cohorts. Unadjusted mean USMLE step 1 score was higher for the post-FCM cohort compared to pre-FCM cohort (212.9 vs 207.5, respectively, p < .001; Table 1). Unadjusted mean USMLE step 2 score was significantly higher for the post-FCM cohort compared to the pre-FCM cohort (215.9 vs 207.7, respectively, p < .0001). Adjusting for USMLE step 1 score, age, race/ethnicity, and gender, the overall model of association between USMLE step 2 scores and FCM cohort achieved an R² of 0.54, p < .0001 (Table 2). FCM cohort (p = .0005), USMLE step 1 score (p < .0001), African-American race (p = .008), age (p < .0001), and gender (p < .0001) were each significantly associated with step 2 score after adjusting for the other variables in the model. Students in the post-FCM cohort scored 4 points higher on the USMLE step 2 after adjusting for the other variables. The pass rate for MUSC students taking the USMLE step 2 remained steady throughout the study period (91–93%) with the exception of a peak (96%) in 2002 (Online Appendix 3).

Table 2 Multivariate Analysis of USMLE step 2 Scores Across Pre-FCM (1999–2001) and Post-FCM (2002–2004) Cohorts

Variable	Estimated coefficient (SE)*	P value
FCM cohort (pre-FCM versus post-FCM)	4.02 (1.14)	<.001
Step 1 score	0.70 (0.03)	<.001
Race/ethnicity
African-American	−4.91 (1.85)	.008
Hispanic	−3.00 (4.46)	.56
Asian†	−1.33 (2.26)	.50
Other‡	−7.28 (15.32)	.64
Non-Hispanic, White	Reference	–
Gender
Female	5.25 (1.17)	<.001
Male	Reference	–
Age	−0.63 (0.15)	<.001

*Adjusted for the other variables listed in the table: cohort, step 1 score, race/ethnicity/gender, and age using multiple linear regression;R²  = 0.54, overall model F test, p < .0001.

†Includes Asian, Pacific Islander, and American Indians.

‡Includes unknown and nonresident aliens.

CONCLUSIONS

Although there have been mandates to integrate the basic sciences of medicine into the clinical curriculum, there are few examples of the successful implementation of such attempts in the literature.^6–9 The FCM course differs from others described in the literature in that it addressed the integration of basic science concepts relevant to the clinical knowledge simultaneously being acquired during the third year clerkships. Students’ ratings of course acceptability substantially improved throughout course implementation, a reflection of improved course organization and incorporation into clerkships. Student course evaluation data indicate satisfaction with a learning experience designed to integrate basic science knowledge into the clinical learning context and that a small-group learning format is acceptable.

To our knowledge, this is the first study to demonstrate an increase in USMLE step 2 scores concordant with the introduction of an integrated basic and clinical sciences third year course. Although the absolute increase in score may not be academically significant for an individual student or class, most importantly, there was no decrease in score for the cohort participating in the curricular innovation as initially feared by the faculty. These findings are noteworthy in that the test cohort of students substituted 38–57 hours of clinical time for the new, small-group curriculum during their third year of study. Whereas the role of basic science knowledge in clinical skill development has been described,^3–5 this study did not attempt a rigorous investigation of how basic science knowledge fosters such clinical knowledge development.

Our study has several limitations. The results are based on a single institution and historical controls were used. Over the course of the study, the first and second years began to incorporate clinically relevant material presented in small-group settings, and these changes may have contributed to higher USMLE scores. However, the curriculum during the third and fourth years was essentially unchanged except for implementation of the FCM course. Although our analysis accounted for 54% of the variability between pre- and post-FCM cohorts, there is the potential for unmeasured effects and secular trends, which could not be captured. Of note, 1999 was the first year of computerized testing for the USMLE, and the effect this may have had on scores at our institution is unknown. The mean national average for step 2 rose over the study period as well (Online Appendix 3).

Finally, we used USMLE scores as a proxy for knowledge acquisition in lieu of testing specifically designed to evaluate FCM course objectives. There is precedent for using USMLE scores to assess the impact of curricular change. Hoffman et al. used the USMLE step 1 and 2 scores as historical controls in assessing the impact of a new problem-based learning curriculum.¹² Similarly, Levine et al. compared the USMLE shelf scores in the years before and after curricular intervention.¹³ Several authors have employed USMLE step 1 scores as a control measure for test-taking ability among students receiving a curricular innovation.^11,14,15 However, future research should include the use of knowledge tests to evaluate specific curricular objectives, as well as rigorously studying the impact of this curriculum at other institutions in randomized controlled trials.

In conclusion, we found that a centrally developed curriculum integrating clinical and basic sciences in the third year of medical training is feasible, and although a substantial number of clinical experience hours were exchanged for small-group learning, USMLE step 2 scores did not drop and may have improved somewhat.