Prolonged Delays for Research Training in Medical School are Associated with Poorer Subsequent Clinical Knowledge

Abstract

Background

Complementary degree programs and research training are important alternative tracks in medical school that typically interrupt the traditional MD curriculum.

Objective

Examine effects of such a break on clinical knowledge after reentry into the MD curriculum.

Design

Retrospective cohort study.

Participants

Three hundred and two graduates of Mayo Medical School.

Main Measurements

Compared years of delay between the second and third years of medical school with third year clerkship grades, National Board of Medical Examiner’s (NBME) Subject Examinations, and United States Medical License Exam (USMLE) Step 2.

Main Results

258, 13, and 31 students spent 0, 1, or ≥3 years pursing research between the second and third year. Baseline measures of knowledge before matriculation and before the third year were similar between groups. Whereas a 1-year delay had no significant effect, a ≥3-year delay was associated with fewer clerkship honors and lower NBME Medicine, Pediatrics, and Psychiatry percentiles compared to no delay (all p < .05). Students with a ≥3-year delay had a 77% reduction in the odds of honors in Medicine. For each year of delay beyond 3, students’ third-year NBME Medicine, Neurology, Obstetrics and Gynecology, and Psychiatry scores decreased as did USMLE Step 2 scores (r = −.38 to −.50, p < .05).

Conclusions

Delays of ≥3 years between the second and third years of medical school are associated with lower grades and scores on clinical knowledge tests. Further research is needed to determine the optimal timing of research training and develop effective interventions to facilitate reentry into the medical school curriculum.

INTRODUCTION

Research training and complementary degree programs are important alternative tracks in the medical school curriculum. For example, medical students report that conducting research during medical school leads to the acquisition of knowledge and skills^1–4 and stimulates research interests.^3,5–7 Research during medical school has also been associated with academic career and rank,^8,9 and M.D.–Ph.D. students often have successful, productive research careers as measured by academic appointments, grants, and publications.⁹ As such, opportunities to develop career-specific skills during medical school meet the needs of society.¹⁰

Acknowledging the benefits, the Association of American Medical Colleges (AAMC) encourages medical schools to offer a flexible curriculum that enables students to pursue joint degrees and research training programs.¹⁰ Many schools agree with the AAMC’s position and offer research opportunities on campus or at the National Institutes for Health and M.D.-Ph.D. dual degree programs, along with advanced degrees in Health Administration, Public Health, Business Administration, and Law.¹¹ Traditionally, students pursue these specific career training programs after completing the first 2 years of medical school and taking the United States Medical Licensing Exam (USMLE) Step 1 examination and before starting the third year of medical school. Despite their incredibly high value, the present formats of these tracks interrupt the medical school curriculum designed to be integrated from years 1 through 4. The effect of breaking from the traditional medical school curriculum to pursue additional training on subsequent clinical knowledge has not been addressed.

The goal of the present study was to investigate whether a break between the end of the second and beginning of the third year of medical school to pursue research or a complementary degree compromises subsequent clinical knowledge of participating students relative to their peers. In a retrospective cohort study, we tested the following 3 hypotheses: (1) students with no delay, a delay of 1 year, and a delay of ≥3 years between the second and third years of medical school differ with respect to clerkship grades and performance on National Board of Medical Examiners (NBME) Subject Examinations and USMLE Step 2; (2) the degree of negative effect of a delay on performance correlates with the length of delay; and (3) the associations of delay and length of delay with performance persist after controlling for USMLE Step 1 score and undergraduate grade point average (UGPA).

METHOD

Study Population

Medical students who graduated from Mayo Medical School (MMS) between 1997 and 2004 were included in the study. The study population included students enrolled in the traditional 4-year curriculum, students who participated in 1 year of research through the Howard Hughes Medical Institute/National Institutes of Health Cloister program or the Sarnoff Endowment program, and students enrolled in the M.D.–Ph.D. joint degree program.

We excluded students enrolled in the M.D./Oral and Maxillofacial Surgery (M.D./O.M.S.) program and students enrolled longer than 4 years due to leaves of absence (LOA) for personal reasons or because of academic difficulties.

The Mayo Clinic Institutional Review Board reviewed the study proposal and considered the study exempt.

Data Collection

For each student we obtained the year of MMS enrollment and graduation. If the time period between enrollment and graduation spanned >4 years, the student’s record was reviewed to determine the reason for the prolonged enrollment and the length of time between the end of the second year and the beginning of the third year. Baseline measures of knowledge obtained for students before medical school and before the third year included: UGPA; Medical College Admission Test (MCAT) total and subscale scores (i.e., Biological Science Score, Physical Science Score, Verbal Score); NBME Introduction to Clinical Diagnosis (ICD) and Pediatric Subject Examination second-year scores; and USMLE Step 1 scores. To evaluate clinical knowledge in the third year and beyond, we obtained Medicine, Neurology, Obstetrics and Gynecology (Ob-Gyn), Pediatrics, Psychiatry, and Surgery clerkship grades (i.e., Honors, High Pass, Pass, Marginal Pass, or Fail) as well as corresponding NBME Subject Examinations scores and USMLE Step 2 scores. The NBME Subject Examination scores were converted to national percentiles.^12,13 NBME Subject Examinations contain retired items from the USMLE examination that are selected based on content outlines developed by committees of subject experts.¹⁴ These examinations are commonly used as external standardized tests to assess educational achievement within a content domain.

Statistical Analysis

Students were separated into the following 3 groups depending on their delay between the end of the second year and the beginning of the third year: no delay (i.e., enrolled in the traditional 4-year curriculum); 1 year delay (i.e., participated in 1 year of research); or ≥3 years delay (i.e., M.D.–Ph.D. joint degree program). Because of small sample sizes in the 2 delay groups and skewed distributions for some measures, nonparametric statistical procedures were used. In the case of multiple linear regression, this was implemented using a rank transformation of the dependent variable.¹⁵

Kruskal–Wallis tests were used to determine whether there were statistically significant differences in UGPA, MCAT scores, total number of clerkship honors, NBME Subject Examination percentiles, and USMLE scores among the 3 groups. For variables where there was a significant difference among the 3 groups, pairwise Wilcoxon rank sum tests were used to determine whether there were statistically significant differences between 2 groups: no delay versus 1 year delay; no delay versus ≥3 years delay; and, 1 year delay versus ≥3 years delay between the second and third year. In each clerkship, the proportion receiving honors was compared between groups using chi square tests. Logistic regression was used to explore relationships between receiving clerkship honors and having a curriculum delay while adjusting for USMLE Step 1 score and UGPA. Among students with ≥3 years delay between the second and third year, Spearman’s Rank correlations (r) were calculated between the length of delay and NBME Subject Examination percentiles and total number of clerkship honors; multiple linear regression models were used to assess the significance of the findings while adjusting for USMLE Step 1 scores and UGPA.

RESULTS

Between 1997 and 2004, 328 students graduated from MMS. Of these students, 26 were excluded for being MD-OMS students (13) and having a LOA (13). Demographic characteristics and overall academic performance of the 302 included students are shown in Table 1. These students had enrolled between 1989 and 2000. Thirteen students spent approximately 1 year (mean 1.02 ± .11 years) doing research and 31 students spent ≥3 years (mean 4.23 ± .79 years) pursuing their PhD between the second and third years of medical school. Two students pursued research training and 1 student worked on a MPH for 1 year between the third and fourth years of medical school. As these 3 students did not have a delay between the second and third year, they were analyzed along with the other 255 students who proceeded directly into the third year. Thus, 258 students were categorized as having no delay, 13 students as having a 1 year delay, and 31 students as having a ≥3-year delay between the second and third year.

Table 1.

Demographic Characteristics and Academic Performance Measures of 302 Mayo Medical School Graduates, 1997–2004

Variable	Medical students N = 302
No. (%) of males	150 (49.7)
Mean years (SD) of enrollment	4.5 (1.2)
No. (%) of students enrolled more than 4 years	47 (15.6)
No. (%) of students with a delay between 2nd and 3rd year	44 (14.6)
No. (%) of students with a delay between 3rd and 4th year	3 (1)
Mean (SD) UGPA	3.7 (.3)
Mean (SD) MCAT Score	33.0 (3.9)
Mean (SD) MCAT VR	10.7 (1.5)
Mean (SD) MCAT PS	11.1 (1.8)
Mean (SD) MCAT BS	11.2 (1.6)
Mean (SD) USMLE Step 1 Score	222.8 (17.7)

Abbreviations: SD, standard deviation; UGPA, undergraduate grade point average; MCAT, Medical College Admission Test; MCAT VR, verbal score; MCAT PS, physical science score, MCAT BS, biological science score; USMLE, United States Medical Licensure Exam

All students had retrievable UGPA. Two-hundred and ninety (96%) students had analyzable MCAT scores; 228 (75%) students had NMBE ICD scores; 209 (69%) students had second-year NMBE Pediatric scores; and, all but 1 student had allowed reporting of their USMLE Step 1 score to the medical school. As shown in Table 2, there were no differences in these measures between groups.

Table 2.

Undergraduate and First and Second Year Medical School Academic Performance among 302 Mayo Medical School Graduates, 1997–2004, by 0, 1, and ≥3 Years between 2nd and 3rd Year

Variable	Years between 2nd and 3rd year*			P value**
	0 Year Mean (SD)	1 Year Mean (SD)	≥ 3 Years Mean (SD)
UGPA	3.74 (.30)	3.67 (.15)	3.72 (.28)	.15
MCAT	32.98 (3.94)	33.25 (4.75)	32.9 (3.70)	.88
MCAT VR	10.77 (1.48)	10.50 (1.68)	10.29 (1.68)	.37
MCAT PS	11.02 (1.80)	11.50 (1.83)	11.38 (1.77)	.44
MCAT BS	11.19 (1.62)	11.25 (1.82)	11.24 (1.67)	.97
NBME ICD	78.66 (20.91)	84.23 (17.57)	79 (17.22)	.47
NBME Peds, Year 2	23.70 (19.80)	23.67 (27.74)	32.40 (24.20)	.53
USMLE Step 1	229.59 (17.30)	224.42 (24.55)	224.03 (17.33)	.18

*Number of years between the end of the second year and the start of the third year of the medical school curriculum

**Kruskal–Wallis was used to calculate the p value.

Abbreviations: SD, standard deviation; UGPA, undergraduate grade point average; MCAT, Medical College Admission Test; NBME ICD, National Board of Medical Examiners Introduction to Clinical Diagnosis score adjusted to national percentile; NBME Peds, Year 2, National Board of Medical Examiners Pediatric Subject Examination, administered in year 2, score adjusted to national percentile; USMLE, United States Medical Licensure Exam.

Third-year clerkship grades, most NBME subject examination scores, and USMLE Step 2 scores were each available for ≥99% of the students; the exception was the NBME Clinical Neurology Subject Examination, which only 264 (87%) students had taken. In all clerkships, students with honors had higher mean NBME shelf scores (all p < .001). These students also had higher USMLE Step 2 scores (all p < .001). Total number of clerkship honors and NBME shelf examination scores correlated moderately with USMLE Step 2 scores (r² = .33–.50, p < .001).

In contrast to the lack of differences found before the third year, significant differences became apparent between the groups when analyzing performance during year 3 (Table 3). There were significant differences in NBME Medicine, Pediatric, Psychiatry, and Surgery percentiles (all p < .05) after adjusting for USMLE Step 1 and UGPA with students who delayed the third year by ≥3 years consistently having the lowest mean score. These students also consistently received lower percentiles on Clinical Neurology and Ob-Gyn Subject Examination; however, these differences did not reach statistical significance. Differences in third-year clerkship grades were also found between the groups. Students with a ≥3-year delay had fewer total clerkship honors (p = .04) and were less likely to have honors in Medicine and Surgery (p < .05). In contrast, there was no apparent difference in USMLE Step 2 scores between groups. Pairwise comparisons are shown in footnotes to Table 3.

Table 3.

Third and Fourth Year Academic Performance among 302 Mayo Medical School Graduates, 1997–2004, by Years between Second and Third Years of Medical School

Clerkship		NBME Subject Examination			Clerkship honors
	Years between 2nd and 3rd year*	Mean (SD)	P value†	P value adjusted	No. (%)	P value	P value adjusted
Medicine	0	66.92 (24.99)	<.001§	<.001‡	78 (30)	.04§	.03§
	1	71.92 (24.22)			5 (39)
	>3	45.77 (26.99)			3 (10)
Neurology	0	67.80 (25.17)	.23	.12	106 (41)	.63	.54
	1	66.91 (28.55)			7 (54)
	>3	56.59 (30.93)			12 (39)
OB-Gynecology	0	58.37 (27.87)	.18	.09	77 (30)	.70	.54
	1	68.00 (23.42)			5 (39)
	>3	50.48 (30.49)			8 (26)
Pediatrics	0	59.60 (26.27)	.001§	<.001§	73 (28)	.18	.24
	1	62.42 (24.58)			4 (30.8)
	>3	39.67 (25.59)			4 (13)
Psychiatry	0	57.75 (27.01)	.02§	.01¶	103 (40)	.22	.33
	1	68.31 (22.69)			6 (46)
	>3	44.60 (28.08)			7 (24)
Surgery	0	60.78 (26.30)	.06**	.02¶	97 (38)	.01¶	.01¶
	1	71.08 (30.67)			9 (69)
	>3	51.93 (28.85)			7 (23)
		USMLE Step 2			Total clerkship honors
		Mean (SD)	P value	P value adjusted	Mean (SD)	P value	P value Adjusted
	0	227.94 (18.80)	.37	.37	2.07 (1.77)	.04§	.02¶
	1	229.42 (23.34)			2.77 (1.96)
	>3	223.45 (23.20)			1.41 (1.76)

*Number of years between the end of the second year and the start of the third year of the medical school curriculum

**Only 1 vs 3 pairwise comparison significant (p < .05)

^†Kruskal–Wallis and Multiple Linear Regression used to calculate the unadjusted and adjusted p values, respectively.

^‡All pairwise comparison (i.e., 0 vs 1, 1 vs ≥3, 0 vs ≥3) were significant (p < .05)

^§Only 1 vs ≥3 and 0 vs ≥3 pairwise comparisons were significant (p < .05)

^¶Only 0 vs 1 and 1 vs ≥3 pairwise comparisons were significant (p < .05)

Consistently, students who delayed the third year by ≥3 years performed worse than their peers without such a delay. A ≥3-year delay resulted in a 77% reduction in the odds of receiving honors in Medicine (Odds ratio .23; 95% confidence interval .05, .82; p = .04). Similarly, ≥3 years delay was associated with decreased odds of honors in Pediatrics, Psychiatry, and Surgery; however, these findings did not reach statistical significance.

The length of time students with a ≥3 year delayed the third year ranged from 3.2 to 6.3 years. As shown in Table 4, with each year delay beyond 3 years, NBME Medicine, Neurology, Ob-Gyn, and Psychiatry percentiles declined significantly(r = −.38–50, p < .05). Although scores on the USMLE Step 2 was similar between groups, a detrimental impact was seen with each additional year of delay beyond 3 years (r = −.43, p = .02). While adjusting for USMLE Step 1 score and UGPA, most correlations remained significant. No relationship was found between total number of clerkship honors and length of time away beyond 3 years.

Table 4.

Correlation between Number of Years Third Year was Delayed and Subsequent Academic Performance among 31 Mayo Medical Graduates with 3 or More Years between the Second and Third Year of Medical School

	Length of delay Spearman’s rank correlation coefficient	P value*	P value adjusted
No. of clerkship honors	−.31	.1	.29
NBME Subject Examinations
Medicine	−.40	.03	.05
Neurology	−.50	.01	.02
Obstetrics and Gynecology	−.43	.02	.05
Psychiatry	−.38	.04	.13
Pediatrics	−.31	.1	.19
Surgery	−.31	.1	.16
USMLE Step 2†	−.43	.02	.03

*Unadjusted p value is testing whether the Spearman rank correlation is significantly different from 0 and the adjusted p values is from the multiple linear regression.

^†USMLE, United States Medical Licensure Exam

DISCUSSION

Research training during medical school is vital to ensuring the ongoing prosperity of the U.S. research industry and getting physician-scientists off the “endangered species list.”¹⁶ We were reassured to find that delaying the third year by 1 year did not have a significant negative influence on subsequent performance. In contrast, there seems to be adverse consequences to longer delays between the second and third year. Students who delayed the third year by ≥3 years had a 77% decreased odds of earning honors in Medicine, and performed poorer on NBME Medicine, Pediatrics, and Psychiatry Subject Examinations than their peers without such a delay, despite similar UGPA, MCAT scores, and second-year NBME Subject Examination percentiles. With each additional year beyond 3, NBME Medicine, Neurology, Ob-Gyn, and Psychiatry percentiles and USMLE Step 2 scores decreased with a degree of change correlating moderately with the length of time away.

Our findings of worsened academic performance after a ≥3-year delay between the second and third year is likely to have practical significance. Assessment is integral to education. Measures of academic performance are used to judge if students are meeting education goals,¹⁷ guide student learning,^18,19 identify areas for curricular revision,^17–19 ensure competency,^19,20 and select residents.^21–27 Grades in required clerkships and total number of honors are considered among the most important measures of academic achievement by program directors in a wide variety of specialty areas.²⁸ In particular, honors in Medicine and Surgery are of high value for competitive residencies.^{24, 27} In the past, the importance of USMLE Step 2 scores in residency selection has been reported to be moderate to low,^25,28,29 but today residency program directors are often requiring students to submit Step 2 scores before the match.

Criteria for granting interviews and making decisions regarding ranking of applicants for the match may differ between traditional MD-only graduates and graduates who have invested time in obtaining additional research training. The degree of influence lower academic performance ultimately has on students’ ability to match to their residency of choice likely depends on multiple factors such as the competitiveness of the residency, the perceived likelihood that the candidate will pursue further research, the candidate’s number of publications, and how the residency selection committee values the intangible benefits of the research experience. Although we could find no publication specifically addressing residency program directors’ interest in M.D.-Ph.D. graduates, we were struck by the low value assigned to research experience³⁰ and publications^25,28,30 by surveyed program directors across a variety of specialty domains. However, M.D.-Ph.D. graduates, especially those of Medical Scientist Training Programs (MSTP), typically pursue fellowship and or postdoctoral research training and have successful, productive research careers.⁹ As such, one could speculate that MSTP resident applicants may be highly valued by residency selection committees.

Clerkship grades may, however, be a marker of performance to come. In a recent BEME systematic review of predictive values of medical school performance and future performance, clerkship honors and grades correlated positively with supervisor rating during residency.³¹ Similarly, Taylor et al. reported that medical school GPA was the best predictor of residency directors’ ratings of interns.³² Others have found clerkship honors to predict performance on the American Board of Specialty Examination³³ along with residency performance.^23,27,34,35 Whether lower grades and fewer clerkship honors obtained by students with a prolonged delay between the second and third year foreshadow suboptimal future competence warrants further study.

NBME subject examination scores played a pivotal role in all of the clerkship’s grading equations, as it commonly does.¹⁴ Given the multiple factors that are likely taken into account when clerkship grades are assigned,^18,36–38 NBME subject examination scores may provide better insight into students’ educational achievement by the end of each clerkship than assigned grade. Paralleling the negative impact of a long delay seen on grades was a similar effect on several NBME subject examination percentiles, suggesting that by the end of the clerkship students with a long delay may not have mastered the subject material to the same extent as students without such a delay. Given the association between NBME subject examination percentiles and USMLE Step 2 scores in our study and in others,^39–41 there may be a critical threshold of delay between years 2 and 3 beyond which students struggle to catch up with their peers. This is supported by our finding that each extra year beyond 3 students delayed the third year there was a reduction in USMLE Step 2. As USMLE Step 2 scores correlate with clinical performance^32,42 and in-training examinations during residency^33,43 as well as USMLE Step 3 scores^31,44 students with a substantial length of delay between the second and third years of medical school may be at risk for not recovering clinical knowledge by their intern year. This possibility is substantiated by Androile et al.’s report that MD-PhD graduates from their institution had lower USMLE Step 3 than their peers who had graduated from the regular MD curriculum (219 vs 227, p = .02),⁴⁴ an outcome not surprising given the high correlation between USMLE Step 2 and Step 3 (r = .72).³¹ In summary, these findings highlight a potentially negative impact of breaking from the traditional curriculum.

Our study is limited by several factors. First, the generalizability of these results from a single Midwestern medical school to other regions of the country is unknown. Although students in this study had higher MCAT scores and UGPA than typical of 1993–2000 matriculating medical students,⁴⁵ USMLE Steps 1 and 2 scores were similar to national averages.⁴⁶ Second, there is a selection bias for the M.D.-Ph.D. program. Although the baseline measures of academic performance were similar between groups, students who enroll in the M.D.-Ph.D. program or choose to do research may be different from students who pursue the traditional medical school curriculum. The differences found may reflect differences in clinical aptitude or interest level between students who invest significant time conducting research and their peers. Students engaged in research may not be as interested in learning clinical medicine (i.e., they may have no intention to practice medicine) or may be distracted by research and other commitments. Finally, this study is limited by its retrospective nature. We cannot determine whether the delay between the second and third years is causally related to the competency measures investigated. Other variables (e.g., children) may have arisen during the absence that subsequently influenced academic performance. Returning students may also be finishing manuscripts or continuing their research and, as a consequence, may not be devoting adequate time to clinical responsibilities and studying.

Our study has several important strengths. To our knowledge, this is the first study investigating the influence of delaying the third year on subsequent academic performance. Second, we assessed academic performance using both clerkship grades and nationally administered tests, allowing comparisons to other students. Also, NBME Subject Examinations and USMLE Step exams are considered valid tools for evaluation of medical education programs.⁴⁷ Third, students’ undergraduate performance and their performance on second-year NBME Subject Examinations were similar, suggesting the study sample used was relevant to evaluating the effects of delaying the third year of medical school. Despite all the inherent variables (e.g., weeks in a clerkship, variable timing of exams), the decrement in year 3 performance was consistently present among students who delayed the third year by ≥3 years.

In summary, despite the benefits of rigorous research training in joint degree programs, delaying the medical school curriculum by ≥3 years is associated with fewer honors and lower scores on national examinations. The magnitude of this effect on NMBE scores increases with the duration of the curricular gap, suggesting that students may benefit from ongoing clinical experiences or, at least, a “reentry” curriculum to brush up on clinical knowledge. Further research is needed to determine the optimal timing to break from the traditional curriculum to pursue additional training, identify factors that place students at risk for lower clinical knowledge upon their return, and establish if there is a critical length of time students can be gone before negative consequences are seen. Hypothesis-driven, prospective multicenter studies with well-constructed clinical knowledge and skill assessment throughout the continuum of undergraduate to graduate education are likely needed to provide valid, generalizable information on these research questions. A major barrier to organizing such research is limited institutional support and funding,⁴⁸ and this problem must be remedied.