Associations Between United States Medical Licensing Examination (USMLE) and Internal Medicine In-Training Examination (IM-ITE) Scores

Abstract

Background

Little is known about the associations of previous standardized examination scores with scores on subsequent standardized examinations used to assess medical knowledge in internal medicine residencies.

Objective

To examine associations of previous standardized test scores on subsequent standardized test scores.

Design

Retrospective cohort study.

Participants

One hundred ninety-five internal medicine residents.

Methods

Bivariate associations of United States Medical Licensing Examination (USMLE) Steps and Internal Medicine In-Training Examination (IM-ITE) scores were determined. Random effects analysis adjusting for repeated administrations of the IM-ITE and other variables known or hypothesized to affect IM-ITE score allowed for discrimination of associations of individual USMLE Step scores on IM-ITE scores.

Results

In bivariate associations, USMLE scores explained 17% to 27% of the variance in IME-ITE scores, and previous IM-ITE scores explained 66% of the variance in subsequent IM-ITE scores. Regression coefficients (95% CI) for adjusted associations of each USMLE Step with IM-ITE scores were USMLE-1 0.19 (0.12, 0.27), USMLE-2 0.23 (0.17, 0.30), and USMLE-3 0.19 (0.09, 0.29).

Conclusions

No single USMLE Step is more strongly associated with IM-ITE scores than the others. Because previous IM-ITE scores are strongly associated with subsequent IM-ITE scores, appropriate modeling, such as random effects methods, should be used to account for previous IM-ITE administrations in studies for which IM-ITE score is an outcome.

INTRODUCTION

Medical knowledge is one of the six competencies required of residents by the Accreditation Council for Graduate Medical Education (ACGME).¹ Standardized exams are well suited to the assessment of medical knowledge. The Internal Medicine In-Training Examination (IM-ITE) is given each October to over 18,000 residents enrolled in internal medicine residencies in the US.² This validated multiple choice examination was developed jointly by the American College of Physicians (ACP), the Association of Program Directors of Internal Medicine (APDIM), and the Association of Professors of Medicine (APM).³ Minimum scores on the IM-ITE are associated with passing scores on the American Board of Internal Medicine Certifying Examination (ABIM-CE).^4–8

The United States Medical Licensing Examination (USMLE) was developed by the Federation of State Medical Boards (FSMB) and the National Board of Medical Examiners (NBME).⁹ These well-validated multiple choice examinations are given in three steps designed to assess skills for basic sciences, supervised medical care, and medical care without supervision, respectively.⁹ Although the exact timing can vary, trainees usually take USMLE Step 1 (USMLE-1) after the 2nd year of medical school with Step 2 (USMLE-2) taken in the 4th year of medical school,¹⁰ and Step 3 (USMLE-3) taken during the 1st year of post-graduate training. Passing all three steps of the USMLE is usually required for medical licensure in the US.^9,11,12

Many medical education research studies have used scores on these standardized examinations as outcome measures.^13–20 The associations of previous USMLE Step scores on subsequent USMLE Step scores and IM-ITE scores are important to correctly account for these examinations in studies of educational outcomes. The purpose of this analysis was to evaluate these associations.

METHODS

Subjects and Study Design

The Internal Medicine Residency, Mayo Clinic, Rochester, MN, trains approximately 48 residents per class in three sequential years of categorical internal medicine residency. All 195 categorical internal medicine residents enrolled in this residency during the 2002 and 2003 administration of the IM-ITE formed the cohort for this study. This cohort was identified for a previous study designed to assess associations of medical knowledge acquisition with IM-ITE scores.¹⁸ Because the IM-ITE is given yearly, a total of 421 IM-ITE scores from the years 2000 through 2003 were available for this analysis (Table 1). USMLE-1 scores were available for all 195 residents who took the IM-ITE. Due to the timing of the other USMLE Steps, scores for these examinations were not available for all residents. USMLE-2 scores were available for 186 residents, and USMLE-3 scores were available for 89 residents (Table 2).

Table 1.

Number of Residents Per Class Who Took the Internal Medicine In-Training Examination (IM-ITE) During Each Year*

Year of IM-ITE administration	Class of 2003 (n = 48)	Class of 2004 (n = 49)	Class of 2005 (n = 52)	Class of 2006 (n = 46)	Total IM-ITE administrations
2000	44	–	–	–	44
2001	48	44	–	–	92
2002	47	48	48	–	143
2003	–	49	47	46	142
All years					421

*All 195 residents were encouraged to take the exam each year, but not all could do so due to absences (e.g., away elective, illness, transfer)

Table 2.

Scores on United States Medical Licensing Examination (USMLE) Steps 1, 2, 3 and Internal Medicine In-Training Examination (IM-ITE) for Total Cohort

	Number of residents	Number of test administrations	Mean (SD)	Range
USMLE-1*	195	195	230 (15.3)	187 to 266
USMLE-2*	186	186	232 (16.5)	191 to 267
USMLE-3*	89	89	222 (13.8)	182 to 253
IM-ITE†	195	421	65.5% (8.97)	38% to 88%

*USMLE scores are reported as a standardized scaled score with minimum passing score of 180

†IM-ITE scores are reported as total percent items correct. Residents are encouraged to take the IM-ITE each year of internal medicine residency, thus the number of test administrations is greater than the number of residents

Data Analysis

Graphic and tabular displays were used to assess the data for patterns and trends. Linear regression techniques were used to assess the associations between USMLE Step scores. Because the multiple scores available for each resident on the IM-ITE were likely autocorrelated, a random effects model was used for assessment of IM-ITE score as a dependent variable.^18,21,22 This model also allowed for adjustment for multiple known and hypothesized predictors of IM-ITE score including: conference attendance in the year prior to the IM-ITE, use of UpToDate in the year prior to the IM-ITE, post-graduate year (PGY), United States vs. International Medical Graduate status (USMG vs IMG), National Residency Matching Program rank (NRMP), Objective Structured Clinical Examination score (test of physical examination skill), class rank (ordinal ranking based on residency rotation grades), age at entry into residency, gender, marital status, and parenthood (whether or not a resident had a child at the time of any IM-ITE administration). The specifics of each type of data, their collection, and their individual associations with IM-ITE scores were described previously.¹⁸ Regression coefficients, which represent the expected change in score on the subsequent test for each unit change in score on the previous test, and R-squared, which represents the percent of the variance in each score accounted for by the previous test performance, were used to describe bivariate associations of subsequent test scores vs previous test scores.

All analyses were conducted by two authors (FSM and SLZ) using Stata 8.2 (StataCorp LP, College Station, TX). Results were considered statistically significant if p-values were less than or equal to 0.05. The study protocol was prospectively approved by the institutional review boards of the Mayo Clinic and the Johns Hopkins Bloomberg School of Public Health. Residents had the opportunity to withhold their data from analysis. All elected to be included in the study.

RESULTS

The association of previous USMLE scores on subsequent USMLE Step scores had correlation coefficients (R-values) ranging from 0.47 to 0.59 (p<0.001). Consequently, the percent of variance (R-squared) accounted for by previous USMLE Step scores ranged from 22% to 35% with the associations gaining in strength with the proximity of one test to another (i.e., USMLE-1 is more predictive of USMLE-2 than of USMLE-3): regression coefficients (R-squared) for USMLE-2 vs. USMLE-1 0.63 (0.35); USMLE-3 vs. USMLE-1 0.43 (0.22); USMLE-3 vs. USMLE-2 0.49 (0.35).

The associations of USMLE Step scores on IM-ITE scores were statistically significant (p<0.001) with the percent variance in IM-ITE scores accounted for by USMLE scores ranging from 17% to 27%: regression coefficients (R-squared) for IM-ITE vs. USMLE-1 0.27 (0.21); IM-ITE vs. USMLE-2 0.29 (0.27); IM-ITE vs. USMLE-3 0.25 (0.17). Previous IM-ITE scores explained 66% of the variance in subsequent IM-ITE scores (regression coefficient 0.75, R-squared 0.66). Assessment of the bivariate associations of all USMLE steps and IM-ITE scores stratified by PGY yielded similar results.

Regression coefficients (95% CI) for the random effects multivariate associations of each USMLE Step with IM-ITE scores were USMLE-1 0.19 (0.12, 0.27), USMLE-2 0.23 (0.17, 0.30), and USMLE-3 0.19 (0.09, 0.29). These associations were adjusted for all 11 variables listed in the methods

DISCUSSION

Investigators who examine the merits of educational initiatives or interventions are tempted to utilize the performance on standardized examinations as an outcome. The IM-ITE is one of the largest standardized post-graduate in-training exams in use. This is the first published study to assess IM-ITE score associations with all three USMLE Step scores appropriately adjusted for previous IM-ITE scores and multiple known and hypothesized predictors of IM-ITE outcome. Our results emphasize the need to take into consideration prior performance when using test scores as an educational outcome.

The percent variance in IM-ITE scores accounted for by USMLE Step scores ranges from 17% to 27%. However, when adjusting for multiple predictors known or hypothesized to affect IM-ITE scores, the regression coefficients for the individual steps of the USMLE on the IM-ITE are quite similar with broadly overlapping confidence intervals. This implies that for multivariate analyses using IM-ITE score as the outcome variable, the use of a single USMLE Step score (e.g., USMLE Step 1) is sufficient to account for the effect of all USMLE Steps on IM-ITE scores. This is important since USMLE Step 1 is almost universally available to residency programs at the time of medical student application to residency, while the other USMLE Steps are not.

As expected, the association of previous IM-ITE score on subsequent IM-ITE score is strong with previous IM-ITE scores accounting for 66% of the variance in subsequent scores. Except for our previous work,¹⁸ we were unable to identify other studies that accounted for prior test performance on subsequent performance.^{14–17,20,23–30} Moving forward, studies that use ITE scores as an outcome should adjust for prior performance on longitudinal repeated measures with appropriate modeling, e.g., random effects methods, as used in this study.^18,21,22

Similar to other reports,^13,31 the association of previous USMLE Step scores on subsequent USMLE Step scores in our study was strong (R ranged 0.47 to 0.59), and the percent of variance (R-squared) accounted for by previous USMLE Step scores ranged from 22% to 35%. Thus, for educational studies that use USMLE Step scores as an outcome, it is important to account for previous scores. However, factors other than previous USMLE Step scores are influencing the variance in these subsequent scores, and these associations should be explored further. The stated purposes of USMLE Steps 1, 2, and 3 are to assess skills and knowledge in the basic sciences, supervised medical care, and independent medical practice, respectively.⁹ These different emphases may account for some of the differences in variances between the individual Steps of the USMLE.

The association of previous test scores on USMLE performance has also been demonstrated. The associations between Medical College Admission Test (MCAT) scores and USMLE Step scores have correlation coefficients of 0.61, 0.49, and 0.49 for USMLE Steps 1, 2, and 3, respectively.³² Donnon et al. recently published a meta-analysis of the predictive ability of the MCAT showing similar results related to USMLE Steps.³³ The association between individual USMLE Steps was not reported in either study. It is notable that the correlation coefficients we report for the effect of previous USMLE Step scores on subsequent USMLE Step scores is similar to those previously reported for MCAT scores. Perhaps this is due to the similar nature of these multiple-choice high-stakes examinations in general or to the similarity of their content in particular.

In addition to the association of MCAT with USMLE scores, the association between IM-ITE performance and subsequent ABIM-CE passage rates has been demonstrated, albeit using much smaller sample sizes.^4–8 Our study, viewed in context of these previous studies, helps to establish the continuum of association between MCAT, USMLE Steps, IM-ITE, and ABIM-CE.

There are limitations to this study. It is a retrospective cohort analysis from a single institution. This institution has a very high caliber of trainee as noted by the high mean scores on each examination (Table 2). However, the range of scores on each examination span the gamut from very low to very high, so inferences based on this study for scores within these ranges would be appropriate.

USMLE Step scores are often used by residency programs in their admissions criteria. Knowing how the scores of the individual USMLE Steps relate to one another and to the more specific assessment of internal medicine knowledge as determined by the IM-ITE is important to educators. Given our results, we would encourage the developers of the USMLE and the IM-ITE, as well as the ABIM-CE and Re-certifying Examination, to assess the associations of these scores using the entire cohort of learners who take these tests annually. This would further our understanding of inter-test relationships of medical knowledge along the continuum of medical training and allow educators to better interpret factors that may be influencing current performance in relation to prior test scores. Until this is done, analyses such as those presented in our study are important for proper assessment of these tests in educational outcome studies.