Investigating the Relationship Between a Clinical Science Composite Score and USMLE Step 2 Clinical Knowledge and Step 3 Performance

Abstract

Previous research has found a moderate relationship between performance on individual clinical science subject examinations and USMLE performance. Given the widespread use of the clinical science subject examinations and the need for measures of clinical knowledge that help predict performance on Steps 2 CK and 3 and performance in residency training, this study explores the use of composite scores based on clinical science subject examinations to predict clinical knowledge outcome measures. The data set included students who took all of the five most widely used clinical science subject examinations (medicine, obstetrics and gynecology, pediatrics, psychiatry, and surgery) between January 1, 2013, and December 31, 2017 (N = 65,516). Composite scores were calculated based on average equated percent correct scores across various combinations of clinical science subject examinations. Stepwise linear regression analyses were performed with composite score and Step 1 score as predictor variables and Step 2 CK score or Step 3 score as the dependent variable. In all cases, the proportion of variance explained (R²) by the composite score (0.62–0.65 for Step 2 CK score and 0.45–0.48 for Step 3 score) was greater than R² for Step 1 by itself (0.52 for Step 2 CK score and 0.37 for Step 3 score). Logistic regression analyses found that higher composite scores were associated with a greater probability of passing Steps 2 CK and 3. Composite scores can be used alone or in conjunction with Step 1 to identify students at risk of failing Step 2 CK and/or Step 3 to facilitate remediation.

Introduction

The National Board of Medical Examiners® (NBME®) clinical science subject examinations are primarily designed to assess performance at the end of traditional block clerkship rotations [1]. Most medical schools in the USA use several of the NBME clinical science subject examinations and incorporate scores as one component of final clerkship grades, with a weight generally ranging from 11% to 40% [2]. Previous research has shown that there is a moderate positive relationship between performance on individual clinical science subject examinations and performance on United States Medical Licensing Examination® (USMLE®) Step 1, Step 2 Clinical Knowledge (CK), and Step 3 [3–5]. This finding is expected given that the subject examinations are developed using the same rigorous process as USMLE and item formats are similar to those used in USMLE [1].

In one study, Zahn et al. found moderate positive correlations of individual clinical science subject examination scores with USMLE Step 1 and Step 2 CK scores as well as correlations of 0.69 and 0.77, respectively, between a composite score based on the average of subject examination scores across six core clerkships and Step 1 and Step 2 CK scores [3]. In another study, Dong et al. found moderate positive correlations of subject examination scores with USMLE Step 3 scores and also found that poor performance on one or more subject examinations resulted in a higher probability of failing Step 3 [4]. Similarly, Ryan et al. found moderate positive correlations of subject examination scores with USMLE Step 1 and argued that Step 1 scores could be used to identify and remediate students at risk of performing poorly on the subject examinations [5]. Each of these studies was based on students from a single medical school.

USMLE Step 1 assesses whether an examinee understands and can apply important concepts of the sciences basic to the practice of medicine [6]. Students typically take Step 1 at the end of a foundational curriculum in medical school. Because Step 1 scores are widely used by residency program directors to screen applicants, some stakeholders have highlighted the unintended consequences that this use of Step 1 scores has on medical schools and medical students [7–10]. Some have suggested that Step 1 scores should not be reported, with only a pass/fail decision being provided [8, 10]. Others have cautioned that eliminating scores on USMLE is likely to have different unintended consequences [7, 9]. Given the current national conversation about the use of Step 1 scores by residency programs and the impact on medical student learning and well-being [7–10], there exists a need for standardized summative measures of knowledge obtained during medical school other than USMLE Step 1 that provide information about a student’s clinical science knowledge. If such measures were based on multiple small assessments occurring in close temporal proximity to clinical experiences, and based on experiences over time (typically a year or more), they could prove more useful to medical education stakeholders than information conveyed by a score on a single day examination (i.e., licensure exam). One such measurement could be a composite score of multiple assessments, similar to the approach used by Zahn [3]. If based on widely used assessments such as NBME clinical science subject examinations, composite scores could provide national performance data.

Medical school faculty could use composite scores to assess a student’s overall clinical science knowledge relative to a national comparison group. If composite scores were shared at the undergraduate to graduate medical education transition point, they might provide residency program directors with a measure of performance across several clerkships that could be used to compare applicants. Program directors may find this more valuable than the current emphasis placed on numeric scores on USMLE Step 1. This could mitigate concerns over high-stakes decisions, such as residency screening and selection, being based on long-duration examinations (USMLE) that cover multiple content areas. Composite scores might also help predict performance on future high-stakes exams such as other licensing and certification exams and could help predict performance in residency training. Given the widespread use of NBME clinical science subject examinations across many medical schools, we therefore sought to investigate the relationship between clinical science subject examination composite scores and other clinical knowledge outcome measures – specifically scores on USMLE Step 2 CK and Step 3 – using a large national sample. We were particularly interested whether a clinical science composite score would be a better predictor of Step 2 CK and Step 3 scores than Step 1 score by itself and would provide even better prediction when used in conjunction with Step 1 score.

Materials and Methods

The data set was obtained from the NBME database and included students from US and Canadian medical schools who took all of the five most widely used clinical science subject examinations (medicine, obstetrics and gynecology, pediatrics, psychiatry, and surgery) for the first time as an end of course/clerkship assessment under standard testing conditions between January 1, 2013, and December 31, 2017 (N = 65,516). This group represents approximately 50% of the 130,000 students who took at least one clinical science subject examination during this time period. First USMLE Step 1, Step 2 CK, and Step 3 scores were also obtained from the NBME database and matched to the clinical science subject examination data; most students had Step 1 and Step 2 CK scores, but fewer students had Step 3 scores because students typically take Step 3 after completing some of their residency training. Clinical science subject examination scores are reported as equated percent correct scores which represent mastery of the content domain assessed by each examination. For this study, equated percent correct scores were calculated for administrations prior to August 2015 that originally reported scaled scores so all scores were on the same scale. USMLE scores are scaled scores with a mean of 200 and standard deviation of 20 for the base reference group defined for each examination. This study was approved by the American Institutes for Research institutional review board.

Identical analyses were conducted for two groups of examinees: (1) examinees who had scores for the five subject examinations, Step 1 and Step 2 CK (Group 1); (2) examinees who had scores for the five subject examinations, Step 1 and Step 3 (Group 2). Group 2 was a subset of Group 1. The decision was made to run analyses for both groups because many students had all five subject examination scores, Step 1 and Step 2 CK scores (n = 58,362 or 89% of the 65,516 students with scores for the five subject examinations), but considerably fewer students also had Step 3 scores (n = 27,118 or 41%).

A preliminary set of stepwise multiple linear regression analyses were conducted based on each group to see which combination of examinations provided the best prediction of Step 2 CK and Step 3 scores. These analyses found that the psychiatry subject examination score had the lowest correlation with Step 2 CK and Step 3 scores and did not meaningfully increase the proportion of variance explained (R²) when included in the prediction equation. The obstetrics and gynecology score had the next lowest correlation with Step2 CK and Step 3 and had minimal impact on R² when included in the prediction equation. The NBME Comprehensive Basic Science Examination (CBSE) score was also considered as a possible predictor variable, but it’s correlation with Step 2 CK and Step 3 scores was considerably lower than the correlations of the five clinical science subject examinations, and it had minimal impact on R² when included in the prediction equation. Based on preliminary stepwise regression results, three composite scores were calculated:

1. Average of medicine, surgery, obstetrics and gynecology, pediatrics, and psychiatry equated percent correct scores (Composite5Exams)

2. Average of medicine, surgery, obstetrics and gynecology, and pediatrics equated percent correct scores (Composite4Exams)

3. Average of medicine, surgery, and pediatrics equated percent correct scores (Composite3Exams)

Several different methods for calculating the composite scores were investigated, including averaging the equated percent correct scores, summing the equated percent correct scores, converting the equated percent correct scores to z-scores and averaging, and converting the equated percent correct scores to z-scores and summing. All methods produced very similar results, so the decision was made to calculate the composite scores as the average of the equated percent correct scores in each subject because this method weights each subject examination score equally, and the resulting composite score can be interpreted in terms of the amount of clinical science content that has been mastered. The computation is also straightforward and comprehensible for score users.

Descriptive analyses were run based on the five subject examination scores, three composite scores, Step 1 scores, and Step 2 CK scores for Group 1 and based on the five subject examination scores, three composite scores, Step 1 scores, Step 2 CK scores, and Step 3 scores for Group 2. Pearson product-moment correlations among the various scores were also run for each group. For each composite score, a stepwise multiple linear regression analysis was performed with the composite score and first Step 1 score as predictor variables and first Step 2 CK score as the dependent variable based on Group 1. The same set of analyses was performed based on Group 2 with Step 3 as the dependent variable. Finally, a logistic regression analysis was run for Group 1 using the composite score based on the five examinations to predict the probability of passing the first Step 2 CK attempt, and a logistic regression analysis was run for Group 2 using the composite score based on the five examinations to predict the probability of passing the first Step 3 attempt.

Results

Table 1 presents descriptive statistics for the various scores for Group 1 and Group 2. Although Group 2 is much smaller than Group 1, the mean scores are nearly identical for both groups. In addition, mean clinical science subject examination scores for both groups are nearly identical to the mean scores for all students who tested during the same time period. Mean Step 1, Step 2 CK, and Step 3 scores for both study groups (Step 1 mean = 231 and 232 for Group 1 and Group 2, respectively; Step 2 CK mean = 243 for both Group 1 and Group 2; Step 3 mean = 225 for Group 2) are slightly higher than the mean scores for all students from US and Canadian medical schools who tested for the first time for each examination during the same time period (Step 1 mean = 229; Step 2 CK mean = 241; Step 3 mean = 224). Thus, both Group 1 and Group 2 are generally representative of the population of students from US and Canadian medical schools from the same time period in terms of proficiency.

Table 1.

Clinical Science Subject Examination Scores – means and standard deviations

	Group 1 (N = 58,362)		Group 2 (N = 27,118)
Score2	Mean	Standard deviation (SD)	Mean	Standard deviation (SD)
Medicine	75	9	74	8
Obstetrics & Gynecology	77	8	77	8
Pediatrics	76	8	76	8
Psychiatry	80	8	79	7
Surgery	73	8	73	8
Composite5Exams3	76	7	76	7
Composite4Exams4	75	7	75	7
Composite3Exams5	75	7	75	7
Step 1	231	19	232	18
Step 2 CK	243	17	243	16
Step 3	---1	---1	225	15

¹Not all students in Group 1 have Step 3 scores; Group 2 is a subset of Group 1 that includes students with Step 3 scores

²All scores are equated percent correct scores except Step 1, Step 2 CK, and Step 3 scores, which are scale scores calculated to have a mean of 200 and standard deviation of 20 for base reference group from early 1990s

³Composite score calculated based on average of equated percent correct scores on medicine, obstetrics and gynecology, pediatrics, psychiatry, and surgery clinical science subject examinations

⁴Composite score calculated based on average of equated percent correct scores on medicine, obstetrics and gynecology, pediatrics, and surgery clinical science subject examinations

⁵Composite score calculated based on average of equated percent correct scores on medicine, pediatrics and surgery clinical science subject examinations

Table 2 presents correlations among the various scores for Group 1 (shown below the diagonal) and Group 2 (shown above the diagonal); reliability estimates are shown along the diagonal. As expected, all scores are moderately to highly positively correlated; notably, the correlations between the various composite scores and Step 2 CK scores (R = 0.79–0.81) and Step 3 scores (R = 0.67–0.69) are higher than the correlation of Step 1 score with Step 2 CK (R = 0.72) and Step 3 (R = 0.61) scores. Further, the correlation between the composite score based on five examinations and Step 3 score (0.69) is essentially the same as the correlation between Step 2 CK score and Step 3 score (0.70).

Table 2.

Pearson product-moment correlations among scores^1,2,3

Score4	Med	Obg	Ped	Psy	Sur	C55	C46	C37	Step 1	Step 2 CK	Step 3
Medicine	0.79	0.58	0.63	0.54	0.61	0.83	0.84	0.86	0.67	0.69	0.60
Obstetrics & Gynecology	0.59	0.74	0.62	0.55	0.56	0.81	0.82	0.68	0.58	0.65	0.55
Pediatrics	0.63	0.63	0.77	0.57	0.62	0.84	0.86	0.87	0.65	0.69	0.58
Psychiatry	0.56	0.56	0.58	0.75	0.51	0.77	0.64	0.62	0.53	0.60	0.52
Surgery	0.61	0.57	0.62	0.52	0.73	0.82	0.83	0.86	0.64	0.65	0.56
Composite5Exams5	0.83	0.82	0.85	0.78	0.82	0.89	0.99	0.96	0.76	0.81	0.69
Composite4Exams6	0.85	0.82	0.86	0.66	0.83	0.99	0.87	0.98	0.76	0.80	0.68
Composite3Exams7	0.87	0.69	0.87	0.64	0.86	0.96	0.98	0.85	0.76	0.79	0.67
Step 1	0.67	0.57	0.65	0.54	0.64	0.75	0.76	0.76	0.93	0.72	0.61
Step 2 CK	0.70	0.65	0.69	0.61	0.66	0.81	0.80	0.79	0.72	0.85	0.70
Step 3	–	–	–	–	–	–	–	–	–	–	0.85

¹Correlations based on Group 1 are shown below the diagonal (N = 58,362); correlations based on Group 2 are shown above the diagonal (N = 27,118)

²All correlations are significant (p < 0.01)

³Reliability estimates are shown along the diagonal

⁴All scores used for correlations are equated percent correct scores except Step 1, Step 2 CK, and Step 3 scores, which are scale scores calculated to have a mean of 200 and standard deviation of 20 for base reference group from early 1990s

⁵Composite score calculated based on average of equated percent correct scores on medicine, obstetrics and gynecology, pediatrics, psychiatry and surgery clinical science subject examinations

⁶Composite score calculated based on average of equated percent correct scores on medicine, obstetrics and gynecology, pediatrics and surgery clinical science subject examinations

⁷Composite score calculated based on average of equated percent correct scores on medicine, pediatrics and surgery clinical science subject examinations

Med = Medicine

Obg = Obstetrics & Gynecology

Ped = Pediatrics

Psy = Psychiatry

Sur = Surgery

The results of the stepwise multiple linear regression analyses are shown in Table 3. For both groups, the composite score based on five examinations is the best predictor of Step 2 CK and Step 3 score, although there is not a large decrease in the proportion of variance explained (R²) when four or three examinations are used to create the composite score instead of five examinations. In all cases, the composite score enters the equation before Step 1 (p < 0.05) because it has the highest correlation with the dependent variable, but including Step 1 in the equation results in a statistically significant increase in R² (p < 0.05). However, the increase in R² when Step 1 is added as a predictor is generally around 0.03 and may be of limited practical significance. The proportion of variance explained is considerably lower for the analyses based on Group 2 that have Step 3 score as the dependent variable than for the analyses based on Group 1 with Step 2 CK as the dependent variable. In all cases, the proportion of variance explained by the composite score (R² = 0.62–0.65 when predicting Step 2 CK score and R² = 0.45–0.48 when predicting Step 3 score) is 0.08–0.13 greater than the proportion of variance explained by Step 1 by itself (R = 0.72 and R² = 0.52 when predicting Step 2 CK score and R = 0.61 and R² = 0.37 when predicting Step 3 score). Figure 1 shows the relationship between the composite score based on five examinations and Step 2 CK and Step 3 scores.

Table 3.

Stepwise linear regression results – predict Step 2 CK score based on composite scores and Step 1 score (N = 58,362); predict Step 3 score based on composite scores and Step 1 score (N = 27,118)¹

	Proportion of variance explained (R2)
Predictor variables	Dependent variable: Step 2 CK	Dependent variable: Step 3
Composite5Exams2	0.654	0.479
Composite5Exams2, Step 1	0.685	0.498
Composite4Exams3	0.643	0.468
Composite4Exams3, Step 1	0.676	0.489
Composite3Exams4	0.620	0.454
Composite3Exams4, Step 1	0.658	0.479

¹All linear regression models are significant (p < 0.01)

²Composite score calculated based on average of equated percent correct scores on medicine, obstetrics and gynecology, pediatrics, psychiatry and surgery clinical science subject examinations

³Composite score calculated based on average of equated percent correct scores on medicine, obstetrics and gynecology, pediatrics and surgery clinical science subject examinations

⁴Composite score calculated based on average of equated percent correct scores on medicine, pediatrics and surgery clinical science subject examinations

Fig. 1.

Relationship of clinical science composite score and First Step 2 CK Score (N = 58,362) and First Step 3 Score (N = 27,118). Composite score calculated based on average of equated percent correct scores on medicine, obstetrics and gynecology, pediatrics, psychiatry, and surgery clinical science subject examinations

Consistent with the linear regression results, the logistic regression analyses confirmed that higher composite scores were associated with a greater probability of passing Step 2 CK (p < 0.01) with odds ratio (OR) = 1.386, 95% confidence interval (CI) [1.371–1.401] and a greater probability of passing Step 3 (p < 0.01) with OR = 1.265, 95% CI [1.247–1.285]. The relationship is most easily interpreted by examining the predicted probability of passing the first Step attempt based on a given composite score as shown in Fig. 2. For example, Fig. 2 shows that the probability of passing Step 2 CK is approximately 0.95 for a composite score of 70%, but only 0.55 for a composite score of 60%. In contrast, the probability of passing Step 3 is approximately 0.95 for a composite score of 70% and 0.80 for a composite score of 60%.

Fig. 2.

Probability of passing First Step 2 CK and First Step 3 Attempt based on Clinical Science Composite Score. Composite score calculated based on average of equated percent correct scores on medicine, obstetrics and gynecology, pediatrics, psychiatry, and surgery clinical science subject examinations

Discussion

The moderate positive correlations of the individual clinical science subject examination scores with USMLE Step 1, Step 2 CK, and Step 3 scores are consistent with results from previous studies based on students from a single institution [3–5]. Further, the moderate to high positive correlations of the clinical science composite scores with USMLE Step 1 and Step 2 CK scores are consistent with results from the study by Zahn et al. that investigated a composite score calculated based on the average of six clinical science subject examination scores at a single medical school [3]. That study included the Family Medicine subject examination score; this study excluded Family Medicine to maximize the number of students from a large national data set with complete data who could be included in the analyses. Of course, correlation does not imply causation, and one should not infer that a given level of performance on the clinical science subject examinations inevitably results in a similar level of performance on USMLE. Many factors, including additional preparation and personal characteristics, impact performance on high-stakes examinations such as Step 1, Step 2 CK, and Step 3.

Composite scores have the potential to provide useful information about students’ clinical science proficiency to both medical school faculty and residency program directors. The clinical science subject examinations that form the basis for the composite scores are administered while the student is actively engaged with the content associated with the particular clerkship. Composite scores therefore may provide an aggregate barometer of a student’s general knowledge of clinical science measured at the point in time when they are most focused on each subject. The composite score provides different information than the Step 1 score because it is based on clinical science knowledge, whereas Step 1 focuses primarily on scientific principles integral to the practice of medicine. Of course, students who perform well on Step 1 are also likely to do well on their clinical science subject examinations and other assessments and all scores are moderately to highly correlated [3–5].

Medical schools could use these composite scores alone or in conjunction with Step 1 to identify students who are at risk of failing Step 2 CK and/or Step 3 so that remediation can be provided. Given the large number of schools that use the clinical science subject examinations, group-level composite score comparison data such as percentile ranks could be provided to enable schools to benchmark the clinical science knowledge of their students relative to a large national group of students. Although the composite score based on five examinations has the highest correlation with Step 2 CK and Step 3 scores, the correlations for the composite scores based on three and four examinations are nearly as high and increase the potential use of the composite score in schools who use fewer subject examinations.

Composite scores may also provide a useful tool for medical school faculty when assessing readiness for postgraduate training or future licensing and certification examinations. To this end, should a composite score be shared at the time a medical student applies to residency, residency program directors may have a more useful measure than the USMLE Step 1 numeric score. Providing composite scores in addition to or instead of Step 1 scores could also alleviate some of the pressure that students experience related to Step 1 [7–10]. On the other hand, providing a clinical science composite score that students could share with program directors would increase the stakes associated with the clinical science subject examinations and could possibly lead to different unintended consequences for medical students and schools.

One limitation of this study is that the R² values reported are based on a single retrospective sample. Additional research is needed to investigate the predictive validity of composite scores based on future cohorts to assess shrinkage in R² values when the prediction equation is applied to a new sample. Another limitation is that some schools do not administer all the subject examinations used to calculate the various composite scores, so the calculation of a composite score may not be possible for all students. This is especially true for international medical schools and students.

This study is the first phase of a research program designed to investigate the utility of clinical science composite scores to predict outcome measures in residency training. In the next phase of the project, we hope to correlate clinical science composite scores with residency performance – particularly outcome measures such as select Accreditation Council for Graduate Medical Education (ACGME) milestones that assess similar constructs as those measured by NBME subject examinations such as medical knowledge and patient care. This work could provide a nationally standardized metric for the undergraduate to graduate medical education hand off. It also has the potential to provide program directors with a broader view of knowledge across many disciplines and to influence a residency selection system that may be overly reliant on a single assessment in the form of USMLE Step 1.

Conclusion

The results of this study demonstrate that clinical science composite scores, based on various combinations of subject exams, are more strongly associated with Step 2 CK and Step 3 performance than is Step 1 score. The composite scores have higher correlations than Step 1 score with Step 2 CK and Step 3 scores; Step 1 score accounts for a small, albeit statistically significant, amount of additional variance in Step 2 CK and Step 3 scores above and beyond the variance explained by the composite scores when included as a predictor variable in a stepwise multiple linear regression model. Given these associations, clinical science composite scores have the potential to provide medical school faculty and residency program directors with an additional source of information about a student’s clinical science knowledge that is based on multiple assessments taken, while the student is actively engaged with each clinical science subject area.

Compliance with Ethical Standards

Conflict of Interest

The authors are employed by the National Board of Medical Examiners.

Ethical Approval

This study was approved by the American Institutes for Research institutional review board.

Statement of Informed Consent

NA – This was a retrospective study that used de-identified aggregate data.