Anesthesiology Resident Performance on the US Medical Licensing Examination Predicts Success on the American Board of Anesthesiology BASIC Staged Examination: An Observational Study

Abstract

Background:

Correlation has been found between the US Medical Licensing Examination (USMLE) Step 1 examination results and anesthesiology resident success on American Board of Anesthesiology (ABA) examinations. In 2014, the ABA instituted the BASIC examination at the end of the postgraduate year-2 year. We hypothesized a similar predictive value of USMLE scores on BASIC examination success.

Methods:

After the Committee for the Protection of Human Subjects at UTHealth Institutional Review Board approved and waived written consent, we retrospectively evaluated USMLE Step examination performance on first-time BASIC examination success in a single academic department from 2014–2018.

Results:

Over 5 years, 120 residents took the ABA BASIC examination and 108 (90%) passed on the first attempt. Ten of 12 first-time failures were successful on repeat examination but analyzed in the failure group. Complete data was available for 92 residents (76.7%), with absent scores primarily reflecting osteopathic graduates who completed Comprehensive Osteopathic Medical Licensing Examination of the United States level examinations rather than USMLE. In the failure cohort, all 3 USMLE examination step scores were lower (P < .02). USMLE Step 1 score independently predicted success on the BASIC examination (odds ratio [OR] 1.11, 95% confidence interval [CI] 1.05–1.17, P < .001). Although USMLE Step 2 score predicted BASIC examination success (OR 1.10, 95% CI 1.04–1.18, P = .001), this did not remain after adjustment for Step 1 score using multiple logistic regression (P = .11). In multivariable logistical regression, first clinical anesthesia in-training examination score and USMLE Step 1 score were significant for predictors of success on the BASIC exam.

Conclusions:

In anesthesiology residency training, our preliminary single-center data is the first to suggest that USMLE Step 1 performance could be used as a predictor of success on the recently introduced ABA BASIC Examination. These findings do not support recent action to change USMLE scoring to a pass/fail report.

Introduction

There has been recent discussion about use of US Medical Licensing Examination (USMLE) Step scores in residency selection.¹ This is accentuated with the recent decision to cease providing USMLE Step 1 scores, likely in 2022.² This decision appears to contrast a growing body of literature that identifies a potential effect between USMLE performance and anesthesiology trainee academic performance. In a meta-analysis of 17 studies and >41 000 applicants, not specific for anesthesiology, USMLE scores were the strongest associated links to physician performance in residency.³ In anesthesiology residency training, there is significant correlation between USMLE Step 1 scores and performance on both the American Board of Anesthesiology (ABA) residency in-training examination (ITE) and the traditional certification examination.^4,5 Despite these recent calls to de-emphasize USMLE score availability and reporting,⁶ many programs, including ours, use USMLE Step 1 scores as one of many important metrics for residency selection committees to anticipate academic risks and appropriately select applicants.

In 2014, the ABA instituted a staged examination process that divided the previously known traditional Part 1 Exam into BASIC and ADVANCED exams. The BASIC exam is offered at the end of the postgraduate year-2 year of training and the ADVANCED exam is offered after completion of residency training. Due to the known associations for USMLE scores and ABA ITE and traditional examinations, and absence of current data for the new ABA BASIC examination, we hypothesized that there would be similar predictive value of the USMLE Step scores to BASIC examination performance. Therefore, we analyzed BASIC exam performance of all residents from our large residency program since the inception of the BASIC examination.

Methods

After the Committee for the Protection of Human Subjects at UTHealth Institutional Review Board approved and waived informed consent due to the de-identification of aggregate blinded data, we retrospectively collated resident USMLE Step 1, 2, and 3 scores for all residents who sat for the BASIC examination for the first time from 2014 (first year of the examination) through 2018. Further, we also recorded information related to the gender, first clinical anesthesia (CA-1) year ITE score, whether the resident was a doctor of medicine (MD) or doctor of osteopathic medicine (DO), and the year of the BASIC exam (Table 1). Data was collected by an administrative assistant so that authors were blinded to specific resident examination performance. However, educational program leaders and resident recruitment faculty are frequently aware of USMLE Step 1 and 2 scores, which are reviewed by the resident selection committee and are required for resident entry into the program. Successful completion of Step 3 is required for continuation into the postgraduate year-2 training year. This data is available in the resident file in the residency administrative office and all de-identified data for this analysis was stored on institutional password protected devices. Similarly, department educational leaders are aware of residents who were unsuccessful on their first BASIC examination attempt but data for this analysis remained de-identified. First-time success on the ABA BASIC examination was determined from ABA reports to the department. The USMLE scores for each of the 3 examinations were compared between the passing and failing cohorts. Subsequent success on the ABA BASIC examination did not influence group assignment as analysis was reserved for first-time examinees.

Table 1.

Baseline Characteristics of Residents Taking the American Board of Anesthesiology BASIC Exam

Baseline Data (N = 120)	n (%)
Male residents	68 (57)
MD/DO
MD	95 (79)
DO	25 (21)
In-training examination scorea	30.35 ± 5.20
In-training Examination Score
≤27	40 (33)
28–32	38 (32)
≥33	42 (35)
Year of Exam
2014	22 (18)
2015	25 (21)
2016	24 (20)
2017	25 (21)
2018	24 (20)

Abbreviations: DO, doctor of osteopathic medicine; MD, doctor of medicine.

^a Depicted as mean ± SD.

Data was collected over the first 5 years of the ABA BASIC examination in a large department. Post hoc power analysis based on the Step 1 scores (mean ± SD, alpha 0.05, beta 0.8) confirmed a minimum required sample size of 8, which was exceeded in our initial fail cohort. Correlation analysis was performed on Step 1–3 scores by evaluating the Spearman correlation coefficient, which suggested moderate level of correlation among Step 1–3 scores. To evaluate possible collinearity, we implemented univariate logistic regression models including each Step score as well as logistic regression models including any combination of Step 1–3 scores. There were only small to moderate levels of inflation on the estimated standard errors when other step scores were added to a univariate logistic regression model. Therefore, collinearity can be safely discounted in multivariable analysis.

The USMLE scores for each examination in the pass/fail BASIC examination cohorts were averaged (mean ± SD) and compared between pass and fail groups (2-sample test for unequal variances, Welch t-test). Analysis of each USMLE score of those who passed or failed the BASIC examination was performed using univariate logistic regression. Subsequently, we performed multivariable analysis including the 3 USMLE step scores, the year of the BASIC exam, gender, CA-1 year ITE score, and type of medical degree awarded (MD vs DO). A forward stepwise algorithm was implemented to select variables. The likelihood ratio test was used to evaluate variables. A significance level of 0.05 was the criterion to add a variable as well as the criterion to keep variables in the elimination step. We conducted the Hosmer–Lemeshow test to evaluate goodness of fit of the final model. P values less than .05 were considered as significant. All statistical analyses were performed by using the SAS software (version 9.4, SAS Institute, Cary, NC). Where applicable, odds ratios (OR) are reported for a 1-point change in USMLE score. The study and manuscript adhere to the Strengthening the Reporting of Observational Studies in Epidemiology requirements.

Results

Within the 5 years of study, 120 residents took the BASIC exam. The USMLE Step 1 scores were obtained for 119 residents (99.2%), Step 2 scores for 117 residents (97.5%), and Step 3 scores for 92 residents (76.7%). One resident did not take any US licensing exams as a graduate of a medical school outside the United States. Additional missing scores for USMLE Step 2 and Step 3 were due to graduates from osteopathic programs who completed Comprehensive Osteopathic Medical Licensing Examination of the US level 3 and occasionally level 2 examinations rather than the USMLE.

Of the 120 eligible residents, 108 (90%) passed the ABA BASIC examination on the first attempt. Ten of the 12 first-time failures passed on repeat attempt but remained in the fail cohort for initial analysis. Residents were included for further analysis only if they had USMLE scores. Thus, we had 119 residents included in Step 1 analysis, 117 in Step 2, and 92 in Step 3. The USMLE Step scores were significantly lower in the Fail group on all 3 Step examinations (Table 2).

Table 2.

Aggregate US Medical Licensing Examination (USMLE) Step Scores in American Board of Anesthesiology (ABA) BASIC Examination Pass/Fail Groups

	ABA BASIC Pass/Fail (No. of tests)	Mean ± SD	P Value
Step 1	Pass (107)	228.0 ± 13.1	<.001a
	Fail (12)	208.4 ± 14.8
Step 2	Pass (105)	236.2 ± 11.5	.01a
	Fail (12)	220.3 ± 17.8
Step 3	Pass (81)	216.8 ± 11.6	.02a
	Fail (11)	210.7 ± 6.5

^a Welch t-test.

In univariate logistic regression modeling, Step 1 score was a predictor of success on the BASIC Exam (OR 1.11, 95% confidence interval [CI] 1.05–1.17, P < .001). Step 2 score was also a predictor of BASIC examination success (OR 1.10, 95% CI 1.04–1.18, P = .001). Step 3 score was not a predictor of success on the BASIC examination (OR 1.06, 95% CI .988–1.13, P = .11).

In multivariable logistic regression modeling including Step scores, ITE scores, gender, year of exam, and MD/DO, only the Step 1 score (OR 1.10, 95% CI 1.03–1.17, P < .002) and the CA-1 ITE score (OR 1.33, 95% CI 1.10–1.61, P < .002) were found to be significant predictors of success on the BASIC exam. This model fit the data well based on the goodness-of-fit test result (Hosmer-Lemeshow test P = .31). We used this information to create a predictive probability curve of success on the BASIC exam (Figure 1). We used an ITE score of 30 (the mean of our CA-1 residents during this time period) as the value for ITE score and the Step 1 score varied in its range. For a resident with the mean ITE score, a Step 1 score of 218 indicated a 95% probability of passing and a score of 210 indicated a 90% probability of passing.

Figure 1.

Probability of passing the BASIC based on a logistic regression model is plotted on the red line (95% confidence interval in red dashed lines) with mean in-training examination score of 30 for given US Medical Licensing Examination (USMLE) Step 1 scores. Abbreviation: ABA, American Board of Anesthesiology.

Discussion

Recent opinions call for decreased emphasis and use of USMLE Step scores in the residency selection process.^1,6 This culminated in a decision by the Federation of State Medical Boards and the National Board of Medical Examiners to change traditional scoring of the USMLE Step 1 examination to a pass-fail result.² We opposed this sentiment⁷ given existing data to support correlation and predictive value of USMLE examination performance on attainment of subsequent academic metrics during anesthesiology training.^4,5,8–10 Our single-center data in this retrospective study confirms the predictive effect of USMLE Step 1 scores on performance in the recently introduced ABA BASIC examination, which mirrors findings in other ABA high-stakes examinations.

In the traditional examination era, studies showed that both USMLE Step 1 and 2 average scores correlated significantly with all residency standardized ABA examinations. In one program, improved performance on the ABA written examination coincided with increased emphasis on USMLE performance in their residency selection.⁵ Similarly, USMLE Step 2 scores were a moderately strong predictor of anesthesiology written board examination performance (P < .001) with a Step 2 score of 181 predicting failure.⁸ More recently, a review9 of pre-residency recruitment measures also demonstrated that higher USMLE scores predicted high ABA examination scores. In addition, survey responses by residents confirmed the association between prior USMLE Step 1 and 2 scores and subsequent performance on ITE scores.¹⁰

Success on ABA examinations is critical for advancement of the individual examinee as well as for the training program, where an overall ABA pass rate is a desired metric. Academic success on ABA examinations is recognized by the Accreditation Council for Graduate Medical Education in the medical knowledge section of the Anesthesiology Milestones Project.¹¹ Further, the Accreditation Council for Graduate Medical Education program requirements for residency programs uses board pass rate as one measure of the effectiveness of the educational program.¹² After graduation from residency training, examination success and ultimate board certification remain important. Hospital credentialing associations and health insurance companies are increasingly requiring board certification; failure of the candidate to achieve success may predict subsequent unsatisfactory state medical board action.¹³

Our data expands prior reports of USMLE examination performance on ABA examination success to the newly instituted BASIC staged examination. Our findings with a relatively small sample suggest that data for USMLE Step 1 performance should remain available to anesthesiology recruitment programs as one measure of residency selection within a comprehensive global assessment. Our data affirms that the USMLE Step 1 score is a predictor of BASIC examination success, but we continue to support the use of a holistic evaluation process for resident selection. It is evident that some residents who scored <218 on Step 1 did pass the BASIC examination at the first opportunity. We have previously described our evaluation process,¹⁴ which recognizes many other desirable characteristics and achievements of potentially successful anesthesiology resident applicants. The knowledge of Step 1 scores assists in identifying at-risk residents, giving us the opportunity to create individualized learning plans upon matriculation into residency before any ITEs have occurred. A simple pass/fail on Step 1 removes one tool in identifying residents who are potentially at risk for failing the BASIC exam.

Our study has several limitations. This is a single-center report that may not be directly applicable to other institutions. The retrospective nature of the evaluation is of less concern given the objective outcome measures of USMLE scores and pass/fail BASIC examination designation, thus eliminating potential bias. Although there was no a priori power analysis, 5 years of data for one of the largest programs in the nation provided an acceptable post hoc powered signal. Nonetheless, these data require verification in a prospective, multi-department study.

Our findings, when added to available evidence prior to introduction of the BASIC exam, support a call to action by academic leaders in the anesthesiology community (including the ABA and the American Society of Anesthesiologists), to respond objectively to elimination of USMLE Step 1 scores for candidates for anesthesiology residency selection committees.

In summary, our observational study found that anesthesiology residents failing the ABA BASIC examination on first attempt had lower USMLE Step scores on all 3 examinations. Step 1 USMLE score and CA-1 ITE scores were independent predictors of success on first-time takers of the ABA BASIC examination, with Step 1 scores of 210 and 218 predictive of passing with 90% and 95% probability, respectively, in this model.