The Use of COMLEX-USA and USMLE for Residency Applicant Selection

Abstract

Background

The Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) and the United States Medical Licensing Examination (USMLE) are recognized by all state medical licensing boards in the United States, and the Federation of State Medical Boards has supported the validity of both examinations for medical licensure. Many osteopathic medical students take both examinations.

Objective

The purpose of this study was to investigate performance on COMLEX-USA Level 1 and USMLE Step 1 of students from colleges of osteopathic medicine where the majority of students took both examinations.

Methods

Data were collected on the entering classes of 2010 and 2011. Relationships between the COMLEX-USA Level 1 and the USMLE Step 1 were quantified using Pearson correlations. The correlation between outcomes on the 2 examinations was evaluated using the phi coefficient. A contingency table was constructed to look at first-attempt outcomes (pass/fail).

Results

Data for 2010 and 2011 were collected from 3 osteopathic medical schools, with 795 of 914 students (87%) taking both examinations. The correlation between first-attempt COMLEX-USA Level 1 and USMLE Step 1 scores was statistically significant across and within all 3 schools. The overall correlation was r(795) = 0.84 (P < .001). Pass/fail status on the 2 examinations was moderately correlated (φ = 0.39, P < .01).

Conclusions

Our study found a strong association between COMLEX Level 1 and USMLE Step 1 performance. Additional studies to accurately compare scores on these examinations are warranted.

What was known and gap

Performance of osteopathic students on the USMLE examination has been studied to assess the concordance of COMLEX-USA and USMLE.

What is new

A study of 914 students from 3 osteopathic schools found scores for COMLEX-USA Level 1 and USMLE Step 1 significantly correlated and pass/fail status moderately correlated.

Limitations

Schools and students in the analysis may not be representative.

Bottom line

The comparability of performance is useful in further validating both examinations in the context of allopathic and osteopathic students “crossing over” into their choice of training programs.

Introduction

The Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) and the United States Medical Licensing Examination (USMLE) are recognized by all state medical licensing boards in the United States. The Federation of State Medical Boards has stated that there is sufficient evidence to support the validity of both examinations for medical licensure.¹

For graduation, all osteopathic medical schools require the passing of COMLEX-USA (Levels 1 and 2),² and the majority of the Liaison Committee on Medical Education (LCME)–accredited medical schools require passing of USMLE (Steps 1 and 2). Program directors use the scores for the evaluation of applicants for residency programs,³ and the Accreditation Council for Graduate Medical Education (ACGME) accepts COMLEX-USA and USMLE for applications to accredited programs.⁴

Having different national standardized examinations recognized as valid for the same purpose is accepted in higher education. Colleges and universities review both ACT and SAT scores to assess the readiness of high school graduates; concordance tables for ACT and SAT examinations have been published.⁵ Similarly, COMLEX-USA and USMLE are examinations that measure overlapping but distinct constructs.^6,7

A number of published studies have demonstrated a strong relationship between COMLEX-USA Level 1 and USMLE Step 1 performance for osteopathic medical students.^8–11 Conversion formulas developed by Slocum and Louder⁹ and Sarko et al¹¹ were tested with larger student cohorts. The authors concluded that these result in underestimated USMLE Step 1 performance based on COMLEX-USA Level 1 scores.⁸

Studies show that performance on the COMLEX-USA series is related to performance in undergraduate universities¹² and in osteopathic medical schools,^13–21 and that it can reliably predict performance in both American Osteopathic Association (AOA)–approved and ACGME-accredited residency programs.^22–25 However, an increasing number of osteopathic medical students are electing to take the USMLE Step 1 examination.^26–28

The purpose of this study was to investigate the relationship between individual examination performances on the COMLEX-USA Level 1 and USMLE Step 1 in a representative and generalizable sample, providing further information about the relationship between the examination scores.

Methods

The Examinations

According to the COMLEX-USA Bulletin of Information, passing the examination “indicates the candidate has demonstrated competence . . . required to solve clinical problems in the supervised . . . setting and to prepare for lifelong learning.”²⁹ The USMLE Step 1 overview states “. . . Step 1 ensures mastery . . . for the safe and competent practice of medicine in the present [and] . . . maintenance of competence.”³⁰ These statements suggest that the examinations—specifically their cut scores—were designed for the purpose of identifying the competency of candidates for licensure. In other words, they serve as “traffic lights” for proceeding to supervised practice (residency training) and subsequently to unsupervised practice (practice with an unrestricted license). The inherent differences in test specifications and blueprints for COMLEX-USA and USMLE are described elsewhere.^31,32

Data Collection

The American Association of Colleges of Osteopathic Medicine (AACOM) contacted all colleges of osteopathic medicine (COMs) to solicit participants for the study. We requested participation from COMs where more than 50% of the students had elected to take USMLE Step 1 in addition to COMLEX-USA Level 1. A total of 3 of 29 COMs met the eligibility criteria and elected to participate.

Each COM provided a data file listing all students beginning enrollment in Fall 2010 and 2011, with the data elements for each student listed in table 1. The National Board of Osteopathic Medical Examiners (NBOME) contributed the data listed in column 2 of table 1, then created a linked research file with these items.

Table 1.

Data Sources

Participants

The sample for this study consisted of 914 students from 3 COMs, representing 10% of osteopathic medical schools and 8% of the population of osteopathic medical students matriculating in 2010 or 2011 (N = 5031 in 2010 and N = 5788 in 2011). Data were collected on all students who matriculated in 2010 or 2011 (table 2).^33,34 Of the 914 participants, 795 (87%) took both the COMLEX-USA Level 1 and the USMLE Step 1 (81% of the eligible students from COM 1, 75% from COM 2, and 100% from COM 3).

Table 2.

Demographics: Study Group Versus All Entering Students

The difference in gender distribution between the sample and the national cohort was not significant in 2010 (χ²₍₁₎ = 0.48, ns) or in 2011 (χ²₍₁₎ = 0.32, ns; table 2)

Regarding ethnicity, the sample percentages differed to some extent from the national cohort in both 2010 (χ²₍₂₎ = 6.59, P < .05) and 2011 (χ²₍₂₎ =  8.01, P < .05), largely because of the underrepresentation of non-White, non-Asian students (table 2).

The Institutional Review Board of the Center for the Advancement of Health Education and Delivery declared this study exempt.

Analysis

The analyses were conducted for first-attempt test scores for both examinations. The relationships between the COMLEX-USA Level 1 and the USMLE Step 1 scores were quantified using Pearson correlations. The association between first-attempt outcomes (pass/fail) on the 2 examinations was evaluated using the phi coefficient. Note that the passing standard for the USMLE Level 1 examination was reset during the period covered by this analysis. Pass/fail information for the students was provided by the schools and reflects the passing standard at the time the students took the examination. The analyses were performed using SAS version 9.4 (SAS Institute Inc, Cary, NC).

Results

Performance for the students who took both examinations is presented in table 3. The normative means and standard deviations are based on the most updated information available at the time this article was written. COMLEX-USA Level 1 information is based on data from all osteopathic examinees from 2010–2014; USMLE information is based on data from US-trained medical students in the 2013 test cycle.

Table 3.

Scores on COMLEX-USA Level 1 and USMLE Step 1 for Current Sample

In the 2013 cycle, the first-attempt pass rate on the USMLE Step 1 examination was 97% for all medical students from LCME-accredited medical schools (n = 19 108) and 94% for all osteopathic medical students who elected to take Step 1 (n = 2680, or 46% of the entering 2011 class of medical students, n = 5788).²⁶ The pass rate for the COMLEX-USA Level 1 in our sample was 97.1%, compared to a 91.5% pass rate for all osteopathic medical students in the 2011–2013 test cycles.

The correlation between first-attempt COMLEX-USA Level 1 and USMLE Step 1 scores was statistically significant across and within all 3 schools. Overall, the correlation was r₍₇₉₅₎ = 0.84 (P < .001; within-school range, 0.82–0.86; figure). Data points are surrounded by a 95% prediction ellipse.

Figure.

Relationship Between the Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) Level 1 Scaled Scores and the United States Medical Licensing Examination (USMLE) Step 1 Scaled Scores

Table 4 presents a 2-by-2 contingency table based on osteopathic medical students who took both examinations. Pass/fail status on USMLE Step 1 and COMLEX-USA Level 1 was moderately correlated (φ = 0.39, P < .01). The majority of students passed both examinations. Osteopathic medical students in this sample were more likely to pass the COMLEX-USA Level 1 and fail the USMLE Step 1 than vice versa.

Table 4.

Comparison of Pass/Fail Status on COMLEX-USA Level 1 and USMLE Step 1

Discussion

We found a strong positive statistical relationship between performance on the COMLEX-USA Level 1 and the USMLE Step 1, corroborating previous investigations.^8–11 However, our findings are based on a reasonably large, multi-school sample of osteopathic medical students, and are therefore likely to be more generalizable. It is important to note that the primary purpose of the COMLEX-USA and the USMLE is for physician licensure, and both the National Board of Medical Examiners and the NBOME have cautioned secondary users regarding the overuse of examination scores in residency program selection.³⁵

The strong association between scores on COMLEX-USA Level 1 and USMLE Step 1 in this sample would suggest that the 2 examination scores, while not interchangeable because of content differences, can be used to make reasonably comparable and consistent estimates of the overall ability of a residency applicant. This relates to osteopathic medical students now, and will pertain to students with USMLE scores from LCME-accredited and international medical schools as they become eligible to apply to formerly AOA-approved residency programs in the new single accreditation system for graduate medical education (GME).

Our study sample also had a slightly higher pass rate on the COMLEX-USA Level 1 compared with the USMLE Step 1. This seems reasonable given that the COMLEX-USA Level 1 is aligned with the medical school curricular focus, philosophical approach to patient care, and practice designation. To further understand the difference, one would need to know how students from LCME-accredited schools perform on COMLEX-USA Level 1.

Our investigation has limitations. Of the 29 schools of osteopathic medicine at the time of the study, 3 schools volunteered to provide data. In addition, the student sample may not have been entirely representative of the larger population of interest because their scores on the COMLEX-USA Level 1 and USMLE Step 1 differed from the normative means by less than 1 standard deviation.

There also may have been selection bias related to motivational factors: osteopathic medical students taking COMLEX-USA as a pass/fail examination for graduation/licensure and USMLE for use in residency application could yield performance differences. We did not collect information on when the examinations were taken, which also may confound the results. Finally, we only looked at Level 1/Step 1 data. For examinations taken later in the training, the associations between scores could be different because of the distinctiveness of the content domains that are measured.

Given the number of osteopathic students applying to ACGME-accredited GME programs, there should be added education for program directors using examination performance data for selection decisions. This could include providing detailed, easily interpretable information on what each test measures, the development of valid score correspondence tables based on large student samples, and/or the exploration of alternate score reporting scales.

Conclusion

There is an association between COMLEX-USA Level 1 and USMLE Step 1 performance of osteopathic medical students who take both examinations. This suggests that the examinations measure similar, yet not identical, constructs. Program directors using both COMLEX-USA and USMLE scores in residency applications are advised to avail themselves of resources designed to assist with score interpretation and appropriate use.