ABSTRACT
Background
While United States Medical Licensing Examination (USMLE) scores correlate with performance on subsequent standardized tests, it is not known whether they predict success in residency. Nonetheless, they remain the most used metric to narrow the large applicant pool.
Objective
To determine whether several filterable metrics reported by the Electronic Residency Application Service (ERAS) predict success in pediatric residency, as measured by Accreditation Council for Graduate Medical Education Pediatrics Milestone (PM) ratings.
Methods
Pediatric residency programs participating in the Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network were invited to participate in 2020; 10 of 149 programs (6.7%) provided data on 518 residents. Metrics included type of medical degree, US or non-US medical school, Alpha Omega Alpha and Gold Humanism Honor Society (GHHS) membership, USMLE Step 1 and Step 2 Clinical Knowledge (CK) numerical scores, and Step 2 Clinical Skills first attempt pass/fail status; primary outcome was mean PM ratings across all residency years.
Results
GHHS was positively associated with overall PM performance (B=0.15, P=.009), as was Step 2 CK (B=0.06, P=.007). By domain, Step 2 CK was positively associated with Medical Knowledge (MK), Patient Care (PC), and Practice-Based Learning and Improvement (PBLI). GHHS membership was positively associated with MK, PC, PBLI, Professionalism, and Systems-Based Practice. Effect size of GHHS was higher than Step 2 CK across all domains.
Conclusions
Among filterable ERAS metrics, GHHS membership showed the strongest association with pediatric residency success, as measured by PM ratings; Step 2 CK had a weaker positive association.
Introduction
Faced with the challenge of rapidly selecting interview candidates from a large applicant pool, residency program directors (PDs) often rely on readily filterable metrics within the Electronic Residency Application Service (ERAS) that may not fully capture the diverse qualities and competencies essential for success in residency. Performance on the United States Medical Licensing Examination (USMLE) is commonly used for initial screening, despite concerns about the ability of standardized testing to predict broader clinical skills.1-6
While USMLE scores have repeatedly been shown to correlate with performance on subsequent standardized tests, such as in-training examinations and specialty board certification examinations,7-14 evidence linking USMLE scores to other markers of success in residency, particularly clinical performance and non-cognitive skills, has been weak, conflicting, or inconclusive.7,10,12,14-22 In some cases, a negative association between USMLE scores and non-cognitive performance has been observed.23,24
In response to the limitations and unintended consequences of overreliance on USMLE scores, as well as the shift to pass/fail reporting for Step 1, there has been a call for alternative and more effective methods to evaluate residency applicants.1-3 PDs report that qualities such as clinical competency, quality of patient care, professionalism, ethics, and communication skills are more important than academic performance alone in assessing resident success.25 However, standardized measures for these competencies during the application process are not yet widely available. The Accreditation Council for Graduate Medical Education (ACGME) Milestones represent a framework for competency-based assessment in residency training, covering 6 core domains: Patient Care (PC), Medical Knowledge (MK), Practice-Based Learning and Improvement (PBLI), Interpersonal and Communication Skills (ICS), Professionalism (PROF), and Systems-Based Practice (SBP). These Milestones describe observable developmental levels of behavior and are used to evaluate resident progression toward unsupervised practice. Studies using Milestones have shown that trainees enter residency with a wide range of skills and progress over time, with Milestones able to differentiate between training levels.17,26,27
This study draws on data from the Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN) to examine whether filterable medical school metrics reported in ERAS predict success in pediatric residency, as measured by Milestone achievement on the ACGME pediatric competencies. The objective was to identify alternative metrics predictive of success across the many competencies required of a physician, providing additional tools for rapidly screening large volumes of applications and supporting a more holistic review process.
KEY POINTS
What Is Known
Program directors are searching for efficient ways to review applications to best identify candidates who will succeed in residency.
What Is New
In this study of 10 pediatric programs, membership in the Gold Humanism Honor Society (GHHS) had the strongest correlation with ACGME Pediatrics Milestone ratings.
Bottom Line
Program directors will want to consider GHHS membership as a potential indicator of residency success, particularly if these findings can be replicated in other specialties.
Methods
Setting and Participants
This study is a multisite cohort study utilizing retrospective data. All pediatric residency programs participating in APPD LEARN were invited to contribute data in 2020. In 2020, 74% of pediatric residency programs (149 of 201 US programs) had joined APPD LEARN, with very small community programs slightly underrepresented.28 Ten of 149 invited programs (6.7%) chose to participate; median program participation in APPD LEARN studies is 11.5 (8-20.5). Data were collected on categorical residents who entered participating programs between 2016 and 2019, prior to the shift of USMLE Step 1 to pass/fail.
Applicant Metrics
The ERAS metrics collected as potential predictors included the type of medical degree, whether the resident graduated from a US medical school, membership in Alpha Omega Alpha (AOA) honor society, membership in the Gold Humanism Honor Society (GHHS), USMLE Step 1 numerical score, USMLE Step 2 Clinical Knowledge (CK) numerical score, and USMLE Step 2 Clinical Skills (CS) first attempt pass/fail status.
Outcomes Measured
The primary outcome measure was mean Pediatrics Milestone (PM) ratings on the 21 ACGME pediatric competencies, assessed at the midpoint and end of each residency year. PM ratings are reported biannually on a scale of 1 to 5. The 6 competency domains evaluated included MK, PC, PBLI, ICS, PROF, and SBP. Year of training served as a positive control.
Program-level data, including program size, number of applicants, number invited to interview, number accepted, and whether Step 1 scores were used for screening, were collected as covariates to account for inter-program differences and potential biases that could be introduced by those differences.
Analysis of the Outcomes
Midyear and year-end PM ratings in each residency year were averaged for each resident. Because residents could appear in the data set in each year of their residency, within-resident correlation was accounted for by including a random effect of learner in the models. PM ratings were analyzed as a continuous variable. Mixed models regression methods were employed, using program as a clustering variable. Resident demographics were included as potential covariates. Whether or not programs reported using USMLE score cut-offs was included as an independent predictor in multivariate regression models. Applicants without reported USMLE scores were excluded from analyses. USMLE scores were analyzed in 10-point increments. Statistical analyses were conducted using R 4.2 (R Core Team). Results are reported in terms of unstandardized regression coefficients (B), as point estimates for overall PM and by competency domain.
The institutional review boards at the University of Illinois at Chicago (data coordinating site) and Tulane University School of Medicine (lead site) determined that the study was exempt.
Results
Data from 518 residents across 10 programs were collected (Table 1). The majority of residents (90%) graduated from US medical schools. Membership in AOA was reported for 76 residents (14.7%), and GHHS membership for 85 residents (16.4%; Table 2). USMLE Step 1 numerical scores were reported for 498 residents (96.1%), and USMLE Step 2 numerical scores were reported for 491 residents (94.8%).
Table 1.
Interviewing and Ranking Characteristics of Participating Programs
| Program | Program Sizea | Mean Applications per Position Offeredb | Mean Applicants Invited to Interview per Positionb | Mean Applicants Interviewed per Position | Mean Applicants Ranked per Position | USMLE Step 1 Cutoff Used | Mean USMLE Step 1 of Matched Residents | Mean USMLE Step 2 of Matched Residents |
|---|---|---|---|---|---|---|---|---|
| 1 | Medium | 87 | 17 | 12 | 11 | no | 208 | 228 |
| 2 | Large | 53 | 9 | 8 | 8 | no | 217 | 228 |
| 3 | Small | 101 | 15 | 12 | 9 | 210 | 217 | 236 |
| 4 | Medium | 74 | 20 | 10 | 10 | 192 | 231 | 246 |
| 5 | Medium | 30 | 11 | 11 | 11 | 220 | 242 | 254 |
| 6 | Medium | 67 | 18 | 15 | 13 | no | 219 | 230 |
| 7 | Large | 70 | 21 | 17 | 16 | no | 229 | 241 |
| 8 | Medium | N/A | 30 | 19 | 18 | no | 226 | 241 |
| 9 | Large | 27 | 13 | 10 | 10 | 220 | 225 | 237 |
| 10 | Small | N/A | N/A | 16 | 15 | no | 233 | 249 |
Small: fewer than 30 residents; medium: 31-60 residents; large: more than 60 residents.
N/A: data not provided by program.
Abbreviations: USMLE, United States Medical Licensing Examination; N/A, not available.
Table 2.
Metrics as Reported via ERAS for Residents at Participating Sites (N=518)
| Predictor | n (%) | Unadjusted Correlation Between Predictor and Overall Milestone Composite | ||
|---|---|---|---|---|
| PGY-1 | PGY-2 | PGY-3 | ||
| MD degree | 460 (88.6) | 0.14a | 0.06 | 0.01 |
| US medical graduate | 465 (89.8) | 0.16b | 0.08 | 0.02 |
| AOA | 76 (14.7) | 0.17b | 0.11 | 0.16 |
| GHHS | 85 (16.4) | 0.18b | 0.11 | 0.23b |
| USMLE Step 1 score available | 498 (96.1) | 0.07 | 0.01 | -0.03 |
| USMLE Step 2 CK score available | 491 (94.8) | 0.11 | 0.06 | 0.02 |
P<.05.
P<.01.
Abbreviations: ERAS, Electronic Residency Application Service; PGY, postgraduate year; AOA, Alpha Omega Alpha Honor Society; GHHS, Gold Humanism Honor Society; USMLE, United States Medical Licensing Examination; CK, Clinical Knowledge.
The year of residency training was positively associated with success across all competency domains (B=1.15; 95% CI, 1.12-1.19; P<.001).
Among the filterable ERAS metrics (Table 3 and Figure), GHHS membership was positively associated with overall mean PM performance (B=0.15; 95% CI, 0.04-0.27; P=.009). Specifically, GHHS membership showed positive associations with performance in the MK (B=0.20; 95% CI, 0.06-0.33; P=.006), PC (B=0.16; 95% CI, 0.03-0.29; P=.019), PBLI (B=0.17; 95% CI, 0.03-0.30; P=.015), PROF (B=0.15; 95% CI, 0.02-0.28; P=.022), and SBP (B=0.13; 95% CI, 0.01-0.25; P=.028) competency domains. The effect size of GHHS on PM ratings was the highest among all variables analyzed, across all domains.
Table 3.
Resident Milestone Achievement, Overall and by Domain
| ERAS Metrics | ACGME Performance Domains, Point Estimate (95% CI) |
||||||
|---|---|---|---|---|---|---|---|
| Overall | ICS | MK | PC | PBLI | PROF | SBP | |
| (Intercept) | 2.60 (1.61 to 3.58) |
3.32 (2.08 to 4.56) |
1.48 (0.33 to 2.63) |
1.49 (0.37 to 2.61) |
2.19 (1.04 to 3.35) |
3.43 (2.33 to 4.52) |
3.26 (2.25 to 4.28) |
| USMLE Step 1 (10-point increase) | -0.01 (-0.05 to 0.02) P=.45 |
-0.02 (-0.07 to 0.03) P=.41 |
0.01 (-0.03 to 0.06) P=.52 |
-0.02 (-0.06 to 0.02) P=.31 |
-0.01 (-0.06 to -0.03) P=.50 |
0.00 (-0.04 to 0.04) P=.96 |
-0.04 (-0.08 to -0.00) P=.036 |
| USMLE Step 2 CK (10-point increase) | 0.06 (0.02 to 0.10) P=.007 | 0.05 (-0.01 to 0.10) P=.09 |
0.06 (0.01 to 0.12) P=.018 | 0.09 (0.04 to 0.14) P<.001 | 0.07 (0.02 to 0.12) P=.05 | 0.04 (-0.01 to 0.09) P=.10 |
0.04 (-0.01 to 0.08) P=.13 |
| USMLE Step 2 CS (pass at first attempt) | 0.04 (-0.34 to 0.41) P=.85 |
-0.09 (-0.55 to 0.38) P=.71 |
0.21 (-0.24 to 0.66) P=.36 |
0.18 (-0.25 to 0.61) P=.41 |
-0.10 (-0.54 to 0.33) P=.64 |
-0.09 (-0.50 to 0.31) P=.65 |
0.30 (-0.09 to 0.68) P=.13 |
| AOA | 0.05 (-0.09 to 0.18) P=.50 |
0.08 (-0.18 to 0.24) P=.32 |
0.02 (-0.08 to 0.24) P=.32 |
0.02 (-0.13 to 0.18) P=.78 |
0.04 (-0.12 to 0.20) P=.63 |
0.05 (-0.09 to 0.20) P=.48 |
0.06 (-0.08 to 0.20) P=.41 |
| GHHS | 0.15 (0.04 to 0.27) P=.009 | 0.14 (0.00 to 0.28) P=.06 |
0.20 (0.06 to 0.33) P=.006 | 0.16 (0.03 to 0.29) P=.019 | 0.17 (0.03 to 0.30) P=.015 | 0.15 (0.02 to 0.28) P=.022 | 0.13 (0.01 to 0.25) P=.028 |
| Non-MD degree | 0.08 (-0.11 to 0.28) P=.41 |
0.05 (-0.19 to 0.30) P=.67 |
0.18 (-0.05 to 0.42) P=.13 |
0.07 (-0.15 to 0.29) P=.53 |
0.09 (-0.14 to 0.32) P=.45 |
0.09 (-0.12 to 0.30) P=.41 |
0.07 (-0.13 to 0.27) P=.51 |
| US medical graduate | -0.01 (-0.17 to 0.15) P=.93 |
-0.02 (-0.23 to 0.18) P=.82 |
0.02 (-0.17 to 0.22) P=.82 |
0.07 (-0.12 to 0.25) P=.46 |
-0.09 (-0.27 to 0.10) P=.38 |
-0.02 (-0.20 to 0.15) P=.79 |
0.02 (-0.15 to 0.19) P=.81 |
| Residency year | 1.15 (1.12 to 1.19) P<.001 | 1.18 (1.13 to 1.23) P<.001 | 1.18 (1.13 to 1.23) P<.001 | 1.32 (1.28 to 1.36) P<.001 | 1.20 (1.16 to 1.25) P<.001 | 0.97 (0.93 to 1.01) P<.001 | 1.15 (1.11 to 1.20) P<.001 |
Abbreviations: ERAS, Electronic Residency Application Service; ACGME, Accreditation Council for Graduate Medical Education; ICS, Interpersonal and Communication Skills; MK, Medical Knowledge; PC, Patient Care; PBLI, Practice-Based Learning and Improvement; PROF, Professionalism; SBP, Systems-Based Practice; USMLE, United States Medical Licensing Examination; CK, Clinical Knowledge; CS, Clinical Skills; AOA, Alpha Omega Alpha Honor Society; GHHS, Gold Humanism Honor Society.
Note: The values in bold are those with statistically significant results.
Figure.
Resident Milestone Achievement, Overall and by Domain
Abbreviations: GHHS, Gold Humanism Honor Society; CK, Clinical Knowledge; AOA, Alpha Omega Alpha Honor Society; USMG, United States Medical Graduate; CS, Clinical Skills.
USMLE Step 2 CK scores were positively associated with overall mean PM performance (B=0.06; 95% CI, 0.02-0.10; P=.007). By competency domain, Step 2 CK was positively associated with performance in MK (B=0.06; 95% CI, 0.01-0.12; P=.018), PC (B=0.09; 95% CI, 0.04-0.14; P<.001), and PBLI (B=0.07; 95% CI, 0.02-0.12; P=.005). Step 2 CK was found to have a weaker positive association with pediatric residency success compared to GHHS membership.
USMLE Step 1 scores were not associated with overall mean PM performance (B=-0.02; 95% CI, -0.07 to 0.03; P=.41). A negative association was found between Step 1 scores and performance in the SBP competency domain (B=-0.04; 95% CI, -0.08 to 0.00; P=.036).
Our study found no correlation between the other filterable ERAS metrics examined and PM performance, including AOA membership, MD versus non-MD (DO, MBBS, MBBCh) degree, whether the applicant graduated from a US medical school, and USMLE Step 2 CS first attempt pass/fail status.
Whether or not programs reported using USMLE score cut-offs was not statistically significant in any of the multivariate regression models.
Mean, standard deviations, and intercorrelations of the overall PM rating and subdomains for each year of training (not adjusted for clustering) are presented in Table 4.
Table 4.
Mean and Standard Deviation of Overall Pediatrics Milestones Rating and Subdomains
| Mean | SD | Overall | ICS | MK | PC | PBLI | PROF | SBP | |
|---|---|---|---|---|---|---|---|---|---|
| PGY-1 | |||||||||
| Overall | 4.91 | 0.76 | 1 | ||||||
| Interpersonal and Communication Skills | 5.07 | 0.98 | 0.86 | 1 | |||||
| Medical Knowledge | 4.71 | 0.89 | 0.76 | 0.70 | 1 | ||||
| Patient Care | 4.68 | 0.86 | 0.92 | 0.80 | 0.75 | 1 | |||
| Practice-Based Learning and Improvement | 4.64 | 0.92 | 0.85 | 0.70 | 0.63 | 0.73 | 1 | ||
| Professionalism | 5.39 | 0.84 | 0.84 | 0.64 | 0.52 | 0.68 | 0.56 | 1 | |
| Systems-Based Practice | 4.66 | 0.88 | 0.84 | 0.70 | 0.61 | 0.74 | 0.75 | 0.58 | 1 |
| PGY-2 | |||||||||
| Overall | 6.20 | 0.59 | 1 | ||||||
| Interpersonal and Communication Skills | 6.40 | 0.81 | 0.80 | 1 | |||||
| Medical Knowledge | 5.98 | 0.79 | 0.74 | 0.54 | 1 | ||||
| Patient Care | 6.17 | 0.70 | 0.90 | 0.68 | 0.70 | 1 | |||
| Practice-Based Learning and Improvement | 5.95 | 0.74 | 0.83 | 0.59 | 0.62 | 0.69 | 1 | ||
| Professionalism | 6.47 | 0.64 | 0.86 | 0.67 | 0.53 | 0.68 | 0.60 | 1 | |
| Systems-Based Practice | 5.98 | 0.66 | 0.81 | 0.61 | 0.60 | 0.71 | 0.62 | 0.58 | 1 |
| PGY-3 | |||||||||
| Overall | 7.21 | 0.50 | 1 | ||||||
| Interpersonal and Communication Skills | 7.41 | 0.70 | 0.79 | 1 | |||||
| Medical Knowledge | 7.06 | 0.68 | 0.72 | 0.53 | 1 | ||||
| Patient Care | 7.30 | 0.57 | 0.88 | 0.63 | 0.61 | 1 | |||
| Practice-Based Learning and Improvement | 7.04 | 0.61 | 0.85 | 0.58 | 0.66 | 0.67 | 1 | ||
| Professionalism | 7.35 | 0.59 | 0.90 | 0.72 | 0.56 | 0.73 | 0.66 | 1 | |
| Systems-Based Practice | 6.94 | 0.52 | 0.69 | 0.44 | 0.48 | 0.53 | 0.57 | 0.52 | 1 |
Abbreviations: SD, Standard Deviation; ICS, Interpersonal and Communication Skills; MK, Medical Knowledge; PC, Patient Care; PBLI, Practice-Based Learning and Improvement; PROF, Professionalism, SBP, Systems-Based Practice; PGY, postgraduate year.
Discussion
Our study identified GHHS membership as the strongest predictor among the evaluated filterable ERAS metrics for overall pediatric residency success and success in multiple individual competency domains (ICS, MK, PBLI, PROF) as measured by PM ratings. USMLE Step 2 CK scores were also positively associated with overall PM performance and specifically with MK, PC, and PBLI domains; however, the strength of the positive association of Step 2 CK was weaker than that of GHHS membership. The mean effect size of GHHS membership on overall PM ratings was 0.15, while the mean effect size of year of residency was 1.15; this translates to a roughly 1.5 month increase in performance of GHHS members compared to their non-GHHS peers.
GHHS recognizes students who are selected by their peers as exemplars of humanism, professionalism, and communication skills.29 Its strong association with Milestones, particularly in domains like ICS and Professionalism, suggests that the qualities recognized by GHHS are highly relevant to success in pediatric residency. Previous literature has explored the value placed on GHHS by PDs, with mixed findings and calls for more data on its predictive value.7,30 A 2021 single-site study evaluating the association between internal medicine residency applicant characteristics and Milestone performance during intern year also found a positive association between GHHS membership and Milestone performance in medical knowledge.15 Our study provides empirical evidence supporting its predictive validity for Milestone achievement across several competency domains in pediatrics.
Previous studies of pediatric, internal medicine, and surgery residents have found Step 2 CK to be predictive of resident performance, including both cognitive and non-cognitive domains.9-11,31 While performance on Step 2 CK may reflect knowledge and clinical skills relevant to some competency domains, our results suggest it is not as broadly or strongly predictive of Milestone achievement in pediatrics as GHHS. The finding that USMLE Step 1 scores were negatively associated with the SBP competency domain is noteworthy, and consistent with previous studies that have reported a lack of correlation or even negative correlations between USMLE scores and non-cognitive performance or faculty evaluations.7,12,15-18,20-24 A 2006 study of radiation oncology residency applicants found that applicants with a USMLE Step 1 score of 235 or greater who listed publications were more than 7 times more likely to have inaccurately listed citations.32 The lack of association with overall PM ratings and negative association for the SBP domain observed in this study provides further support to the argument that Step 1 scores do not adequately measure or predict performance, and may give support to the decision to shift Step 1 to pass/fail reporting.
While our study provides support to the inclusion of GHHS membership as a component of residency applicant evaluation, it should be noted that the pool of applicants who have been inducted into GHHS is small (16% [85 of 518] of our sample; GHHS student chapters may induct up to 15% of each medical student class). There are only 7 non-US medical schools with GHHS chapters; among approximately 230 allopathic and osteopathic US medical school campuses, 184 US chapters are listed on the GHHS website (80%); at the time our data were collected, there were approximately 160 chapters.33,34 Members of GHHS are peer-selected; as such, selection may favor extroverted students and may be impacted by bias. Disparities in medical student honor society membership are well described and may place applicants from marginalized backgrounds at a disadvantage; disparities in GHHS membership, however, have been shown to exist across fewer identities and to sometimes favor the marginalized group, when compared with AOA.35
This study has limitations. It is a retrospective study conducted within a single specialty, and while it is multi-institutional, data were provided by a subset of invited programs. Pediatric residency training differs from other specialties in many important ways, including patient population, culture, training structure, communication demands, and career pathways; generalizability to other specialties would require further research. Because the data collection window preceded the shift of USMLE Step 1 reporting to pass/fail, the data analyzed differs somewhat from that available for most current residency applicants. The data collection window also preceded the COVID-19 pandemic, the transition to virtual interviews, and the resultant increase in application volumes, as well as the advent of program signaling and recent decline in number of applicants to pediatric residency. It is possible that our findings would be different should we repeat the study today. The reliance on ERAS-filterable metrics means other potentially valuable predictors, such as detailed clinical evaluations, narrative comments, or performance during structured interviews or sub-internships, were not the primary focus of this analysis, although these have been explored in other studies with varying degrees of success. The Milestone assessment process itself is inherently subjective, with variability between institutions and raters; efforts were made to standardize data collection and analysis, including the inclusion of program as a clustering variable in the mixed-models regression. Finally, our study includes only matched residents (not all applicants to pediatric residency), and is thus subject to survivorship bias.
Despite these limitations, the findings offer direction for pediatric residency selection. Including metrics like GHHS membership as a component of holistic application review could lead to a selection process better aligned with identifying applicants likely to succeed across the comprehensive set of competencies required in modern pediatric practice. The development of additional filterable measures of clinical competence outside of Medical Knowledge, in addition to GHHS, has the potential to improve the ability of PDs to efficiently and effectively screen applications and to reduce the likelihood of bias in the application review process.
Conclusions
In this multisite study of pediatric residents, among the filterable ERAS metrics examined, GHHS membership showed the strongest positive association with pediatric residency success as measured by mean ACGME Pediatrics Milestone ratings. USMLE Step 2 CK scores had a weaker positive association, and USMLE Step 1 scores were negatively associated with success in the Systems-Based Practice competency domain.
Author Notes
Funding: Support for this study was provided in part by the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network (APPD LEARN), which assisted with study design; data collection, analysis, and interpretation; and in the writing of the report. Alan Schwartz, PhD, JD, is supported as APPD LEARN Director through a contract from APPD to the University of Illinois at Chicago.
Conflict of interest: The authors declare they have no competing interests.
Disclosure: AI was not used in writing this article.
This work was previously presented as a poster at the Association of Pediatric Program Directors Annual Spring Meeting, March 25-28, 2025, Atlanta, Georgia, USA.
References
- 1.McGaghie WC, Cohen ER, Wayne DB. Are United States Medical Licensing Exam Step 1 and 2 scores valid measures for postgraduate medical residency selection decisions? Acad Med. 2011;86(1):48–52. doi: 10.1097/ACM.0b013e3181ffacdb. doi: [DOI] [PubMed] [Google Scholar]
- 2.Prober CG, Kolars JC, First LR, Melnick DE. A plea to reassess the role of United States Medical Licensing Examination Step 1 scores in residency selection. Acad Med. 2016;91(1):12–15. doi: 10.1097/ACM.0000000000000855. doi: [DOI] [PubMed] [Google Scholar]
- 3.Sklar DP. Matchmaker, matchmaker, make me a match: is there a better way? Acad Med. 2019;94(3):295–297. doi: 10.1097/ACM.0000000000002553. doi: [DOI] [PubMed] [Google Scholar]
- 4.Radabaugh CL, Hawkins RE, Welcher CM, et al. Beyond the United States Medical Licensing Examination score: assessing competence for entering residency. Acad Med. 2019;94(7):983–989. doi: 10.1097/ACM.0000000000002728. doi: [DOI] [PubMed] [Google Scholar]
- 5.Chen DR, Priest KC, Batten JN, Fragoso LE, Reinfeld BI, Laitman BM. Student perspectives on the “Step 1 climate” in preclinical medical education. Acad Med. 2019;94(3):302–304. doi: 10.1097/ACM.0000000000002565. doi: [DOI] [PubMed] [Google Scholar]
- 6.Angus SV, Williams CM, Stewart EA, Sweet M, Kisielewski M, Willett LL. Internal medicine residency program directors’ screening practices and perceptions about recruitment challenges. Acad Med. 2020;95(4):582–589. doi: 10.1097/ACM.0000000000003086. doi: [DOI] [PubMed] [Google Scholar]
- 7.Lipman JM, Colbert CY, Ashton R, et al. A systematic review of metrics utilized in the selection and prediction of future performance of residents in the United States. J Grad Med Educ. 2023;15(6):652–668. doi: 10.4300/JGME-D-22-00955.1. doi: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Mccaskill QE, Kirk JJ, Barata DM, Wludyka PS, Zenni EA, Chiu TT. USMLE Step 1 scores as a significant predictor of future board passage in pediatrics. Ambul Pediatr. 2007;7(2):192–195. doi: 10.1016/j.ambp.2007.01.002. doi: [DOI] [PubMed] [Google Scholar]
- 9.Welch TR, Olson BG, Nelsen E, Beck Dallaghan GL, Kennedy GA, Botash A. United States Medical Licensing Examination and American Board of Pediatrics Certification Examination results: does the residency program contribute to trainee achievement. J Pediatr. 2017;188:270–274.e3. doi: 10.1016/j.jpeds.2017.05.057. doi: [DOI] [PubMed] [Google Scholar]
- 10.Sharma A, Schauer DP, Kelleher M, Kinnear B, Sall D, Warm E. USMLE Step 2 CK: best predictor of multimodal performance in an internal medicine residency. J Grad Med Educ. 2019;11(4):412–419. doi: 10.4300/JGME-D-19-00099.1. doi: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Spurlock DR, Jr, Holden C, Hartranft T. Using United States Medical Licensing Examination (USMLE) examination results to predict later in-training examination performance among general surgery residents. J Surg Educ. 2010;67(6):452–456. doi: 10.1016/j.jsurg.2010.06.010. doi: [DOI] [PubMed] [Google Scholar]
- 12.Harfmann KL, Zirwas MJ. Can performance in medical school predict performance in residency? A compilation and review of correlative studies. J Am Acad Dermatol. 2011;65(5):1010–1022.e2. doi: 10.1016/j.jaad.2010.07.034. doi: [DOI] [PubMed] [Google Scholar]
- 13.Fening K, Vander Horst A, Zirwas M. Correlation of USMLE Step 1 scores with performance on dermatology in-training examinations. J Am Acad Dermatol. 2011;64(1):102–106. doi: 10.1016/j.jaad.2009.12.051. doi: [DOI] [PubMed] [Google Scholar]
- 14.Boyse TD, Patterson SK, Cohan RH, et al. Does medical school performance predict radiology resident performance? Acad Radiol. 2002;9(4):437–445. doi: 10.1016/s1076-6332(03)80189-7. doi: [DOI] [PubMed] [Google Scholar]
- 15.Golden BP, Henschen BL, Liss DT, Kiely SL, Didwania AK. Association between internal medicine residency applicant characteristics and performance on ACGME milestones during intern year. J Grad Med Educ. 2021;13(2):213–222. doi: 10.4300/JGME-D-20-00603.1. doi: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Tolan AM, Kaji AH, Quach C, Hines OJ, De Virgilio C. The Electronic Residency Application Service application can predict accreditation council for graduate medical education competency-based surgical resident performance. J Surg Educ. 2010;67(6):444–448. doi: 10.1016/j.jsurg.2010.05.002. doi: [DOI] [PubMed] [Google Scholar]
- 17.Miller B, Nowalk A, Ward C, Walker L, Dewar S. Pediatric residency milestone performance is not predicted by the United States Medical Licensing Examination Step 2 Clinical Knowledge. MedEdPublish. 2023;13:308. doi: 10.12688/mep.19873.1. doi: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Borowitz SM, Saulsbury FT, Wilson WG. Information collected during the residency Match process does not predict clinical performance. Arch Pediatr Adolesc Med. 2000;154(3):256–260. doi: 10.1001/archpedi.154.3.256. doi: [DOI] [PubMed] [Google Scholar]
- 19.Daly KA, Levine SC, Adams GL. Predictors for resident success in otolaryngology. J Am Coll Surg. 2006;202(4):649–654. doi: 10.1016/j.jamcollsurg.2005.12.006. doi: [DOI] [PubMed] [Google Scholar]
- 20.Spitzer AB, Gage MJ, Looze CA, Walsh M, Zuckerman JD, Egol KA. Factors associated with successful performance in an orthopaedic surgery residency. J Bone Joint Surg Am. 2009;91(11):2750–2755. doi: 10.2106/JBJS.H.01243. doi: [DOI] [PubMed] [Google Scholar]
- 21.Stohl HE, Hueppchen NA, Bienstock JL. Can medical school performance predict residency performance? Resident selection and predictors of successful performance in obstetrics and gynecology. J Grad Med Educ. 2010;2(3):322–326. doi: 10.4300/jgme-d-09-00101.1. doi: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Ryden AG, Fuller TC, Rose RS, Lappe KL, Raaum S, Johnson SA. Comparison of internal medicine applicant and resident characteristics with performance on ACGME milestones. Med Educ Online. 2023;28(1):2211359. doi: 10.1080/10872981.2023.2211359. doi: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Greenburg DL, Durning SJ, Cohen DL, Cruess D, Jackson JL. Identifying medical students likely to exhibit poor professionalism and knowledge during internship. J Gen Intern Med. 2007;22(12):1711–1717. doi: 10.1007/s11606-007-0405-z. doi: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Brothers TE, Wetherholt S. Importance of the faculty interview during the resident application process. J Surg Educ. 2007;64(6):378–385. doi: 10.1016/j.jsurg.2007.05.003. doi: [DOI] [PubMed] [Google Scholar]
- 25.National Resident Matching Program. Results of the 2020 NRMP Program Director Survey. Aug, 2020. Published. Accessed June 9, 2026. https://www.nrmp.org/wp-content/uploads/2022/01/2020-PD-Survey.pdf.
- 26.Li STT, Tancredi DJ, Schwartz A, et al. Competent for unsupervised practice: use of pediatric residency training milestones to assess readiness. Acad Med. 2017;92(3):385–393. doi: 10.1097/ACM.0000000000001322. doi: [DOI] [PubMed] [Google Scholar]
- 27.Turner TL, Bhavaraju VL, Luciw-Dubas UA, et al. Validity evidence from ratings of pediatric interns and subinterns on a subset of pediatric milestones. Acad Med. 2017;92(6):809–819. doi: 10.1097/ACM.0000000000001622. doi: [DOI] [PubMed] [Google Scholar]
- 28.Schwartz A, King B, Mink R, et al. The APPD Longitudinal Educational Assessment Research Network’s first decade. Pediatrics. 2023;151(5):e2022059113. doi: 10.1542/peds.2022-059113. doi: [DOI] [PubMed] [Google Scholar]
- 29.Levin R, Cohen J, White L. Gold Humanism Honor Society membership indicator to be added to ERAS applications beginning with 2016 application cycle. J Grad Med Educ. 2015;7(1):136. doi: 10.4300/JGME-D-14-00723.1. doi: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Rosenthal S, Howard B, Schlussel YR, et al. Does medical student membership in the Gold Humanism Honor Society influence selection for residency? J Surg Educ. 2009;66(6):308–313. doi: 10.1016/j.jsurg.2009.08.002. doi: [DOI] [PubMed] [Google Scholar]
- 31.Hemrajani R, Vettese T, Law K, Turbow S. Association of interview and holistic review metrics with resident performance-related difficulties in an internal medicine residency program. J Grad Med Educ. 2023;15(5):564–571. doi: 10.4300/JGME-D-22-00726.1. doi: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Yang GY, Schoenwetter MF, Wagner TD, Donohue KA, Kuettel MR. Misrepresentation of publications among radiation oncology residency applicants. J Am Coll Radiol. 2006;3(4):259–264. doi: 10.1016/j.jacr.2005.12.001. doi: [DOI] [PubMed] [Google Scholar]
- 33.University of Hawaii News. Med school students send ‘aloha’ to alumni on the frontlines. May 5, 2020 Published. Accessed May 24, 2026. https://www.hawaii.edu/news/2020/05/05/jabsom-students-sends-aloha/ [Google Scholar]
- 34.Gold Humanism Honor Society. Accessed May 24, 2026. https://www.gold-foundation.org/programs/ghhs/chapters/
- 35.Hill KA, Desai MM, Chaudhry SI, et al. Association of marginalized identities with Alpha Omega Alpha Honor Society and Gold Humanism Honor Society Membership among medical students. JAMA Netw Open. 2022;5(9):e2229062. doi: 10.1001/jamanetworkopen.2022.29062. doi: [DOI] [PMC free article] [PubMed] [Google Scholar]