Understanding the Interview and Ranking Behaviors of Unmatched International Medical Students and Graduates in the 2013 Main Residency Match

Abstract

Background 

Over the past decade, the number of unfilled positions in the National Resident Matching Program (NRMP) Main Residency Match has declined by one-third, while the number of unmatched applicants has grown by more than 50%, largely due to a rise in the number of international medical school students and graduates (IMGs). Although only half of IMG participants historically have matched to a first-year position, the Match experiences of unmatched IMGs have not been studied.

Objective 

We examined differences in interview and ranking behaviors between matched and unmatched IMGs participating in the 2013 Match and explored strategic errors made by unmatched IMGs when creating rank order lists.

Methods 

Rank order lists of IMGs who failed to match were analyzed in conjunction with their United States Medical Licensing Examination (USMLE) Step 1 scores and responses on the 2013 NRMP Applicant Survey. IMGs were categorized as “strong,” “solid,” “marginal,” or “weak” based on the perceived competitiveness of their USMLE Step 1 scores compared to other IMG applicants who matched in the same specialty. We examined ranking preferences and strategies by Match outcome.

Results 

Most unmatched IMGs were categorized as “marginal” or “weak”. However, unmatched IMGs who were non-US citizens presented more competitive USMLE Step 1 scores compared to unmatched IMGs who were US citizens. Unmatched IMGs were more likely than matched IMGs to rank programs at which they did not interview and to rank programs based on their perceived likelihood of matching.

Conclusions 

The interview and ranking behaviors of IMGs can have far-reaching consequences on their Match experience and outcomes.

What was known and gap

Only half of international medical graduates (IMGs) match into a residency position, and the factors that influence success have not been fully explored.

What is new

A study of interview and ranking behaviors of matched and unmatched IMGs, highlighting strategic errors made by unmatched IMGs.

Limitations

Use of self-reported data, potential for respondent bias, and assessment limited to interviewing and ranking practices.

Bottom line

The interview and ranking behaviors of IMGs affect their match outcomes, which suggest a need for added education of applicants.

Introduction

Each year, the National Resident Matching Program (NRMP) conducts the Main Residency Match (the Match) in which medical school students and graduates (applicants) are matched to residency positions in US graduate medical education programs (programs). In 2013, 34 355 applicants and 4621 programs offering 29 171 postgraduate year (PGY)-1 and PGY-2 positions participated in the Match.¹

For more than 30 years, the number of applicants participating in the Match has exceeded the number of available positions.¹ Between 1992 and 2013, the number of unfilled first-year positions declined from more than 4500 to approximately 1000. At the same time, the number of unmatched applicants grew considerably, mainly due to a rise in the number of international medical graduate (IMG) participants in the Match (figure 1).²

FIGURE 1.

Number of Unmatched Applicants by Type and Number of Unfilled Positions in Main Residency Match, 1992–2013

Abbreviation: IMGs, international medical graduates

Research efforts have been directed toward identifying the traits of applicants who have successful matching experiences. The NRMP publishes “Charting Outcomes in the Match,” which provides normative, specialty-specific data on characteristics of applicants who match to their preferred specialties.^3,4 The NRMP also publishes the results of a biennial program director survey that highlights factors program directors consider when selecting applicants to interview and rank.⁵ Other researchers have also explored factors deemed important to applicants and program directors during the matching process.^6–8 Findings point to applicants' academic strength, performance on the United States Medical Licensing Examination (USMLE), and years since graduation from medical school as crucial to a successful interview and matching experience.

Strategies surrounding the creation of rank order lists (ROLs) also have been widely addressed in research. Submitting a ROL based on the applicant's perceived chance of matching is unlikely to improve the applicant's Match result, and the NRMP advises applicants to create ROLs based on true preferences, not their speculated probability of matching to any specific program.^9–12 The NRMP also advises applicants to rank all programs where they would be willing to train. Data for the past 13 years show that, in general, unmatched applicants had shorter ROLs than matched applicants,¹³ a finding supported by analysis of contiguous ranks of applicants' preferred specialties in “Charting Outcomes in the Match.”³

Although the body of research examining the characteristics of successful Match candidates is considerable, much less effort has been devoted to evaluating the experiences of unmatched applicants, especially IMGs. Since 1992, approximately half of the IMGs who submitted program preferences in the Match have failed to obtain a position, compared to only 6% of US allopathic medical school senior students. This article describes interview and ranking behaviors of unmatched IMGs in an effort to better understand the impact of those behaviors on the Match outcomes. Applicants participated in the 2013 Main Residency Match, which is the most recent year for which all required data were available.

Methods

We combined data from the 2013 Main Residency Match with USMLE scores and responses to the 2013 NRMP Applicant Survey by US citizen and non–US citizen IMGs. We considered US citizen IMGs to be US citizens who attended medical school outside the United States and Canada. Non–US citizen IMGs are individuals who were not US citizens at the time of the Match and who attended medical schools outside the United States and Canada. The Applicant Survey solicits information about factors that influence applicants' selection and ranking of programs and strategies applicants employed when creating ROLs.

We used scores on the USMLE Step 1 Basic Science examination to gauge perceived competitiveness of matched and unmatched IMGs. The USMLE Step 1 examination assesses the understanding and application of basic science principles to the practice of medicine.¹⁴ Although the examination is not designed to measure academic achievement, program directors cite it as 1 of the most important factors in selecting candidates for interviews and ranking.⁵ Because applicants can take the Step 1 examination multiple times, we used the last Step 1 score obtained prior to the ROL deadline in 2013. A passing Step 1 score is an examination requirement for certification by the Educational Commission for Foreign Medical Graduates (ECFMG).¹⁵ An IMG must meet all ECFMG examination requirements to participate in the Match.

We defined unmatched applicants as those who submitted ROLs of programs and did not obtain any position when the matching algorithm was processed. In the 2013 Match, 6228 IMGs (2339 US citizen IMGs and 3889 non–US citizen IMGs) who submitted ROLs failed to match. USMLE Step 1 scores were available for 5915 of them (95%), each of whom was classified into 1 of 4 categories based on the perceived competitiveness of their Step 1 scores relative to the mean scores of IMGs who matched in the same preferred specialty. Preferred specialty is defined as the specialty listed first on the applicant's ROL. Applicants who ranked preliminary positions first were treated as not having a preferred specialty. The classification used the following cut points: (1) “strong” applicants (scores > 1 SD above group average, n = 533); (2) “solid” applicants (scores between 0.01 and 1 SD above group average, n = 1494); (3) “marginal” applicants (scores between 1 and 0 SD below group average, n = 2341); and “weak” applicants (scores < 1 SD below group average, n = 1547).

We compared the numbers of interview invitations that were received to determine whether an applicant had attended all interviews. We also compared the number of interviews attended and the programs ranked to determine whether an applicant had ranked all programs where the applicant had interviewed. Lastly, we examined the placement of unmatched applicants on the ROLs of unfilled programs to determine whether an applicant could have matched. If an unmatched applicant was ranked by an unfilled program not ranked by the applicant, the applicant was considered to have had an opportunity to match.

We used Pearson χ² tests on matched versus unmatched IMGs to examine ranking strategies as reported on the Applicant Survey. Analysis focused on 714 unmatched US citizen and 1681 non–US citizen IMGs (n = 2395) with a preferred specialty for whom USMLE Step 1 scores were available.

This study retrospectively examined the Match outcomes, USMLE scores, and NRMP Applicant Survey responses of IMGs. The Applicant Survey was reviewed and approved by the American Institutes of Research Institutional Review Board. Participation in the Applicant Survey was voluntary, confidentiality was assured, and answering the questions reflected agreement to participate. The USMLE scores were provided by the ECFMG through an executed data-sharing agreement between the NRMP and the ECFMG.

Results

The majority of unmatched applicants in the 2013 Match were either US citizen IMGs (27.9%, 2339 of 8388, of all unmatched applicants) or non–US citizen IMGs (46.4%, 3889 of 8388, of all unmatched applicants). More than two-thirds (69.0%), 4117 of the 5974 unmatched IMGs with preferred specialties favored family medicine, internal medicine, or pediatrics.

Competitiveness of Unmatched IMGS

Using USMLE Step 1 scores relative to the mean scores of IMG applicants who matched in the same preferred specialty, most unmatched IMGs were identified as “marginal” or “weak.” At the same time, more than 40% (1552 of 3701) of unmatched non–US citizen IMGs were identified as “strong” or “solid” compared to approximately 20% (475 of 2214) of unmatched US citizen IMGs (table 1).

TABLE 1.

Distribution of Unmatched International Medical Graduates (IMGs) by Type Based on Competitiveness of United States Medical Licensing Examination Step 1 Scores

Interview and Ranking Behavior

The 2013 NRMP Applicant Survey was sent to 12 986 IMGs, and responses were received from 6261 IMGs (48%). Of those, 2395 (38%) were unmatched. On average, unmatched IMGs received very few interview offers relative to the number of applications submitted. Nonetheless, 11% (65 of 585) of unmatched US citizen IMGs did not attend all interviews, and 7% (101 of 1424) declined to rank all programs at which they had interviewed. By comparison, 17% (97 of 585) of unmatched non–US citizen IMGs did not attend all interviews, and 22% (314 of 1424) declined to rank all programs at which they interviewed. For each interview invitation received, “weak” unmatched non–US citizen IMGs submitted fewer applications than those in the “solid” and “marginal” categories, and they attended the fewest interviews.

Analysis of the ROLs of unfilled programs showed that 70 unmatched IMGs who had a preferred specialty (27 US citizens and 43 non–US citizens) would have matched if they had ranked unfilled programs that ranked them: Of the individuals who would have matched, 19 (27%) were in the “strong” grouping, 24 (34%) were in the “solid” grouping, 21 (30%) were in the “marginal” grouping, and 6 (8.6%) were in the “weak” grouping, according to USMLE performance (table 2).

TABLE 2.

Applications, Interviews, and Programs of Unmatched International Medical Graduates Ranked by Applicant Category^a

Ranking Strategies

Analysis of responses to the 2013 NRMP Applicant Survey showed that unmatched IMGs were more likely than their matched IMG counterparts to rank programs based on the likelihood of matching (P < .001) and less likely to rank a mix of competitive and less competitive programs (P < .001) or to rank programs according to their true preference (P < .001). Data also showed that unmatched non–US citizen IMGs were less likely than unmatched US citizen IMGs to rank all the programs they were willing to attend. Unmatched US citizen IMGs were more likely than non–US citizen IMGs to rank programs where they had not interviewed (figures 2 and 3).

FIGURE 2.

Ranking Strategies of US Citizen International Medical Graduates by Match Status in 2013 Main Residency Match

FIGURE 3.

Ranking Strategies of Non-US International Medical Graduates by Match Status in 2013 Main Residency Match

Discussion

Failing to obtain a residency position is a reality for many international medical school students and graduates. The match rate for IMGs has improved over the past few years, perhaps because the NRMP's “all in” policy requiring Match-participating programs to place all positions in the Match has prompted many well-qualified IMGs to enter the Match to obtain a position. However, Match rates for IMGs remain considerably below those of US allopathic seniors.¹⁶

The findings of our analysis of unmatched IMGs' behaviors suggest that certain interview and ranking strategies can have far-reaching consequences. Programs did not base interview invitations solely on USMLE Step 1 scores, as “weak” non–US citizen IMGs submitted fewer applications for each interview invitation received compared to “solid” and “marginal” non–US citizen IMGs. Program interest in applicants is multifactorial, and IMGs who did not attend all interviews failed to capitalize on every opportunity to market themselves. The strategic focus (or lack thereof) when creating ROLs also is telling, given the sizable percentage of respondents to the Applicant Survey who reported that they declined to rank all programs at which they interviewed or did not rank all programs they would be willing to attend. The reasons applicants decide not to attend all interviews or rank all programs at which they interviewed likely vary, and the impact of personal factors on ranking decisions merits further evaluation. Geographic, financial, and cultural considerations, both in the United States and abroad, may be particularly salient for non–US citizen IMGs.

A misunderstanding about how the Match works also can affect the outcome. Applicants will not match to programs where they have not interviewed, and to rank those programs suggests a fundamental misunderstanding of the relationship between the interview and Match success. At the same time, failing to rank programs based on true preferences or ranking programs based on the perceived likelihood of matching implies a poor understanding of how the matching algorithm works. Many for-profit companies claim to be able to maximize IMGs' chances of Match success, but NRMP staff routinely hear that program directors disdain these services. Increased education could improve Match outcomes for IMGs and empower them to independently champion their own capabilities.

A limitation of our study is that it was narrowly focused to address only behaviors related to interviewing with and ranking programs. In addition, competitiveness varies among specialties, and the results of the study cannot be applied to a single specialty. Lastly, analyses of the Applicant Survey results were based on self-reported data, and the response rate of 48% could limit the generalizability of the results.

IMGs do not match to residency programs for many reasons. Although IMGs are persistent in seeking graduate medical training, and many unmatched IMGs eventually obtain a position,¹⁷ the field would benefit from additional studies of unmatched IMGs, particularly how factors other than USMLE performance affect Match outcomes. Future studies could examine the definition of what constitutes an acceptable program, including such variables as program geographic location, cultural preferences, and preferred specialty by IMG type. A broader study profiling unmatched IMGs would add important information about the adequacy and demographics of the physician workforce. It also would provide valuable insight into the complexity of the Match experience for this applicant population.

Conclusion

Interviewing and ranking behaviors and strategies can affect the Match outcomes for IMGs. A small number of IMGs, regardless of the perceived competitiveness of their USMLE scores, would have matched if they had ranked unfilled programs that ranked them. Enhanced education about the relationship between interviewing and matching and how the matching algorithm works could change the ranking strategies employed by unmatched IMGs and result in a positive Match outcome for some.