Probability of Success in the Ophthalmology Residency Match: Three-Year Outcomes Analysis of San Francisco Matching Program Data

Abstract

Objective

To develop a probability model of matching into a US ophthalmology residency program using San Francisco Matching Program (SF Match) data.

Design

Retrospective data analysis of de-identified application and matching data.

Participants

Registrants for the 2013, 2014, and 2015 ophthalmology residency matches conducted by the SF Match.

Methods

Descriptive statistics of candidates, comparison of continuous and categorical variables between matched and nonmatched candidates, and linear regression modeling were performed. A recursive partitioning method was used to create a probability of matching algorithm.

Main Outcome Measures

Probability of successfully matching based on quantifiable candidate characteristics.

Results

Over the 3-year period, 1,959 individuals submitted an average of 64 applications and received a mean of nine interview invitations. The overall match rate was 71%, with 78% matching at one of their top five choices. Successful matches were more likely to occur in US medical school graduates (78% vs 20%, p < 0.001) and applicants on their first attempt (76% vs 29%, p < 0.001). The association between matching and number of programs applied became negative with > 48 applications. Probability of matching was “high” (> 80%) among US graduates with a step 1 United States Medical Licensing Examination (USMLE)score>243(regardless of number of programs applied to), a step 1 USMLE score of 231 to 243 who applied to at least 30 programs, and first-time applicants with a step 1 score >232. No international medical graduates or repeat applicants had a “high” probability of matching.

Conclusions

Although advice must be individualized for each candidate, applicants for ophthalmology residency who fall into a “high” probability of matching group are likely to be successful with applications to 45 or fewer programs. Applying to 80 or more programs should be considered for international medical graduates and/or applicants who are previously unmatched. Modification of the match application data form may allow more detailed analysis of variables such as Alpha Omega Alpha or Gold Humanism Honor Society membership, research activity, and composite evaluation on a standardized letter of recommendation.

Candidates for ophthalmology residency apply through the ophthalmology residency matching program, administered by the San Francisco Residency and Fellowship Matching Services (SF Match). A universal application is submitted to SF Match, which in turn distributes it to the programs specified by the applicant. After completion of the interview process, the programs and the applicants each submit rank lists to SF Match, which uses an algorithm based on preferences submitted to place candidates in a training program.

Ophthalmology continues to be perceived as one of the most competitive medical specialties. This is demonstrated by the steady increase in the mean number of applications submitted per applicant over the past 10 years, from 48 in 2008, to 68 in 2017 based on SF Match tracking data. Despite this trend, the overall match rate has remained relatively stable (mean 74%, range 70–78%, for the period 2008–2017). This continued rise in the number of applications submitted per applicant places a significant financial burden on students as well as a tremendous administrative load on residency programs. The cost structure consists of a $100 registration fee and a $60 base fee for application to 10 programs, with incremental cost increases of $ 100 to 350 for each additional 10 programs. Thus, the total fee for applying to 40 programs is $610, which increases to $2,010 for 80 programs and $2,710 for 100 programs.

Due to the perceived difficulty of the ophthalmology match, applicants may feel the need to apply to more programs to gain more interview invitations and increase the likelihood of matching successfully. However, this assumption may be based on incomplete or inaccurate information from peers or advisors, given that the match is a complex process based on both quantitative and qualitative characteristics.

Although the National Residency Matching Program (NRMP) provides an annual report detailing match outcomes for other specialties, this information is not available for ophthalmology. For example, NRMP data for otolaryngology (found at https://students-residents.aamc.org/applying-residency/article/apply-smart-data-consider/) demonstrate that candidates with United States Medical Licensing Examination (USMLE) step 1 scores > 250 have a mean match rate of 88% with diminishing chances of matching for students who applied to more than 39 programs; similar data for USMLE score 237–249 and <237 are 80% and 66%, and 43 and 44 programs, respectively. Providing similar data to candidates for ophthalmology residency programs will assist candidates in effectively allocating effort and investment in the application process, while at the same time maximizing the successful match rate among qualified applicants.

The goal of this study was to develop a model to predict the probability of matching to an ophthalmology residency program based on applicant characteristics and number of application submissions by analyzing SF Match data.

Methods

Data Source:

Deidentified application and ophthalmology residency matching data for the 2013, 2014 and 2015 match cycles collected by the Association of University Professors of Ophthalmology were used in the study. Independent variables include the number of program applications, the number of invitations for interview, USMLE step 1 score, international medical graduate (IMG) status, Alpha Omega Alpha membership, presence of research activity as stated by the applicant, number of times that the applicant applied for an ophthalmology residency, number of programs ranked by the applicant, the total number of programs that ranked the applicant, the matched position on the program’s rank list, and the applicant’s rank of the matched program. Data on individual interview invitations offered and interviews completed is provided by programs and applicants, respectively, through a self-reporting process, which may result in inadvertent errors, and is discussed in more detail later.

Statistical Analysis Methods:

Descriptive statistics were estimated to summarize the characteristics of the applicants across the three application cycles. Summaries were presented after stratifying by program matching success, first versus repeat attempt, and IMG status. An independent sample t-test was used to compare the means of continuous variables, and a chi-square test was used to compare categorical variables between independent groups. The Wilcoxon rank sum test and Fisher’s exact test were used in the case of skewed distributions for continuous variables or small expected frequencies for categorical variables, respectively. Linear and segmented regression modeling¹ was used to quantify the association between number of applications submitted and the number of interview invitations. SAS 9.4 was used to perform the descriptive summary and regression modeling (SAS Institute, Cary, NC).

To provide an intuitive model to predict the probability of matching, a recursive partitioning method was used in the multivariable analysis.² The recursive partitioning analysis used USMLE score and number of program applications as the two predictors of matching status. The analysis was first stratified by IMG status and then by first-attempt status as a separate stratification factor. Models were estimated separately within each resulting subgroup. The minimum terminal node size was set to 20. A fivefold cross-validation method was used to identify a best-fit model. A fivefold R² value was presented, which was the proportion of the variability in the response that was explained by the model. The resulting models were presented as decision trees. Subgroups with similar probabilities of matching in the decision trees were combined. JMP software was used to fit the decision trees (JMP version 11.2.0, 2013 SAS Institute).³

Results

Across the 2013, 2014, and 2015 application cycles, 1,959 unique individuals were identified. ►Table 1 includes a summary of characteristics for all candidates. On average, applicants submitted 64 applications (standard deviation [SD] = 27, 95% confidence interval [CI]: 62.9–65.3), received 9 invitations for interview (SD = 6, 95% CI: 9.0–9.6), and had a score of 238 in USMLE step 1 (SD = 16, 95% CI: 237.6–239.0). Among applicants, 13% were IMGs (95% CI: 11–14%), 88% were first-time submissions (95% CI: 87–90%), and 71% accomplished a successful match (95% CI: 69–73%). For applicants who submitted multiple applications (n = 226, 12%), the most recent attempt was retained in the study. The most recent attempt was selected to reflect the overall status of each applicant during the study time period (2013–2015) and was necessary to ensure independent applicants, avoiding correlated measures, in the analysis set. The approach results in a slightly increased probability of matching (71%), compared with analysis of first attempts only (68%).

Table 1.

Descriptive summary for all applicants (n = 1,959)

Variable	Mean	SD	Min	Max	Median	Q1	Q3	IQR
No. of program applications	64.06	26.91	1	113	63	45	83	38
No. of invites for interview	9.29	6.3	0	29	9	4	14	10
USMLE step 1 scorea	238.3	16.42	182	275	241	229	250	21
No. of attempts	1.16	0.49	1	5	1	1	1	0
No. of institutions ranked by the applicantb	10.45	5.04	1	113	11	8	13	5
Total no. of programs that ranked the applicantb	9.85	4.19	1	14	10	1	4	3
Matched position on the program’s rank listb	12.49	9.52	1	57	11	5	18	13
Applicant’s rank of the matched programb	2.92	2.41	1	14	2	1	1	0
	N	%
IMG	247	13
Successful match	1391	71
AOA membera	422	44
Published research	1948	99
Invited for ≥ 1 interview	1802	92
First attempt	1733	88

Abbreviations: AOA, Alpha Omega Alpha Honor Medical Society; IMG, international medical graduate; IQR, interquartile range (75th percentile–25th percentile); Max, maximum; Min, minimum; Q1, 25th percentile; Q3, 75th percentile; SD, standard deviation; USMLE, United States Medical Licensing Examination.

^aNumber of missing observations for USMLE = 46, AOA status = 992.

^bFor these four variables, summaries are restricted to applicants who matched (n = 1,391).

►Table 2 includes a summary of characteristics according to matching status (matched or did not match). Data analyzed included number of interviews offered, number of programs ranked by the candidate, number of programs that ranked the applicant, and rank position of program by candidate and candidate by program. Applicants who successfully matched submitted 66 applications (SD 24) and ranked 12 programs (SD 7) on average, whereas those who failed to match submitted 60 applications (SD 33) (p = 0.0004) and ranked 2 programs (SD 4) on average (p < 0.0001). In addition, matched applicants performed better on USMLE step 1, with a mean score of 243 (SD 13), compared with a mean score of 226 (SD 18) for unmatched applicants (p < 0.0001). Furthermore, 95% of matched applicants were applying on their first attempt, whereas only 72% of unmatched applicants were on their first attempt (p < 0.0001). IMGs were significantly less likely to be in the matched group (4% vs 35%, p < 0.0001).

Table 2.

Descriptive summary of characteristics according to matching status

Variable	Matched (n = 1,391)		Did not match (n = 568)		p-Valuea
	Mean	SD	Mean	SD
No. of program applications	65.63	23.64	60.21	33.31	0.0004
No. of invites for interviewb	12	7	2	4	<0.0001b
USMLE step 1 scorec	242.99	13.28	226.23	17.54	<0.0001
No. of attempts	1.06	0.28	1.4	0.74	NPd
	Count	%	Count	%	p-Valuee
IMG	49	4	198	35	<0.0001
AOA memberc	391	55	31	12	<0.0001
Published research	1390	99	558	98	<0.0001
Invited for ≥ 1 interview	1391	100	411	72	NPf
First attempt	1325	95	408	72	<0.0001

Abbreviations: AOA, Alpha Omega Alpha Honor Medical Society; IMG, international medical graduate; IQR, interquartile range (75th percentile–25th percentile); Max, maximum; Min, minimum; NP, not performed; Q1, 25th percentile; Q3, 75th percentile; SD, standard deviation; USMLE, United States Medical Licensing Examination.

^ap-Values are based on t-test.

^bDescriptive summary based on median and interquartile range. p-Value based on Wilcoxon rank sum test.

^cNumber of missing observations for USMLE (matched) = 13, AOA status (matched) = 679, USMLE (not matched) =33, AOA status (not matched) =313.

^dGiven the skewed nature of the variable indicating the number of application attempts, hypothesis testing is based on the dichotomous variables indicating the first application attempt.

^ep-Value for published research is based on Fisher’s exact test, other p-values are based on chi-square test.

^fHypothesis testing was not performed given that by definition, those who match to a program had at least one interview.

►Figs. 1 and 2 illustrate the flow of applicants through the application, interview, and matching process after stratifying by IMG status and the number of attempts. The overall matching rate was higher among non-IMGs compared with IMGs (78% vs 20%, p < 0.0001) and among those on a first attempt compared with a repeat attempt (76% vs 29%, p < 0.0001).

Fig. 1.

Flow of applicants (stratified by international medical graduate [IMG] status) through the application, interview, and matching process. Note: percentage was calculated based on the number of applicants in the previous level.

Fig. 2.

Flow of applicants (stratified by first/repeat attempts) through the application, interview, and matching process. Note: percentage was calculated based on the number of applicants in the previous level.

The association between the number of applications submitted and the number of interviews offered at programs ranked is summarized in ►Fig. 3 after stratifying by IMG status. The red line corresponds to the best-fit simple linear association. The blue line allows for a nonlinear association and indicates the change point for the slope of the best-fit line. Based on the estimated segmented regression line, the number of invitations for interviews will generally increase among IMGs. In contrast, for non-IMGs, the change point of the association between number of applications and number of interviews is 39. This indicates that the association was positive for individuals who submitted up to 39 applications, after which point, the association became negative (p < 0.0001). It is important to note that among the 1,712 non-IMGs, 1,484 (87%) submitted more than 39 applications. When all applicants are considered, regardless of IMG status, the change point of the association between number of applications and number of interviews is 48, indicating that when the number of applications is less than 48, the association is positive; when the number of applications is greater than 48, the association is negative. Among the 1,959 applicants, 1,398 (71%) submitted more than 48 applications.

Fig. 3.

Plot number of interviews (y-axis) versus number of applications (x-axis) for (A) international medical graduates (IMGs) (n = 247) and (B) non-IMGs (n = 1,712).

►Fig 4 demonstrates that the probability of matching reaches the peak of 81% with the group who submitted 41 to 60 applications and decreases thereafter. Additionally, 538 (39%) applicants matched with their top-ranked program, while 268 (19%) applicants matched at their second-ranked program. A majority (87%) of the applicants who matched did so within their top five choices.

Fig. 4.

Summary of matching characteristics. (A) Probability of matching by categories of number of applications (n = 1,959) and (B) cumulative percentage of applicants’ rank of the matching program for applicants who matched (n = 1,391).

After univariate data analysis, a multivariable, recursive partitioning algorithm was created to identify three different groups based on their predicted probability of matching, resulting in those with a low probability of matching (probability < 0.4), a moderate probability of matching (probability between 0.4 and 0.8), and a high probability of matching (probability > 0.8). The analysis used USMLE score and number of program applications as the two predictors because these were nonmissing for a large percentage of applications, with 1,913 applicants who had nonmissing observations for these variables. The analysis was first stratified by IMG status and then by first-attempt status as a separate stratification factor. Models were estimated separately within each resulting subgroup. ►Figs. 5 and 6 present the prediction models based on stratification factors. In summary, among the non-IMG applicants, those with a “high” probability of matching were those with USMLE ≥ 244 or those with a USMLE between 231 and 243 who submitted at least 30 applications. Similarly, among those at their first attempt, individuals with a USMLE ≥ 233 were predicted to have a “high” probability of matching. None of the IMG applicants and none of those at a second attempt were categorically predicted to have a “high” probability of matching. Each model explained no more than 18% of the variability in the probability of matching (fivefold cross-validation R² ≤ 0.18).

Fig. 5.

Classification tree summary of factors predictive of matching for (A) non-IMG (international medical graduate) applicants (n = 1,671). The resulting decision tree explains 18% of the variability in the probability of matching (5-fold cross-validation R² = 0.18). (B) IMG applicants (n = 242). The resulting decision tree explains 7% of the variability in probability of matching (5-fold cross-validation R² = 0.07). Green, high probability of matching (>0.8); NPA, number of program applications; P, predicted probability of matching; Red, low probability of matching (<0.4); USMLE, United States Medical Licensing Examination; Yellow, moderate probability of matching (0.4–0.8).

Fig. 6.

Classification tree summary of factors predictive of matching for (A) first attempt applicants (n = 1,690). The resulting decision tree explains 14% of the variability in probability of matching (5-fold cross-validation R²= 0.14). (B) second or higher attempt applicants (n = 233). The resulting decision tree explains 14% of the variability in probability of matching (5-fold cross-validation R²= 0.14). Green, high probability of matching (>0.8); NPA, number of program applications; P, predicted probability of matching; Red, low probability of matching (<0.4); USMLE, United States Medical Licensing Examination; Yellow, moderate probability of matching (0.4–0.8).

Discussion

Roughly 70% of candidates match in ophthalmology, and 70% match at one of their top three choices. Yet a majority (53%) apply to more than 60 programs and over one-quarter (27%) apply to more than 80. Clearly, more data are needed for program directors and faculty advisors to assist them in the application and matching process.

The recursive partitioning model that we have described is extremely useful in this regard. Since the match rates are so disparate for IMG versus US graduates and first- versus nonfirst attempts, these are logical stratification points. The group with the highest match rate (93%) is US graduates, regardless of first or repeated attempt, with a USMLE step 1 score of 244 or greater. The second highest match rate (87%) was among all first-time candidates with a USMLE score of 233 or greater. For these two groups, the number of programs applied to or ranked was not a factor in matching success. The third highest match rate (83%) occurred among US graduates with USMLE scores from 231 to 243 who applied to 30 or more programs. Although our data do not allow specific recommendations on exact numbers, it would seem unlikely that many candidates in these groups should be advised to apply to more than 40 programs, especially as among all applicants overall, applying to more than 48 programs is not associated with an increasing number of interviews; this number decreases to 39 when confined to US graduates alone. These suggested thresholds are well below the average number of applications submitted by all applicants (mean 64, SD 27), US graduates (mean 65, SD 25), and IMG applicants (mean 56, SD 36).

The fourth highest success rate (66%) was among US graduates with USMLE scores of 217 to 230 who applied to 43 or more programs, and the next highest (61%) among first-time applicants with USMLE < 233 who applied to 47 or more programs. However, those with scores 229 to 232 who applied to fewer than 47 programs had a similar match rate (60%). These groups seem the most likely to benefit from applying to more than 40 programs. For candidates who have previously not matched, the single most important factor in matching was a USMLE score > 235 (55%), independent of number of programs applied to.

For candidates not in the groups described above, there were no subgroups with match rates > 50%. US graduates and first-time applicants with USMLE scores < 231 should be advised to apply to more than 45 programs. Non-US graduates, regardless of USMLE score, and previously unmatched candidates with USMLE score 220–235 did not achieve a moderate probability of matching until they had applied to more than 100 programs (43 and 42%, respectively).

While we did find that AOA (Alpha Omega Alpha Honor Medical Society) membership conferred an increased probability of matching (55% vs 12%, p < 0.0001), this finding must be interpreted cautiously. AOA membership status was missing for 992 (51%) of applicants and, therefore, could not accurately be evaluated as an independent predictive factor of matching, relative to USMLE score or number of submitted applications in our analysis. Some US medical schools, and many non-US schools, may not have an AOA chapter, and some offer both junior and senior admission, while others offer only senior admission. Additionally, differences among schools, timing of application submissions, and the fact that individual chapters have nonstandard admission criteria and protocols are significant confounders which limit interpretation of results.

Similarly, research activity was common in both matched and unmatched groups. However, the analysis simply used the presence of absence of any research activity; we did not distinguish many relative criteria regarding research, for example, whether the research activity was print publication(s), the impact factor of any journals, presentation at local versus regional versus national meetings, or whether the activity was merely participation of some sort in an uncompleted project. Finally, in recent years the Gold Humanism Honor Society has grown across the medical community and is increasingly associated with many of the character traits felt to be important for competent physicians. It is certainly possible that any of these factors could be significantly correlated with probability of matching, and revision of the application form to clarify and standardize reporting of these factors would offer further assistance in advising candidates. In a similar vein, a standardized letter of recommendation format has been recommended for ophthalmology candidates, although to date it has not been widely adopted.

A final set of limitations to this study surrounds the fact that data on interview invitations and completions is self-reported by both programs and applicants. In the match system, programs must mark an applicant as invited for an interview before they can view candidate photos or add them to a rank list. Similarly, applicants are asked to mark an interview as having been completed, but in some cases may make or not make this designation erroneously. As an example of the consequences of this methodology, in the 2015 application cycle there were 726 candidates who submitted 634 rank lists. Of these, 624 applicants were ranked. Programs reported offering a total of 6,594 interview invitations with completion of 5,503 of interviews. However, in the same cycle, applicants reported 6,655 interview invitations with 5,749 interviews completed. The newly formed SF Match Oversight Committee of the AUPO intends to review this design system and consider alternatives to minimize such discrepancies.

Although this analysis yields the ability to provide overall advice to various categories of candidates, it does not permit more detailed candidate-specific data that includes individual portfolios and “good fits” for either candidates or programs in creating their rank lists. The focus of this study may make many candidates and advisors more comfortable with a smaller number of programs applied to, but does not assist with determining which candidates should apply to which programs. In this regard, programs should consider providing candidates with standardized data regarding their residents (e.g., program mean USMLE scores, grade point average (GPA), AOA status, research participation in medical school) to help them determine their best application strategy.

In summary, these data indicate that many candidates for ophthalmology residency (first-time applicants/US grads with USMLE scores >243) need not apply to 60 or 80 programs to successfully match; for most of them, 40 to 45 applications should suffice. The 45 to 60 range may be indicated for initial applicant US graduates with USMLE scores in the 217 to 243 range. A relatively small number of candidates, especially international graduates and previously unmatched candidates (12% each of the entire applicant pool), should consider applying to more than 80 programs, particularly if their USMLE score is <236.

Finally, we recognize that these recommendations must always be individualized, as there are many additional factors that cannot be easily standardized or quantified which nevertheless play important roles in the likelihood of candidates matching. These include variable and qualitative grading rubrics, communication skills, strength of letters of recommendation, extracurricular and employment experiences, and interview performance. In the end, applicants for ophthalmology residency are ranked, and thus matched, by a holistic consideration of their entire candidate portfolio.