Applying Smarter: A Critique of the AAMC Apply Smart Tools

Medical students today submit more residency applications than ever before. This trend is costly for students and imposes a substantial burden on program directors. Yet, despite a steady increase in the number of applications submitted per applicant, overall match rates have not improved.¹ Put another way, applicants could collectively apply to fewer programs than they do now—and enjoy essentially the same overall match rate.

To assist students in determining the optimal number of residency programs to which they should apply, the Association of American Medical Colleges (AAMC) unveiled Apply Smart in 2016. The website notes that “there is a point where the relationship between the number of applications submitted and the likelihood of entry into a residency changes,” and suggests that students should consider limiting their applications at this point of diminishing returns.² Responses to Apply Smart have been positive, with deans and program directors praising the tools and encouraging their use in counseling medical students.^3–6

At first glance, the Apply Smart analyses seem highly informative. Yet, closer inspection reveals methodologic issues that introduce bias and suggest the need for improvements.

Overview of the Apply Smart Tools

The Apply Smart analyses include data and graphics for 19 medical specialties, broken down by applicant type (US MD, US DO, and both US and non-US citizen international medical graduates). For simplicity and consistency, this article will only consider the analyses of MD graduates of US medical schools.

On each graphic, the x-axis represents the number of residency programs in that specialty to which a group of applicants applied. The y-axis indicates the probability of entering a program in that specialty. Three curves appear on each graphic. These were fitted with spline regression and correspond to the probability of entering that specialty for applicants within the top, middle, or bottom tertile of United States Medical Licensing Examination Step 1 scores.

The curves for each specialty have a similar form. Students who apply to few programs have a low probability of entering a residency program in that specialty. Applicants who apply to more programs have a greater probability of matching, up to a “point of diminishing returns” (Table 1). The point of diminishing returns is set as the first knot in the spline regression model and reflects the point at which the likelihood of entering a program in that specialty does not significantly increase when compared to applicants who submitted fewer applications.

Table 1.

Mean Applications Submitted, “Point of Diminishing Returns,” and Ranks Required for > 90% Match Rate by Specialty

Specialty	Mean Applications Submitted (2018–2019)9	Point of Diminishing Returns by USMLE Step 1 Tertile2			Contiguous Ranks for > 90% Match Rate8
		Bottom	Middle	Top
Anesthesiology	34.0	29	21	18	5
Dermatology	68.8	43	N/A	34	9
Diagnostic radiology	43.5	37	23	20	7
Emergency medicine	51.3	32	23	23	9
Family medicine	33.4	19	17	14	5
General surgery	41.1	49	37	36	11
Internal medicine	30.8	30	20	15	4
Internal medicine–pediatrics	24.3	27	16	15	6
Neurology	25.6	18	14	10	5
Obstetrics and gynecology	51.5	28	16	15	10
Orthopedic surgery	80.7	58	39	37	12
Otolaryngology	43.5	45	40	33	9
Pathology	21.6	19	17	14	4
Pediatrics	29.2	22	15	13	4
Plastic surgery–integrated	38.8	N/A	29	27	12
Psychiatry	38.8	21	15	14	8
Radiation oncology	39.8	N/A	32	22	8
Urology	72.1	34	N/A	21	N/A
Vascular surgery–integrated	15.3	23	20	18	20+

Abbreviations: USMLE, United States Medical Licensing Examination; N/A, not available.

Problems With Apply Smart

Close inspection of the Apply Smart graphics reveals several surprising features. For instance, the probability of successfully entering a residency program for applicants who apply to fewer than 5 to 10 programs is strikingly low—sometimes just 20% to 40%. Given the high cost of not securing a residency position, it is curious that so many candidates would “underapply,” especially since the Electronic Residency Application Service (ERAS) charges a flat rate for the first 10 applications within a specialty.⁷ What type of student would apply to less than 10 programs and accept such a low probability of success? The likely answer is one who is not especially interested in matching to that specialty in the first place.

Importantly, although the y-axes of the Apply Smart graphics are labeled “probability of entering a residency program,” the analyses are specialty-specific. That is, applicants are considered to have entered a residency program only if they enter the specialty shown in the graphic. Applicants who enter another specialty are considered as not having entered a residency program. Yet many medical students apply to multiple specialties. For instance, in 2018, 83% of applicants whose preferred specialty was dermatology applied to at least one other specialty, even though the vast majority of these applicants (82%) ultimately matched to dermatology.⁸

Similarly, the Apply Smart graphics show a maximal likelihood of entering a residency program that is curiously low, even for those submitting the ideal number of applications. For instance, the graphic for anesthesiology shows an asymptote around 75%—even though 96% of US seniors who preferred anesthesiology successfully matched in 2018.⁸ Notably, the National Resident Matching Program (NRMP) Match data are limited to an applicant's “preferred specialty,” as determined by their first-ranked program. Therefore, for every specialty, the Apply Smart probability of entering a residency program is lower than the corresponding match rate (Table 2).

Table 2.

Apply Smart Probability of Entering a Residency Program Versus Match Rate by Specialty

Specialty	% of Applicants Entering a Residency Program by USMLE Tertile2			Match Rate (2018), %8
	Bottom	Middle	Top
Anesthesiology	53	68	65	96
Dermatology	39	60	65	82
Diagnostic radiology	42	67	70	89
Emergency medicine	69	86	88	91
Family medicine	60	72	77	95
General surgery	50	64	59	84
Internal medicine	67	81	83	98
Internal medicine–pediatrics	47	78	82	94
Neurology	53	66	67	96
Obstetrics and gynecology	72	86	91	88
Orthopedic surgery	54	77	86	82
Otolaryngology	43	73	81	96
Pathology	44	64	70	96
Pediatrics	76	85	85	99
Plastic surgery–integrated	19	54	66	86
Psychiatry	66	77	81	84
Radiation oncology	43	60	62	93
Urology	49	60	65	N/A
Vascular surgery–integrated	21	46	50	91

Abbreviations: USMLE, United States Medical Licensing Examination; N/A, not available.

What Is the Likely Effect of Apply Smart?

For most specialties, the Apply Smart “point of diminishing returns” is lower than the mean number of applications submitted by applicants applying in that field (Table 1). But will Apply Smart actually encourage fewer applications?

Predicting the overall effect of Apply Smart requires some knowledge of the overall distribution of applications submitted by medical students. While the AAMC does not publicly report this information, several studies show a right-tailed distribution, with a relatively small number of candidates applying to the greatest number of programs. For instance, approximately 25% of internal medicine residents who graduated from US medical schools submitted fewer than 15 applications, while only approximately 7% submitted more than 75 applications.¹⁰ Similarly, in general surgery, the median number of applications submitted by US medical graduates was 26 versus a mean of 41 and a point of diminishing returns ranging from 36 to 49.^6,9,11

Unfortunately, by failing to exclude applicants who are applying to backup specialties, the Apply Smart analyses present biased estimates of the probability of residency entry. How these probabilities are likely to be interpreted by students should be carefully considered. For instance, it is debatable whether Apply Smart data will successfully convince medical students applying to the greatest number of programs that they should instead apply only to the point of diminishing returns—especially when doing so appears to confer only a 60% to 70% probability of entering a residency program in that specialty. On the other hand, for students applying to fewer programs, the interpretation of the Apply Smart analyses is unambiguous: they should apply to more programs, up to the point of diminishing returns, to avoid the seemingly low probability of success associated with submitting fewer applications. Yet, given that the number of applications submitted has a right-tailed distribution, more candidates may find themselves in the latter category than the former. Thus, in its current form, Apply Smart may actually cause an increase in overall applications.

How to Help Medical Students Apply Smarter

How can the AAMC provide higher quality information to inform medical students' decision-making? There are 2 broad possibilities.

The first is to partner with the NRMP and limit the Apply Smart analyses to a student's preferred specialty. Forging such a partnership may be administratively challenging, but would permit analyses that are unbiased by candidates applying to backup specialties. Yet such a solution still fails to address a fundamental problem with Apply Smart: the probability of entering a residency program is not primarily determined by the number of programs to which an applicant applies. The nature of those programs and the competitiveness of the applicant matter more.

Therefore, a second strategy would be to leverage ERAS data to provide applicants with more informative statistics. The AAMC could do this by focusing not on the number of applications submitted, but on the probability that a given application will result in an interview offer.

Unlike the relationship between the number of applications submitted and residency entry, the number of ranked programs needed to successfully match is quite predictable. Through its Charting Outcomes in the Match reports, the NRMP reports the probability of matching by the number of contiguous ranks submitted (ie, the number of programs an individual ranked within one specialty before ranking a program in another specialty). For almost every specialty, ranking 5 to 10 programs is associated with a better than 90% chance of Match success (Table 1). Because applicants may not rank all programs at which they interview, the number of ranked programs does not always equal the number of interviews completed. Yet the former is a reasonable surrogate for the latter: in 2019, the median number of interviews applicants attended was the same as the median number of programs ranked for almost all specialties.¹² Thus, instead of encouraging applicants to apply haphazardly to a particular number of programs in the hope of generating enough interviews to successfully match, a better strategy may be to help applicants specifically select a group of programs where their application is likely to result in an interview.

With the support of residency programs, the AAMC could use multivariable logistic regression to estimate the probability that a given applicant would be offered an interview at a particular program, based on interview decision data from recent application cycles and readily accessible ERAS data (standardized test scores, citizenship status, geographic proximity, honor society membership, etc). This information could be incorporated into the existing Residency Explorer tool and would provide more individualized and interpretable information to students than the general statistics that this website currently contains. If students know that they have a better than 90% (or less than 10%) chance of being offered an interview at a certain program, they may adjust their application strategy accordingly.

Overapplication is costly, for applicants and programs alike. Given the incentives for medical students to overapply, it is unclear whether informational strategies alone can curtail overapplication. Until graduate medical education leaders are willing to support application caps or a fundamental restructuring of the Match to better allow signaling between applicants and programs,¹³ it is imperative that informational strategies present unbiased data that can aid students in applying to an appropriate number of programs. We cannot be satisfied for students to Apply Smart—we need to help them apply smarter.