Perceived Barriers to Conducting Holistic Review in Dermatology Resident Selection

Abstract

This survey study examines potential barriers to the integration of holistic review into dermatology resident selection.

Dermatology residency programs are competitive, often relying on grade point average and US Medical Licensing Examination (USMLE) scores, which may overlook other valuable qualities. The Accreditation Council for Graduate Medical Education (ACGME) and the AAMC (Association of American Medical Colleges) propose a holistic review approach that considers applicants’ experiences and attributes alongside academic metrics, promoting diversity.¹ However, research on this approach is limited. We surveyed dermatology residency program directors (PDs) to identify barriers to implementing holistic review.

Methods

Participants were recruited using the Association of Professors in Dermatology listserv. Inclusion criteria included current program directors (PDs) or associate program directors answering on their behalf of ACGME-accredited dermatology residency programs. Data were collected anonymously via a survey (Qualtrics) between August 23 and October 30, 2022, and were analyzed using Excel (Microsoft Corp) (Supplement 1). This survey study was deemed exempt by the Loma Linda University Institutional Review Board, and completion of the survey implied informed consent. We followed the AAPOR reporting guideline.

Results

Of 143 dermatology programs contacted, 36 responded (response rate, 25.2%) and 35 (97.2%) used holistic review. Most respondents (25 of 35 [71.4%]) applied holistic review to all applications, while 11 of 34 (32.4%) applied it to prescreened applications using metrics such as USMLE scores. Many PDs (31 of 36 [86.1%]) spent over 20 hours conducting application review, and 22 of 36 (61.1%) lacked protected administrative time. During the 2021 to 2022 application cycle, 29 of 36 programs (80.6%) received greater than 500 applications, and 5 of 36 (13.9%) received 401 to 500 (Table 1).

Table 1. Program Demographics.

Characteristic	No. (%) (N = 36)
Geographical location
Midwest	12 (33.7)
West	11 (30.6)
Northeast	7 (19.4)
South	6 (16.3)
Time in service as program director, y
<5	18 (50)
5-10	13 (36.1)
11-15	3 (8.3)
16-20	1 (2.8)
>20	1 (2.8)
No. of faculty assigned to review applications
<2	2 (5.6)
3-5	8 (22.2)
6-10	9 (25.0)
11-15	10 (27.8)
>15	7 (19.4)
No. of applications received
0-200	1 (2.8)
201-300	0
301-400	1 (2.8)
401-500	5 (13.9)
>500	29 (80.6)
No. of ACGME-approved residency positions
1-2	1 (2.8)
3-4	20 (55.6)
5-7	9 (25.0)
>8	6 (16.7)
No. of interviews conducted in the 2021-2022 residency cycle
10-20	1 (2.8)
21-40	17 (47.2)
41-60	13 (36.1)
61-80	5 (13.9)
>80	0

Abbreviation: ACGME, Accreditation Council for Graduate Medical Education.

PDs prioritized attributes such as honesty and integrity (14 of 23 [60.9%]), work ethic (19 of 33 [57.6%]), compassion and empathy (10 of 24 [41.7%]), communication skills (9 of 25 [36.0%]), clinical clerkship performance, grade point average, and class rank (8 of 26 [30.8%]). Secondary priorities included USMLE scores (5 of 21 [23.8%]), publications and presentations (6 of 20 [30.0%]), and honors, awards, and scholarships (3 of 19 [15.8%]).

Seventeen of 36 PDs (47.2%) cited increasing diversity as the rationale for implementing holistic review. Other motivations included capturing candidates overlooked by conventional metrics (9 [25.0%]) and fostering opportunities for underrepresented in medicine (URiM) students (5 [13.9%]) (Table 2). Barriers included volume of applications (29 [80.6%]), followed by insufficient protected administrative time (19 [52.8%]), and insufficient guidance for conducting holistic review (10 [27.8%]) (Table 2).

Table 2. Rationales for and Barriers to Implementing Holistic Review.

	Responses, No. (%) (N = 36)
Rationale
To increase the diversity of our residency program, including diversity of background and life experiences	17 (47.2)
To better capture candidates who may have been screened out based on traditional metrics such as USMLE scores and AΩA membership	9 (25.0)
To increase opportunities for underrepresented minorities in dermatology	5 (13.9)
To consider candidates who did not score highly on metrics such as USMLE scores, AΩA membership, or class rank	4 (11.1)
NA, holistic review was not conducted at our program	1 (2.8)
Barrier
Too many applications to review	29 (80.6)
Not enough protected or administrative time allocated	19 (52.8)
Lack of guidance and best practices for how to conduct holistic review	10 (27.8)
Not enough faculty participating in review	8 (22.2)
Othera	3 (8.3)

Abbreviations: NA, not applicable; USMLE, US Medical Licensing Examination.

^a Other barriers cited were that the Electronic Residency Application Service (ERAS) is not a user-friendly program, the supplemental ERAS system had issues, and the quantity of data in each application made it difficult to distinguish between applicants.

Discussion

The AAMC and ACGME advocate for holistic review to promote more inclusive evaluation.¹ However, PDs face high application volume, inadequate protected time, and lack of standardization for holistic review. Potential solutions include increasing protected time and using standardized rubrics. Baylor College of Medicine reduced review time to approximately 5.5 hours using such strategies and offered guides adaptable to other specialties.²

We found inconsistency in the interpretation of holistic review. Although 35 PDs reported using holistic review, one-third still prioritized USMLE scores for initial screening, potentially overemphasizing test scores. Some PDs gave more weight to metrics, such as AΩA membership and publications. These findings highlight a possible need for clearer guidance to ensure consistent interpretation and implementation of holistic review. Critics of holistic review cite concern of favoring diversity over merit.³ However, a study found that holistic review increased representation of women and URiM students in general surgery without differences in USMLE scores.⁴ Another study noted more URiM interviews, with no significant score differences.⁵ Beyond dispelling concerns about declining academic standards, research shows enhanced workforce diversity is crucial in meeting the needs of our diverse society.⁶

Holistic review may improve diversity and has not been shown to substantially impact traditional metrics of academic excellence. Our findings underscore the need for additional investigation and a standardized approach to holistic review. Our study is limited by potential bias, as respondents using holistic review may have been more inclined to participate. Low response rate limits the generalizability of our findings. Further research is warranted to delineate best practices for implementation of holistic review.

Supplement 1.

eMethods

Supplement 2.

Data Sharing Statement