Holistic Review in Applicant Selection: A Scoping Review

Abstract

Purpose

To avoid overreliance on metrics and better identify candidates who add value to the learning environment, some medical schools and residency programs have begun using holistic review for screening and selection, but limited data support or refute this use. This scoping review examines holistic review definitions and practice in medical education, summarizes research findings, and identifies gaps for future research.

Method

The authors searched 7 databases using a comprehensive search strategy including the keywords holistic, attributes, mission-based, mission-centric, and socially accountable for articles on holistic review within undergraduate medical education (UME) and graduate medical education (GME) published from database inception through July 5, 2024. Author pairs independently screened articles for inclusion and extracted data. Discrepancies were resolved via discussion. Quantitative and qualitative synthesis was performed.

Results

6,511 articles were identified, with 33 included in this review. Twenty-five studies (76%) focused exclusively on GME, with only a few assessing holistic review in UME. Holistic review was implemented at 3 main stages: screening, interviewing, and ranking. Common rationales included service patterns, patient-physician identity concordance, enhancing patient trust, professional advocacy, and educational benefits. Holistic review elements varied, with most falling within the Association of American Medical Colleges experiences, attributes, and metrics framework. Nearly all studies reported an increase in the percentage of underrepresented in medicine trainees interviewed or selected. Several studies also demonstrated increases in other groups (e.g., women, lower socioeconomic status). Many studies included additional interventions to promote diversity, limiting the ability to assess holistic review in isolation.

Conclusions

This scoping review summarizes the literature on the rationale, development and implementation process, structure and components, outcomes assessed, barriers, and strategies for success for holistic review. This work can inform institutions and departments seeking to develop or refine their own holistic review systems and serve as a nidus for future research.

Holistic review is a comprehensive selection process that considers applicants’ experiences, attributes, and academic metrics as well as the value applicants are likely to contribute to learning, practice, and teaching.¹ Although many medical school and residency admissions processes have historically placed a strong emphasis on test-based metrics,^2–5 these metrics are prone to bias and inconsistently predict residency outcomes.^6–9 Holistic review offers an alternative to avoid overreliance on isolated metrics, instead identifying candidates who are academically qualified and bring added value to the learning environment through the experiences and attributes they possess (e.g., resilience, motivation). In doing so, holistic review seeks to minimize bias in the selection process and select applicants who may better align with an institution’s or program’s goals.^10,11

In 2023, there were 52,577 medical school applicants and 49,258 residency applicants.^12,13 Despite an increase in overall medical school and residency applications, substantial racial and ethnic disparities remain among physicians and physicians in training.¹⁴ One study¹⁵ of the 20 largest specialties observed that no residency program represented Black or Hispanic or Latino populations at rates comparable to the U.S. population. Similar findings have been seen among medical student matriculants.¹⁶ Because exclusively test-based metrics can bias against racially marginalized communities,^6–8 holistic review has been proposed as an approach to improve diversity among physicians and physicians in training.

Prior research has sought to understand holistic review in other fields, such as social work, pharmacy, and primary education.^17–19 However, to date, there have been no systematic or scoping reviews on the holistic review process within undergraduate medical education (UME) or graduate medical education (GME). Despite the increasing descriptions of holistic review in the literature and its application in practice, there is narrow understanding of the benefits, limitations, and data to support or refute its use in medical school admissions and residency or fellowship selection. This scoping review aims to examine holistic review definitions and practice in medical education, to summarize research findings, and to identify gaps for future research.

Method

We followed best practice guidelines for scoping reviews and adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews guideline.^20–24 Our study was preregistered at Open Science Framework (https://osf.io/xfu9d/).

Identifying the research question

The overarching purpose of this review was to map the current literature on holistic review in applicant screening and selection within UME and GME to inform current practices and identify gaps requiring future research. To address these goals, we sought to answer the following questions: (1) How was holistic review defined in studies? (2) What were the primary rationales for developing or implementing holistic review? (3) How was holistic review designed and implemented? (4) What were the specific criteria used in holistic review? (5) What outcomes have been evaluated for holistic review? and (6) What barriers to implementing holistic review and strategies for success have been reported?

Identifying relevant studies

In conjunction with an experienced medical librarian (J.W.), we conducted a search of PubMed/MEDLINE, Scopus, ERIC, PsycINFO, MedEdPORTAL, MedEdPublish, and Google Scholar without restrictions on date or language. We used a comprehensive search strategy with both controlled vocabulary (e.g., Medical Subject Heading terms) and keywords, such as holistic, attributes, mission-based, mission-centric, and socially accountable. The search was conducted for peer-reviewed articles published from inception of each database to July 5, 2024. The search was conducted on May 1, 2023, and updated on July 5, 2024. The full search strategy is included in Supplemental Digital Appendix 1 (at http://links.lww.com/ACADMED/B631). In addition to database searching, we manually reviewed the bibliographies of identified studies for potential missed articles. We also consulted topic experts to help identify any further relevant studies. Studies were imported into Covidence (Veritas Health Innovation Ltd, Melbourne, Australia) for screening and selection.

Study selection

We included studies of holistic review used at any point in the screening and selection process in UME and GME. Studies must have applied the criteria prospectively or retrospectively to a population to be included. We excluded studies that described but did not implement or apply holistic review to a population, those conducted in other health professions (i.e., not medical students, residents, or fellows), guideline statements, review articles, and those conducted outside the United States. We focused exclusively on studies conducted in the United States, given the unique processes of applicant selection and the context-dependent nature of holistic review.²⁵ Two investigators (M.G., D.D.) independently assessed studies for eligibility based on the above criteria according to their titles and abstracts. These investigators met regularly and discussed inclusion and exclusion criteria to clarify any issues or ambiguities as they arose. All records meeting the initial inclusion criteria were downloaded and reviewed as full-text articles. In the event of disagreement regarding inclusion at the abstract screening stage, the article was selected for full-text review. During the full-text stage, 2 investigators (M.G., D.D.) independently assessed articles for inclusion in the final dataset. Any discrepancies were resolved by consensus. If consensus was not achieved between both reviewers, the article was presented to the full authorship group for discussion with the final decision made by group consensus.

Recording the data

We used the descriptive-analytic model described by Arksey and O’Malley to guide data extraction and summarization.²⁰ Before beginning data extraction, the full group developed, piloted, and refined a data collection instrument. Holistic review criteria were reported as individual items and grouped according to the Experiences-Attributes-Metrics (EAM) holistic review framework proposed by the Association of American Medical Colleges (AAMC).²⁶ Four investigators (M.G., A.L., J.B., M.D.) independently extracted data from the included studies in duplicate. These investigators met regularly to discuss extraction categories and clarify any issues or ambiguities as they arose. Any discrepancies were resolved by consensus.

Collating, summarizing, and reporting the results

We synthesized and collated the data, performing both quantitative and qualitative analyses. For the quantitative portion, we provided a descriptive summary of the extent, nature, and distribution of the studies included in this review. For the qualitative synthesis, we conducted a narrative review of information addressing our study questions, identifying the current state of knowledge with an emphasis on the broader application of the findings and directions for future research as recommended by Levac et al.²¹

Consultation

We sought external consultation from a diverse group of 10 medical educators (including program directors, medical school deans, and diversity, equity, and inclusion experts) at various stages in their career to ensure that we had identified the key literature and interpreted it appropriately. Examples of changes enacted based on this expert consultation included clarifying the intended audience, better framing the role of holistic review in the context of mission alignment, and expanding the discussion of balance and feasibility.

Results

Overview of studies included in the review

Our search identified 6,511 articles, of which 1,483 were duplicates, resulting in 5,028 total articles for review (Figure 1). After title and abstract screening, we excluded 4,931 records, leaving 97 articles for full-text review. Sixty-four were excluded, and 33 studies were included in the final analysis (see Supplemental Digital Appendix 2 at http://links.lww.com/ACADMED/B632).

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram for studies included in this review of articles on holistic review within undergraduate medical education and graduate medical education published from database inception to July 5, 2024.²⁴

The earliest study explicitly describing the application of holistic review was published in 2012.²⁷ Thirty studies (91%) were published after 2019 (see Supplemental Digital Appendix 3 at http://links.lww.com/ACADMED/B631). The 3 studies^27–29 published before 2019 were all in UME, highlighting its earlier adoption in UME admissions. Six studies^28–33 (18%) focused exclusively on UME admissions to medical school, 25 studies^10,34–57 (76%) described GME selection into residency, 1 study²⁷ (3%) discussed both UME and GME admissions and selection, and 1 study⁵⁸ (3%) described holistic review for a combined baccalaureate and medical degree program. Two studies^51,52 (6%) described the use of holistic review for selection into fellowship programs. Among the GME studies, 3 studies^34–36 (9%) were in urology, 3 studies^38,39,56 (9%) were in general surgery, 2 studies^10,37 (6%) were in emergency medicine, and 2 studies^40,54 (6%) were in internal medicine. The remaining specialties (family medicine, medicine-pediatrics, neurology, neurosurgery, orthopedic surgery, otolaryngology, pediatrics, pediatric emergency medicine, physical medicine and rehabilitation, plastic and reconstructive surgery, psychiatry, pulmonary critical care medicine, and radiology) were represented once (3% each).^{41–50,53,55,57} Two GME studies^42,49 (6%) included multiple specialties (see Supplemental Digital Appendix 4 at http://links.lww.com/ACADMED/B631).

The geographic location of institutions publishing on holistic review was skewed, with only 18 states and the District of Columbia represented (see Supplemental Digital Appendix 5 at http://links.lww.com/ACADMED/B631). Five studies^{36,38,42,45,55} (15%) were published from Pennsylvania, 4 studies^30,40,43,49 (12%) from Texas, 4 studies^37,41,46,51 (12%) from California, 2 studies^10,56 (6%) from Colorado, 2 studies^31,32 (6%) from Maryland, 2 studies^28,58 (6%) from New Mexico, 2 studies^35,52 (6%) from New York, and 1 study from the remainder of states and the District of Columbia (3% each). Of the states represented, 5 are listed in the top 10 most diverse states in the nation by US News and World Reports (California, Maryland, New Mexico, New York, Texas),⁵⁹ with 4 of these states (California, Maryland, New Mexico, and Texas) being majority-minority states (defined as states whose population is composed of < 50% non-Hispanic White people).⁶⁰

All studies were single-institution studies. Fourteen studies^{10,35–40,42,45–47,50,55,56} (42%) used an observational pre-post design, wherein outcomes before and after holistic review implementation were compared. Eleven studies^{27–31,33,41,48,49,57,58} (33%) used retrospective cohorts, whereas 7 studies^{32,34,43,51–54} (21%) used prospective cohorts. One study⁴⁴ (3%) used the Delphi method.

Holistic review was implemented at 3 main stages in the admissions or selection process: during screening (to decide whether to offer an interview), during interviewing (to develop holistic interview questions or to score the interviews using a holistic tool), and during final selection and ranking (to decide whom to admit, wait-list, or reject or whom to rank highly in the Match). Twenty-seven studies (82%) used holistic review for screening applicants: 16 studies^{10,29–31,34,35,39–41,45,47,49,50,52,55,56} (48%) used holistic review exclusively for screening, 2 studies^36,57 (6%) used it for screening and interviewing, 3 studies^37,43,48 (9%) used it for screening and final selection and ranking, and 6 studies^{32,33,38,46,53,54} (18%) used it for screening, interviewing, and final selection and ranking. Only 1 study²⁸ (3%) used holistic review for final selection and ranking alone, and 2 studies^44,51 (6%) used it for interviewing and final selection and ranking. In 3 studies^27,42,58 (9%), the stage at which holistic review was performed was not described.

The precise number of applicants evaluated using holistic review was reported in 20 studies (61%) and ranged from 55 to 6,901. The number of raters for the holistic review process was not explicitly described in 21 studies (64%). In the remaining studies, a single rater was used in 5 studies^{31,39,43,47,56} (15%), with an additional rater added in 1 of these studies³¹ only when the initial review was unfavorable; 2 raters were used in 6 studies^{28,29,33,50,55,57} (18%); and automated Electronic Residency Application Service (ERAS) filters for experiences and attributes were used in 1 study⁴⁹ (3%). The type of rater was not described in 13 studies^{10,27,28,30,31,34,35,42,44,45,49,54,58} (39%). In 17 studies^{29,32,36,38–41,43,46–48,51–53,55–57} (52%), raters were faculty, either alone (n = 5 [15%]) or in combination with students, residents, and staff (n = 10 [30%]). In 3 studies,^33,37,50 ratings were performed by committee; however, in 1 of these,⁵⁰ the initial screen was conducted exclusively by staff.

Definition of holistic review

Nineteen studies (58%) used the AAMC definition of holistic review: “a flexible, individualized way of assessing an applicant’s capabilities by which balanced consideration is given to experiences, attributes, and academic metrics.”¹ Nine studies^{27,36,47,48,50,52,54,55,57} (27%) did not define holistic review. The remaining 5 studies^{32,39,49,53,56} (15%) described a definition similar to that used by the AAMC but referenced other sources.

Rationale for diversification and holistic review

The rationale for diversification of trainees varied, although largely fell into the 5 categories previously described by Saha⁶¹: (1) service patterns (i.e., underrepresented trainees are more likely to practice in underserved communities or areas, enhancing health equity and mitigating health disparities; n = 16 [48%])^{10,27–29,32,33,36,38–42,45,48,56,58}; (2) concordance (i.e., patients are likely to experience improved communication and health outcomes when their physician shares their identity; n = 14 [42%])^{10,27–29,33,36–39,45,48,55–57}; (3) trust (i.e., underserved or disadvantaged communities are more likely to trust the medical establishment if it reflects the population it serves; n = 5 [15%])^{27,29,38,42,57}; (4) professional advocacy (i.e., underrepresented physicians are more likely to advocate for and change policies to benefit disadvantaged communities; n = 3 [9%])^31,36,39; and (5) educational benefits (i.e., a diverse group of learners and teachers leads to better education for all physicians; n = 9 [27%]).^{10,27,29,30,38,48,51,55,57} Nine studies^{34,35,43,44,49,50,52–54} (27%) did not describe a rationale for diversification. One study⁴⁷ discussed the role for compliance and accreditation, whereas another⁴⁶ reported a desire to reduce oppression in selection. Rationales for using holistic review largely mirrored the above rationales for diversification. Sixteen studies^{10,27,28,30–33,36,38,45,48,51,53–55,58} highlighted mission alignment or a responsibility to produce a workforce reflective of the population they serve, whereas 11 studies^{29,38,41,44,46,52–57} aimed to address bias in admissions and selection.

Process for developing and implementing holistic review

In several studies, holistic review was just one component of broader strategies to increase the diversity of their applicants and trainees. Thus, many studies lacked details concerning the development and implementation of holistic review.

Thirteen articles^{27–32,34,36,39,42,45,47,58} (39%) failed to provide a description of their process for developing holistic review. Of the 20 articles^{10,33,35,37,38,40,41,43,44,46,48–57} (61%) that provided some description (all but 1 within GME), only 7 articles^{10,40,41,49,52,53,58} (21%) described deliberate mission alignment. Among those describing mission alignment, key elements in their missions included addressing health care disparities and promoting health justice^10,40,41,49; representing and serving the surrounding community^53,58; a commitment to diversity, equity, and inclusion¹⁰; altriusm¹⁰; and patient-centered care.¹⁰ The stakeholders involved in development of the holistic review varied across studies and included combinations of program leadership, faculty, residents, and diversity leaders. Processes ranged from simply enumerating important experiences, attributes, and metrics to prioritizing and assigning weights to certain characteristics in scoring rubrics. One study⁴⁴ used a rigorous Delphi process to develop their rubric. Five studies^{37,41,46,54,57} (15%) described actively devaluing metrics with known bias and limited predictive value, such as United States Medical Licensing Examination (USMLE) scores and Alpha Omega Alpha induction. Another study¹⁰ described attempts to strike a balance across the 3 domains (experiences, attributes, and metrics).

Thirty studies^{10,27–29,31–35,37–44,46–58} (91%) described at least some aspects of how they implemented holistic review. In 23 studies^{27,29,30,32,34–38,41,43,45–53,55,57,58} (70%), holistic review was applied to all applicants. However, in other studies, the labor-intensive nature of holistic review led to various strategies to winnow the pool of applicants subjected to holistic review. In 8 studies^{28,31,33,40,44,47,54,56} (24%), applicants were still required to meet score cut points for grade point average, Medical College Admission Test (MCAT), or USMLE or first achieve a specific score based on academic metrics or other criteria before files were holistically reviewed. In these studies, applicants were screened out before holistic review was even applied. Other studies specifically discussed holistic review in the context of applicants who were underrepresented in medicine (URiM). In 1 study,¹⁰ applicants who were URiM and below these screening thresholds were deliberately pulled into the pool for holistic review. In another study,³⁹ holistic review was only used for applicants who were URiM. Seven studies^{29,34,35,46–48,57} (21%) described blinding reviewers to photographs, race, or ethnicity, whereas 4 studies^32,34,38,57 (12%) described blinding reviewers to USMLE scores, MCAT scores, and grades.

Most studies described the application of rubrics or the use of score cards during screening, interviewing, or ranking. However, they often provided only partial descriptions of the rubrics and their implementation. The most well-described model appears in the article by Sungar et al.¹⁰ In this article, the authors explicitly outlined the rubric’s development and implementation in alignment with the AAMC EAM model and provided a detailed weighting of the components, allowing replicability by others.²⁶ Interestingly, although multiple programs described mission or value alignment, Sungar et al¹⁰ specifically included a weighted mission score, which assessed how well the applicant directly aligned with their mission.¹⁰ The authors also included a perspective score that “allowed an objective weight to be assigned to applicants who provided a unique perspective amongst [their] residency community based on background, including but not limited to: race and ethnicity, sexual orientation, first generation college graduate, underrepresented group in our program, low socioeconomic status, or disadvantaged background.”¹⁰

Although most studies reported that raters received training, this was often poorly described, and no studies reported on assessing the reviewers’ accuracy or interrater reliability before initiating the review. One study²⁸ assessed interrater reliability for 2 reviewers during the official review process and reported strong (> 90%) interrater reliability for both the metric-based and non–metric-based components. Another study⁵³ assigned each reviewer a single characteristic to focus on and reported reduced time to completion and higher satisfaction among reviewers.

Almost all studies described manual processes for holistic review data entry. One study⁴⁹ uniquely described translating each experience and attribute into 1 or more searchable ERAS filters based on available filter categories to semiautomate their screening process. The authors of this study highlighted the future potential of artificial intelligence to augment current labor-intensive human processes.

Components of holistic review

The level of detail concerning the experiences, attributes, metrics, or other characteristics used for holistic review varied across studies. Table 1 includes examples for each criterion with corresponding citations. Of note, there was some overlap of characteristics listed in experiences and attributes (e.g., leadership roles were experiences, whereas leadership was an attribute).

Table 1.

Holistic Review Elements Identified in the Scoping Review of Articles on Holistic Review Within Undergraduate Medical Education and Graduate Medical Education Published From Database Inception to July 5, 2024

Examples by category	References
Experiences
Artistic experience (e.g., drawing, painting)	27,33,35
Athletic experience	27,35,49
Completion of advanced degrees (e.g., higher education or health science degree)	49
Distance traveled (e.g., quality of the early educational environment, overcoming adversity, first-generation college or medical student)	10,27–30,35–37,41,44,47,52,53,56
Experiences demonstrating commitment to geographic location	33
Experience educating others (e.g., Teach for America, providing health education, coaching)	49,52
Extracurricular or professional activities (e.g., student organizations, professional societies, committee membership)	47,49,50	47,49,50
Health care experience or exposure (e.g., experiences that demonstrate interest in medicine or a specialty)	27–32,43,49,52
Health equity or DEI experience	27,35–37,40,44,49
Interprofessional experience	49
Leadership roles	10,27,31,34,35,37,40,43,44,49,50,52–54,56
Military service	31,47,49
Nonacademic activities and hobbies	34
Prior sustained work experience	10,27,44
Quality improvement or patient safety experience	49
Relevant or influential life experiences	28,43,44,47,48,53,54
Relevant work experiences (e.g., medical scribe, customer service, restaurant server or host)	10,28,33,49
Research experience	30,31,34,35,38,41,47,49,50,52,53,56
Volunteer, community, or service experiences	10,27,28,30,31,33,35–37,41,44,50–56
Attributes
Academic potential	38,52
Adaptability	33,44,46
Altruism or service orientation	29,38
Clinical knowledge or ability	33,52,57
Commitment to DEI (including a focus on health equity or health disparities)	44,46,55
Communication skills	10,33,51
Compassionate, empathetic, humanistic (e.g., Gold Humanism Honor Society selection)	33,44,49,51,52,54
Conflict resolution skills	33
Critical thinking skills	33,53,54
Cultural competence	43
Enthusiasm for the profession	29,38,43,57
Geographic origin (e.g., hometown, in-state, national origin, citizenship, rural or inner-city background)	27,49
Geographic ties (e.g., local or regional ties, rotated at institution)	28,35,48,50,52,55,56,58
Gender identity	10,31,48
Intellectual curiosity	10,38,43,54
Languages spoken	10,35,37,49
Leadership	33,38,41
LGBTQIA+	49
Motivation (e.g., takes initiative)	10,44,46
Professionalism, integrity, ethics, and trustworthiness	10,32,33,41,44,49,54,57
Race, ethnicity, or URiM status	10,28,31,37,48,49,52,55
Reliability and dependability	10
Resilience	10,33,35,52,54,55
Self-awareness (e.g., maturity, responsive to feedback)	33,44
Socioeconomic status	10,31
Team player or teamwork	10,29,33,38,43,44,46
Teaching ability	52
Tolerance for uncertainty	33,54
Work ethic	44,54
Metrics
AOA selection	10,34,35,49,50
Awards	44
Class rank	10,34,47,55
Clerkship rotation grades (e.g., honors, failures)	10,34,35,41,43,44,50,53,55
GPA (e.g., science and nonscience, undergraduate and graduate)	27–31,47,50,58
Letters of recommendation	55
MCAT (e.g., scores, subscores, number of attempts)	27–29,31,58
Medical school	10
MSPE	55
Scholarship or scholastic achievement (e.g., number of publications)	44,48
SLOE	10
USMLE Step 1, USMLE Step 2, or COMLEX (e.g., scores, failures)	10,35,37–40,43,47,50,53,55

Abbreviations: AOA, Alpha Omega Alpha; COMLEX, Comprehensive Osteopathic Medical Licensing Examination of the United States; DEI, diversity, equity, and inclusion; GPA, grade point average; MCAT, Medical College Admission Test; MSPE, Medical Student Performance Evaluation; SLOE, Standardized Letters of Evaluation; URiM, underrepresented in medicine; USMLE, United States Medical Licensing Examination.

List 1.

Potential Strategies to Enhance Diversification in Conjunction With Holistic Review

• Commitment of senior leaders to diversity • Creation of a diversity committee • Diversification of admissions and interviewing committees • Early exposure opportunities • Explicit expression of commitment to diversity during interviews • Funded clerkships and second look weekends • Implicit bias training • Pathway programs • Standardization of interview questions • Recruitment at SNMA and LMSA regional and national meetings and at HBCUs • URiM mentorship programs • Website redesign to communicate commitment to diversity

Abbreviations: HBCUs, Historically Black Colleges and Universities; LMSA, Latino Medical Student Association; SNMA, Student National Medical Association; URiM, underrepresented in medicine.

Summary of outcomes

Programs primarily assessed outcomes as the change in percentage of women and URiM applicants, interviewees, or matches before vs after implementation of holistic review. Nearly all studies reported an increase in URiM recruitment with use of holistic review (see Supplemental Digital Appendix 2 at http://links.lww.com/ACADMED/B632), although most included holistic review as part of larger, multifaceted interventions (e.g., implicit bias training, diversification of admissions committees, commitment of senior leaders to diversity, pathway programs, funded clerkships). One study³⁸ saw the percentage of women ranked increase from 42% to 61% with the implementation of holistic review. However, others^35,50 found no difference in women or a decrease over time. One program reported a substantial increase in the number of URiM students who matched at their residency program from 0% before implementation to 29% after implementation.⁴⁵ Interestingly, a study⁴² of multiple surgical residency programs at a single institution found that only certain groups were affected (i.e., Black and Latinx interviewees), whereas there was no appreciable increase in Native American interviewees. Despite the emphasis on mission alignment, none of the studies specifically assessed whether the applicants ultimately aligned with other aspects of their mission (e.g., subsequent engagement in health equity initiatives, work with underserved communities after training).

Barriers to implementing holistic review and strategies for success

Study authors identified several key barriers and strategies for success. Studies emphasized that it was time consuming to apply holistic review processes to each application received.^43,46,48,54 One study⁴² noted that it took 10 to 15 minutes to review an application via holistic review compared with 2 to 3 minutes via traditional review. Combining this with the volume of applications led one group to suggest that some academic metrics may still be needed to prioritize applications.³⁹ Others highlighted that many of the desired experiences and attributes were difficult to find using ERAS filters, limiting the ability to conduct holistic review in a more efficient or automated fashion.^49,54 However, one program addressed this by having dedicated administrative assistants review applications using an objective tool and create reports to reduce the burden on program leadership.⁵⁰

Some studies^10,36,37,46 also highlighted that faculty members and residents must buy in to the process because this is a human resource–intensive commitment and culture change can take longer than simply changing policy. Another study¹⁰ suggested that programs may still be disproportionately relying on metrics even if they are instituting holistic review, emphasizing that faculty and resident training are also likely needed. Two studies^43,50 (6%) highlighted the need for admissions champions who value diversity to help holistic review to scale and be sustainable. Three studies^28,32,47 (9%) cited concern about ensuring that all students will have a strong academic foundation that allows them to complete medical school. However, a separate study⁴² commented on the harms of the narrative that admitting more students who are URiM will lower standards of excellence. Two other studies^31,37 (6%) emphasized a need to shift away from USMLE scores given the impact of factors, such as socioeconomic status or race, that can influence the scores and exacerbate biases.

Some programs expressed concerns about holistic review being an isolated initiative without other accompanying measures to increase diversity. One program emphasized that tools for holistic review should also be paired with training and interventions to address the harmful and biased conditions that many trainees face.⁴² Another program highlighted that holistic review cannot fix a shallow applicant pool, and the use of holistic review without antibias training for the screeners may uphold structural inequities.³² Two programs (6%) highlighted the need to develop a comprehensive strategic plan, including other resources, such as time and administrative support, for more expanded initiatives.^39,48 Others recommended increasing support for applicants, such as housing support for visiting students, to help increase the interview pool.^36,45 List 1 includes a summary of different initiatives used in conjunction with the holistic review.

Discussion

This scoping review found that holistic review has been used for UME and GME, with representation across both medical and surgical specialties, and that interest has been increasing during the past several years. This is not surprising given increasing efforts to combat disparities in medical education. Decades of research has shown that bias exists in traditional academic-based standardized tests, such that students with low socioeconomic status often score lower than students who are White and/or more economically privileged, despite limited ability to predict success in residency.^6–8,62,63 Holistic review offers a process to give balanced consideration of academic metrics with nonacademic factors, recognizing that varied experiences and attributes of each applicant are also important in contributing to the educational environment and enhancing the institutional mission.

Despite all the studies focusing on holistic review, there was substantial variability among the individual models used. Although some were modeled after components of the AAMC EAM guidelines, others had more limited components and emphasized only specific items. It is possible that part of this may be due to fundamental differences between the American Medical College Application Service and ERAS applications, secondary applications, and the information collected between them (e.g., Pell Grant eligibility, parental income, school lunch program participation). Alternatively, given that the AAMC considers a core tenet of holistic review to include “the value an applicant would contribute to learning, practice, and teaching,” this may reflect the differing values and priorities among institutions and specialties.¹ For example, Schulz et al³⁵ used holistic review for a urology residency and included skills associated with manual dexterity (e.g., sports, playing a musical instrument, wood carving), given the procedure-oriented nature of the specialty. Table 1 provides a list of the various metrics that programs or institutions could select from when tailoring a new or refining an existing holistic review program. In doing so, it is critical to identify which components best align with a given institution’s mission and core values.

We also identified variability in the categorizations and definitions of holistic review. For example, a single item could be counted across multiple categories (e.g., leadership as an attribute, leadership role as an experience, leadership title as a metric), leading to duplicative value for certain items. Additionally, some categories were vague or difficult to measure, increasing the potential risk of misinterpretation (e.g., distance traveled, scholastic achievement). Therefore, it is critically important to ensure sufficient training and provide explicit criteria and examples for each item when conducting holistic review to avoid bias or misinterpretation of items.⁶⁴

Interestingly, although holistic review began in the UME realm before its implementation in GME, we noted that most published studies focused on GME. This may be due to the veil of secrecy that often shrouds medical school admission processes to prevent applicants from gaming the system by highlighting those attributes and experiences that a specific school favors. As such, it is possible that holistic review is occurring to a substantially greater degree than published, and future research is needed to better understand holistic review in this setting.^65,66 In contrast, 1 recent study³³ in UME formally published its criteria online and even provide a worksheet to guide applicants.

Although most studies included in this review emphasized improvements in applicant diversity, this was primarily assessed through the distribution of applicants who identified as URiM. A smaller number of studies assessed other categories, such as gender, socioeconomic status, and first-generation college status. Other categories (e.g., applicants with disabilities, English as a second language) remain largely unexplored, highlighting the need for further research to understand the influence of holistic review on other measures of diversity.

Holistic review is time consuming, requiring multiple personnel and a more intense review of applications than traditional models using filters and screening criteria.⁴⁶ This challenge can be further compounded in programs and institutions with longer applicant lists and more robust criteria, which was cited in several studies as a potential limitation. Some programs described using filters or mission-aligned, prescreening criteria to narrow the pool for which holistic review is used. Others have begun to explore the role of artificial intelligence and machine learning to rapidly screen and categorize, thereby increasing the number of applicants who can undergo holistic review and the number of holistic review components that can be included, while freeing up faculty capacity for other areas.^67–71 However, it is important to ensure adequate safeguards (e.g., selective or secondary review by a human) because artificial intelligence and machine learning may miss relevant areas and are subject to training biases.⁷²

Moreover, holistic review should be multifaceted, including developing the rubric, training the file reviewers, and measuring and analyzing the outcomes and impact.⁶⁴ Many programs also combined this with initiatives to enhance diversity and combat bias. Recently, others have suggested that holistic review principles be extended beyond the initial screening and selection process to provide holistic support and resources from matriculation through graduation.⁷³

This review has several limitations. We restricted holistic review to UME and GME. These findings may not reflect holistic review for faculty recruitment or in other health professions sectors. Although we used 7 databases and conducted the search in concert with an experienced medical librarian, we may have missed some relevant literature. Our findings are also limited to schools and programs that elected to publish their findings. Data on the programmatic implementation of holistic review (e.g., the geographic distribution of programs) were limited to programs with published works on the topic and likely underrepresent the true prevalence of this approach. Many of the included studies had limited descriptions of the specific process for developing or implementing the holistic review processes. Future research should include more robust descriptions of these components, as well as the specific training and rater assessment performed. Only 1 study⁵² reported on interrater reliability. Future research should better determine the interrater reliability across metrics, in addition to the ideal number of raters needed to ensure consistent scoring. Multiple studies did not describe the specific rubrics, and few described the weights applied to each item. Future research should provide more details regarding the specific components and weights, as well as how the weighting was selected. Moreover, although most programs focused on mission alignment regarding diversity and community representation, there was a dearth of outcomes on other areas of mission alignment (e.g., health equity initiatives, underserved community work). Future research should assess the effect on other areas of mission alignment. Finally, the use of multifaceted interventions to increase diversity made it difficult to isolate and determine the effects of holistic review processes on measurable outcomes. Future work should assess the specific effect of holistic review on these outcomes.

Conclusions

Holistic review is a model for better recognizing the varied attributes of applicants, including academic metrics and nonacademic factors, for trainee screening and selection. This scoping review summarized the existing literature regarding the rationale, development and implementation process, structure and components, outcomes assessed, barriers, and strategies for success. This work can inform institutions and departments seeking to develop or refine their own holistic review systems and serve as a nidus for future research.