Impact of holistic review on diversity of interviewed and matriculating residents in graduate medical education: a systematic review protocol

Abstract

Introduction

Diversity in the physician workforce improves patient-centred outcomes. Patients are more likely to trust in and comply with care when seeing gender/racially concordant providers. A current emphasis on standardised metrics in academic achievement often serves as a barrier to the recruitment and retention of gender and racial minorities in medicine. Holistic review of residency applicants has been supported as a means of encouraging diversification but is not yet standardised. The current body of evidence examining the effects of holistic review on the recruitment of racial and gender minorities in surgical residencies is small. We therefore propose a systematic review to summarise the state of holistic review in graduate medical education in the USA and its impact on diversification.

Methods and analysis

Our systematic review protocol has been designed with plans to report our review findings in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols guidelines. PubMed and Embase will be searched with the assistance of a health sciences librarian with expertise in systematic review. We will include studies of graduate medical education programmes that describe the implementation of holistic review, outline the components of their holistic review process and compare proportions of under-represented minorities (URM) and women interviewed and matriculating before and after holistic review implementation. We will first report a summary of the findings regarding the operationalisation of holistic review as described by studies included. We will then pool the percentages of URM and women for interviewee and matriculant populations from each study and report the collective odds ratios of each for holistic review compared with traditional review as our primary outcome.

Ethics and dissemination

This study is a protocol for systematic review, and therefore does not involve any human subjects. Findings will be published in the form of a manuscript submitted to a peer-reviewed journal.

PROSPERO registration number

CRD42023401389.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• We will use systematic review to identify core components of holistic review, which is currently ill-defined.

• Meta-analysis will be employed to quantify the impact of holistic review on proportions of under-represented minorities interviewed and matriculating into graduate medical education.

• If reported data regarding outcomes on race/gender proportions are scarce or too heterogenous for meta-analysis, we will publish a summary of the findings regarding the logistical operationalisation of holistic review as described by studies included.

• Our systematic review findings will be reported according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines with assistance from a health sciences librarian with expertise in the conduct of systematic review.

Introduction

Diversity improves patient-centred outcomes and fosters productivity and innovation.¹ Racial minorities are more likely to seek care from physicians of their own race and report increased satisfaction with the care they receive.² Patients who are racially concordant with their clinician require less frequent emergency room care and lower healthcare expenditures.³ Studies have also shown that gender and ethnic concordance are associated with improved preventive screening compliance.⁴ Unfortunately, barriers to the recruitment and retention of under-represented minorities (URM) in surgery persist and include lack of access to mentors from minority backgrounds, disparate opportunities and a focus on metrics that may bias the selection of physician trainees toward the majority.^{5 6}

Though the number of women and racial minorities has increased at the medical school level, a ‘leaky pipeline’ persists, characterised by an inability to retain individuals from under-represented demographics throughout more advanced career stages, including residency.⁷ A starting point for recruiting and retaining racial, ethnic and gender minorities beyond undergraduate medical education is to eliminate sources of bias that may be present in the evaluation of graduate medical education applications. A holistic review of applicants has been supported as a means of encouraging diversification but is not standardised. The Association for American Medical Colleges (AAMC) defines holistic review as ‘a flexible, individualised way of assessing an applicant’s capabilities by which balanced consideration is given to experiences, attributes and academic metrics and, when considered in combination, how the individual might contribute value as a medical student and physician’.⁸ However, the current body of evidence examining the effects of holistic review on the recruitment of racial and gender minorities in surgical residencies is small; it is unable to yield conclusions about the impact of holistic review on diversifying residency cohorts.^{9 10}

The lack of published data about holistic review in surgical residencies provides the impetus to pursue a broader review of all graduate medical education; these practices and findings may also be relevant to general surgery. Because holistic review is intended to consider how well an applicant’s attributes align with the characteristics and mission of a programme, these approaches can be tailored to any given specialty/program.¹¹ We, therefore, propose a systematic review to summarise the state of holistic review in graduate medical education in the USA. First, we will identify discrete components described as comprising the holistic review process. We will also perform a meta-analysis of any data comparing racial and gender makeup of residency programmes employing holistic review compared with traditional review. We hypothesise that there will be significant variability among different programmes in the elements contributing to holistic review, and that those programmes implementing holistic review will interview and matriculate increased proportions of URM and women when compared to traditional applicant review.

Methods and analysis

Our systematic review results will be reported according to the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) reporting checklist. These guidelines have served as the framework for our study design. We plan to complete our final literature search and begin title/abstract screening in February of 2023, with plans to complete full-text screening and publish our results in September 2023.

Search strategy and eligibility criteria

PubMed and Embase (Elsevier) will be searched with the assistance of a health sciences librarian (RS) with published expertise in the conduct of systematic reviews and meta-analyses. The core concepts that will be represented by search terms including ‘holistic review’ and ‘medical residency’. A combination of Medical Subject Headings terms and natural language will be used to develop the initial PubMed search. The PubMed search will then be adapted to Embase. The Amsterdam Efficient Deduplication Method will be used during the search process to remove an initial set of duplicate records.¹² Final search strategies and search details will be provided in the final systematic review results manuscript. In addition to the above, the bibliographies of included studies will be examined to identify additional relevant articles not found through database searches.

Studies meeting the PICO criteria (table 1), published any date before 10 February 2023, and written in English will be included for analysis. Programmes will be considered as implementing holistic review if either explicitly described as such by the authors/programme or if the methodology described meets the AAMC definition above. This systematic review is being performed, in part, with the aim of further understanding the approach employed by programmes claiming to use holistic review.

Table 1.

PICO criteria for study eligibility

Population	Graduate medical education programmes in the USA that describe implementation of holistic review
Intervention	Implementation of holistic review to evaluate applicants to residency training programme
Comparison	Traditional review of residency applicants (do not identify as or describe the components of holistic review)
Outcomes	Components of holistic review described by programmes reporting its use Proportions of interviewed and matriculating residents at each programme identified as URM Proportions of interviewed and matriculating residents at each programme that are male/female

URM, under-represented minorities.

Studies that only describe the implementation of holistic review or results of holistic review implementation without comparison to traditional review will still be included in the study. This will aid in the purpose of characterising the strategies employed by programmes identifying as using holistic review.

Study selection and data collection

Database records will be exported to an EndNote library (Clarivate) and duplicate records will be removed using the Bramer method.¹³ The remaining records will be uploaded to DistillerSR (V.2.37, Evidence Partners, 2022) to continue and complete study selection. All studies identified on initial search with the assistance of an expert research librarian will be independently screened by two project members. Studies will be initially screened by title and abstract to determine if the manuscript describes investigation(s) that include the population of interest (ie, graduate medical education programmes in the USA), addresses the role of holistic review in the application process, and is written in English. Studies that do not sufficiently meet criteria and are rejected by both reviewers will be discarded. Studies that are rejected by one reviewer but not the other will be arbitrated by a third individual.

Full-text review of culled articles will then be undertaken by two reviewers. A standardised data extraction form will be used by each reviewer independently to document: title, author, years of publication and study duration, study type, sample size, name of institution, programme specialty, components of holistic review described, populations compared, gender and racial/ethnic makeup of interviewing and matriculating residents, USMLE Step 1 scores of applicants, and OR/RR or other results from analysis. Using this information, each reviewer will document whether the full manuscript meets criteria for inclusion in the systematic review and/or meta-analysis. For any disagreement, consensus will be reached through discussion by the two screeners. If consensus is not possible, a third project member will serve as final adjudicator. Inter-rater reliability will be assessed, with a kappa of ≥0.80 considered acceptable. If inter-rater reliability is poor, selected articles will be reviewed by the final adjudicator.

The process of staged screening and study selection will be reported in the final manuscript and presented as a PRISMA flow diagram. A list of studies excluded based on full-text criteria and sorted by reason for exclusion will be provided in the final systematic review results manuscript.

The risk of bias for randomised controlled trials will be calculated using the Cochrane Risk of Bias (RoB) 2 instrument (Sterne et al, 2019), and the ROBINS-I will be used to assess risk of bias for non-randomised interventions (Sterne et al, 2016). The results of these assessments for each included study will be reported in tables/figures in the final manuscript.

Statistical analyses

We will first report a narrative summary of the findings regarding the logistical operationalisation of holistic review as described by studies included. If studies are sufficiently homogenous in terms of design and comparator, we will conduct meta-analyses using a random-effects model. Per our primary outcome measure, we aim to identify reports of race/ethnicity and gender distribution of interviewed applicants and matriculating residents at each programme included. We will pool the percentages of URM and gender for interviewee and matriculant populations from each study and report the collective odds ratios (95% CI) of racial and gender minority recruitment for holistic review compared with traditional review. If unable to perform quantitative synthesis (eg, heterogeneity), we will describe the interventions performed at each programme included, as well as describe the primary outcome measures and their results. We may also categorise intervention type within holistic review if common themes/interventions are identified.

We will test the clinical heterogeneity by considering the variability in participant factors among trials and trial factors (randomisation concealment, blinding of outcome assessment, losses to follow-up, treatment type, co-interventions). Statistical heterogeneity will be tested using the χ² test (significance level: 0.1) and I² statistic (0%–40%: might not be important; 30%–60%: may represent moderate heterogeneity; 50%–90%: may represent substantial heterogeneity; 75%–100%: considerable heterogeneity). If high levels of heterogeneity among the trials exist (I² ≥50% or p<0.1), the study design and characteristics will be analysed.

Risk of bias assessment will be used to drive sensitivity analyses that determine heterogeneity of effect on the basis of study quality (ie, analyses will be performed with all studies and then only with studies deemed to be of sufficiently high quality). Confounders that will be considered when using the ROBINS-I tool will include location of residency and number of teaching hospitals affiliated with the residency as the catchment area served by the institution may influence the diversity of the staff and commitment to diversity, equity, and inclusion.

Grading of recommendations assessment, development and evaluation

The quality of evidence for all outcomes will be judged using the Grading of Recommendations Assessment, Development and Evaluation working group methodology. The quality of evidence will be assessed across the domains of risk of bias, consistency, directness, precision and publication bias. Additional domains may be considered where appropriate. Quality will be adjudicated as high (further research is very unlikely to change our confidence in the estimate of effect), moderate (further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate), low (further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate) or very low (very uncertain about the estimate of effect).

Patient and public involvement

Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Ethics and dissemination

Ethical approval was not needed for this study as it does not involve human/animal subjects and represents a protocol for systematic review. Data from previously published studies will be described and analysed. Our protocol is registered with PROSPERO.

This protocol has been developed with plans to report our results according to the PRISMA-P guidelines. The review conduct and reporting of study results will also be in accordance with PRISMA guidelines. After data collection and analyses are complete, a complete manuscript will be submitted for publication in a peer-reviewed journal. By systematically summarising the components of holistic review in its current state, the results of this study will provide actionable steps for implementation and guide standardisation of the holistic review process. Meta-analysis will result in concrete evidence for or against the use of holistic review to improve diversity in graduate medical education.

Ethics statements

Patient consent for publication

Not applicable.