Perspectives Viewpoints.
-
•
Virtual recruitment is new to nearly all internal medicine residency programs.
-
•
The majority of internal medicine program directors were equally or more satisfied with their 2020-2021 incoming intern class compared with the prior 3 intern classes.
-
•
Cost savings for residency programs are substantial for virtual interviews, compared to in-person interviews.
-
•
Internal medicine program directors express varying opinions about the future of virtual and in-person recruitment.
Alt-text: Unlabelled box
Introduction
Due to the coronavirus disease 2019 (COVID-19) pandemic, internal medicine residency recruitment in fall 2020 was exclusively conducted virtually.1 This change represented a major shift in recruitment strategies and operations for US training programs, with limited data on best practices for virtual interviews.2 , 3 Prior to COVID-19, applicants routinely traveled for in-person interviews with program faculty, met with program leadership and current residents, and visited the location and health care facilities. The in-person interview process provided applicants the benefit of learning about the culture and multiple aspects of the program and its location and had a major influence on how applicants ranked programs.4 In 2020, conditions associated with the pandemic demanded a change to a virtual interview process with preparation occurring in a compressed timeline that coincided with unpredictable COVID-19 surges.
Intern recruitment is a core responsibility for internal medicine program directors5 and a sizable undertaking with significant investments of time and resources by residency programs and applicants. Prepandemic, internal medicine programs spent a median of $14,162 per filled intern position with an overall median cost of recruitment per program at $148,000.6 Added to the costs, students, medical schools, and program directors have grappled with escalating numbers of residency applications for several years.7 In 2020, the mean number of Electronic Residency Application Service (ERAS) applications submitted for each applicant to internal medicine programs had risen to a high of 70.7 from 59.3 in 2016,8 representing approximately $1500 in direct application costs per applicant,9 which does not include travel. In response to this “application inflation,” some program directors have increased the number of invitations extended and interviews conducted and have raised their screening standards for interview offers.10 , 11 Virtual recruitment imposed another dimension of uncertainty in an already cost- and labor-intensive process.
Limited data are available about the effects of the wholesale shift to virtual recruitment and how it might affect future recruitment. To better understand this, we conducted a national survey of internal medicine program directors about their experiences and program outcomes in the first completely virtual recruitment season. Further, we solicited program director perspectives about application inflation as well as guiding principles for future recruitment seasons.
Methods
The Association of Program Directors in Internal Medicine (APDIM), a professional organization that represents internal medicine residency program directors, core faculty, and program administrators, is a charter organization of the Alliance for Academic Internal Medicine (AAIM). The APDIM Survey Committee regularly surveys internal medicine program directors on a variety of critical topics in graduate medical education (GME).12
Survey Instrument
In fall 2020, the APDIM Survey Committee recognized the need to conduct a survey of internal medicine program directors regarding the residency virtual interview and recruitment process. The tentative subject matter for the survey was curated from exchanges via the APDIM Discussion Forum (an email listserv including more than 4000 GME faculty and administrators) and the final topics were identified via committee consensus. The initial survey sections were drafted by a subset of committee members, and the complete survey was reviewed and revised by the entire committee. The instrument was programmed in the Qualtrics Surveys13 platform by AAIM staff and was pilot-tested by the committee as well as 6 GME experts outside of the committee. The instrument consisted of 25 questions with conditional (skip or display) logic and included multiple-choice-select-one, 5-point Likert scale, numeric entry, and open-text response questions.
In March 2021, the survey was distributed via a personalized email invitation to 436 internal medicine residency program directors, representing 82% (436 of 532) of US internal medicine residency programs with Accreditation Council for Graduate Medical Education (ACGME) accreditation prior to July 1, 2019 (ie, at least 1 year prior to the academic year to which the survey applied). Four email reminders to nonrespondents were sent, and the survey closed in May 2021. The study (21-AAIM-117) was deemed exempt by Pearl Institutional Review Board (US DHHS OHRP #IRB00007772) under FDA 21 CFR 56.104 and 45CFR46.104(b)(2).
Data Analysis
Data analysis was conducted in Stata 16.1 SE14 by AAIM staff. Before deidentifying the final responses for analysis, the study data set was appended with data from external sources that explained the most survey population variance, including US Census Bureau geographic region.15 Residency program characteristics such as number of approved positions were obtained from the Accreditation Council for Graduate Medical Education Accreditation Database System (Public).16 Program type and other program characteristics were obtained from the American Medical Association Fellowship and Residency Electronic Interactive Database Access System Online.17 American Board of Internal Medicine (ABIM) rolling 3-year residency pass rates were provided by the board.18
Summary statistics included frequencies and percentages for categorical variables and measures of central tendency (eg, mean, median, standard deviation, interquartile range) for continuous variables. To assess the statistical representativeness of the responses, essential program characteristics from the data sources described previously were used to compare respondents and their programs to the survey-eligible population. We tested for goodness-of-fit or associations between categorical variables using the Adjusted Wald (Pearson) χ2 test (1 degree of freedom); a design-based Pearson χ2 (2 degrees of freedom) was used to assess goodness-of-fit when both categorical variables exceeded 2 categories. Due to unequal variances, we used the Welch t-test to compare the means of continuous variables against dichotomous variables. Statistical significance was designated with an alpha level of P ≤ .05. Due to conditional logic or item nonresponse, some denominators do not equal the total number of survey respondents.
Results
The response rate was 61.9% (270 of 436 responses). There was no statistical association between respondents and nonrespondents based on essential characteristics that defined the complete survey population (Table 1 ). Although there was slight over-representation of university-based programs (39.3% among respondents; 31.9% for the population), that difference was not statistically significant (P = .053).
Table 1.
Essential Program Characteristics of Respondents and Nonrespondents: 2021 APDIM Spring Survey of Internal Medicine Residency Program Directors
| Respondents (n = 270) | Nonrespondents (n = 166) | Total (n = 436) | ||
|---|---|---|---|---|
| Characteristic | No. (Column %) | No. (Column %) | No. (Column %) | P Value* |
| Program Type (AMA-FREIDA) | ||||
| University-based | 106 (39.3) | 33 (19.9) | 139 (31.9) | .053 |
| Community-based | 37 (13.7) | 43 (25.9) | 80 (18.4) | .109 |
| Community-based, university-affiliated | 122 (45.2) | 89 (53.6) | 211 (48.4) | .324 |
| Military-based | 5 (1.9) | 1 (0.6) | 6 (1.4) | .349 |
| Census Region (US Census Bureau)† | ||||
| Northeast | 83 (30.7) | 51 (30.7) | 134 (30.7) | .998 |
| Midwest | 60 (22.2) | 38 (22.9) | 98 (22.5) | .922 |
| West | 41 (15.2) | 27 (16.3) | 68 (15.6) | .769 |
| South | 86 (31.9) | 50 (30.1) | 136 (31.2) | .842 |
| VA Affiliation: Yes (ACGME) | 107 (39.6) | 47 (28.3) | 154 (35.3) | .096 |
| Accreditation Status (ACGME) | ||||
| Continued or continued with warning | 260 (96.3) | 152 (91.6) | 412 (94.5) | .099 |
| Initial or initial with warning | 10 (3.7) | 14 (8.4) | 24 (5.5) |
| Mean (SD) | Mean (SD) | Mean (SD) | P Value‡ | |
|---|---|---|---|---|
| Program size: No. ACGME approved positions (Q3-Q1: 38-102; 30-69; 36-86)§ | 56.2 (18.1) | 53.1 (21.2) | 55.0 (20.0) | .177 |
| ABIM pass rate: 2017-2019 (%); n = 248, n = 141, n = 389 | 92.4 (6.3) | 89.4 (10.8) | 91.3 (8.2) | .999 |
| Program director tenure as of March 2021 (years: ACGME) | 7.0 (6.1) | 6.1 (5.7) | 6.7 (5.9) | .101 |
| Program accreditation year (ACGME) (Q3-Q1: 1956-1979; 1960-2015; 1957-2001)§ | 1969.6 (9.8) | 1971.4 (11.6) | 1969.9 (10.5) | .120 |
| Average USMLE Step 1 Score (FREIDA); n = 214, n = 131, n = 345 | 212.5 (11.5) | 213.9 (12.5) | 213.0 (11.9) | .252 |
ABIM = American Board of Internal Medicine; ACGME = Accreditation Council for Graduate Medical Education; AMA-FREIDA = American Medical Association Residency and Fellowship Database; APDIM = Association of Program Directors in Internal Medicine; SD= standard deviation; USMLE = US Medical Licensing Examination; VA = Veterans Affairs.
Bivariate (Adjusted Wald [Pearson]) test of association with 1 degree of freedom).
Excludes programs from 2 US territories, due to small cell sizes/data confidentiality.
Welch t-test with unequal variances.
Interquartile range test (Welch t-test).
Ninety percent of programs (243 of 270) reported that virtual interviews were not employed routinely prior to the 2020-2021 recruitment season. Compared to the 2019-2020 recruitment season, a majority of program directors reported receiving more applications from US medical school students (59.1%, 159/269) and offering more interviews (61.5%, 166/270). The median percentage increase in categorical position interviews offered was 20% (mean: 22, standard deviation: 16.3). Sixty-five percent of respondents (174 of 270) reported that fewer applicants declined or canceled interviews compared to the prior year.
Applicant favorability varied, defined by asking where on the rank list their last matched applicant was compared to their program's average match over the past 3 years. Of program directors who could answer this question, 32.8% (87/265) reported that the last matched position was more favorable, 37.7% (100 of 265) reported it was about the same, and 29.5% (78 of 265) reported it was less favorable. Reported favorability of the last matched position varied by geographic region (Table 2 ) but did not vary by median program size, type, or program director's tenure (median years in position) (data not shown). For program directors who reported less favorable matches, most (85.9%, 67 of 78) attributed that outcome to a “moderate” or “great” extent to virtual recruitment. For program directors who reported more favorable matches, a minority (27.6%, 24 of 87) attributed that outcome to a “moderate” or “great” extent to virtual recruitment. Overall, 81.2% of program directors (216 of 266) reported that they were similarly or more satisfied with their incoming intern class compared with the past 3 years; the responses of which did not vary by program type (Appendix A, available online).
Table 2.
Regional Differences in the Location of Last Matched Applicant on the Rank List (2020-2021 Recruitment Cycle) Compared to the Previous 3 Years
| Residency Program US Census Region | Much More Favorable/More Favorable (n = 87) | About the Same (n = 100) | Less Favorable/Much Less Favorable (n = 78) | Total (n = 265) | P Value* |
|---|---|---|---|---|---|
| Midwest | 18 (20.7) | 21 (21.0) | 21 (26.9) | 60 (22.6) | .552 |
| Northeast | 26 (29.9) | 42 (42.0) | 12 (15.4) | 80 (30.2) | .035 |
| South | 36 (41.4) | 22 (22.0) | 27 (34.6) | 85 (32.1) | .035 |
| West | 7 (8.1) | 15 (15.0) | 18 (23.1) | 40 (15.1) | .019 |
Note: For 265 of 270 respondents; an additional 5 respondents reported “Unsure or Not applicable.”
Design-based Pearson χ2 goodness-of-fit test with 2 degrees of freedom.
Virtual recruitment affected what program directors were able to convey about their programs and what they learned about applicants. Most program directors indicated that virtual recruitment reduced their ability to showcase their program's facilities (89.9%, 240/267), city (87.7%, 235/268), and esprit de corps (82%, 219/267). Program directors (71.3%, 189/265) reported a decreased ability to determine which applicants had a sincere interest in their program. In contrast, the majority of program directors reported that the ability to highlight their program's defining features was unchanged (50.3%, 135 of 268) or had increased (14.6%, 39 of 268).
Excluding program directors who responded “unsure or not applicable,” almost all (94%, 250 of 266) reported a somewhat or substantial decrease in recruitment costs, and 40.9% of program directors (108 of 264) reported increased ability to get faculty to conduct interviews. Slightly less than half of program directors (46.6%, 125 of 270) reported investing more of their time for interview days.
Although a majority of program directors (74.4%, 200 of 269) reported to favor or strongly favor a guiding set of principles on whether internal medicine recruitment should be conducted virtually, in-person, or in a combined fashion, there was not broad consensus about what those principles should be (Appendix B, available online). A minority of program directors endorsed a fully virtual (5.6%, 15 of 269) or a fully in-person (7.4%, 20 of 269) recruitment process. Twenty-one percent (55 of 269) of program directors expressed that programs should make their own determinations and the remainder (28.6%, 164 of 269) preferred a hybrid option. Excluding respondents who reported “unsure,” more than half reported that if unrestricted travel and hospital visitors are allowed for the next recruitment season, they would retain virtual interviews with faculty (59.7%, 151 of 253), videos featuring the program or city (88.5%, 224 of 253), and a fully virtual interview option (61.3%, 155 of 253).
Program directors reported that virtual recruitment contributed to greater (79.5%, 214 of 269) application inflation. Two-thirds (68%, 183 of 269) agreed with the statement that internal medicine residency programs should pursue voluntary preference signaling, whereas 11.2% (30 of 269) reported “neutral,” 16% (43 of 269) disagreed, and 4.8% reported “unsure.”
Discussion
This national survey of internal medicine program directors demonstrated that although virtual recruitment was new to most residency programs, the vast majority (more than 80%) of respondents indicated satisfaction with their incoming class of interns. Respondents reported more applicants to their programs (compared to the previous recruitment season), an increased number of interviews conducted, fewer canceled or declined interviews, and the perspective that virtual interviewing contributed to application inflation. Among program directors who reported less favorable match outcomes, virtual recruitment was perceived to be a contributing factor and, conversely, less so for program directors who reported more favorable match outcomes. Benefits of virtual recruitment were several and included lower costs for nearly all programs and increased ease of faculty participation in recruitment. However, almost half of responding program directors invested more of their time on the interview day. Program directors indicated that it was more challenging to communicate key aspects of their program such as esprit de corps. Although program directors did not express agreement about what principles should shape recruitment in the future, many planned to retain some aspects of virtual recruitment.
Our results are in keeping with the results of a 2021 National Residency Matching Program (NRMP) cross-specialty survey19 and preliminary 2021 ERAS data on internal medicine recruitment.20 Broadly, approximately one-third of program directors indicated that their programs received more applications, invited more applicants, and had fewer cancellations during the 2020-2021 recruitment cycle. For internal medicine residency and many internal medicine subspecialty fellowships, the number of applicants further increased in 2021.21 Internal medicine applications rose by 6% and the mean number of applications submitted by an internal medicine applicant increased by 11% compared to the prior year.21
Similar to internal medicine program directors perspectives in our study, about three-quarters of all residency applicants in a 2021 National Residency Matching Program survey found it challenging to develop an understanding of program culture and degree of fit.19 It remains to be seen what effect this has on resident satisfaction over time. Despite these issues, internal medicine program directors reported satisfaction with their match results. Program directors of colon and rectal surgery programs noted a high degree of satisfaction with match outcomes.22 Otherwise, few studies have reported program directors perceptions of their match results.
It is interesting that program directors who reported less favorable matches attributed it to virtual recruitment in contrast to those who reported more favorable matches and perceived that virtual recruitment played less of a role. Attributing performance of the virtually recruited intern classes solely to virtual recruitment (favorably or unfavorably) overlooks many confounding factors and likely focuses on an obvious but not comprehensive assessment of trainees.
Virtual recruitment likely benefits some programs and not others. It may also explain the variety of opinions reported by program directors about best practices moving forward: For program directors who believed there should be a national policy, only 5.6% expressed a preference for a purely virtual process, suggesting that getting to know applicants and “selling” the culture, facility, and city is important.
Cost savings for applicants and programs are one of the major advantages of virtual recruitment. In this study, virtual interviewing was not associated with a decrease in program directors time spent on interview days, which may be due to the newness of virtual recruitment for 90% of programs. Additionally, the environmental footprint of recruitment was substantially reduced23 as carbon emissions are estimated to be 51,665 metric tons of carbon dioxide per year for residency interviews across the United States.
A majority of internal medicine responding program directors favored implementation of some form of signaling preference pathway in response to ongoing application inflation, a problem seemingly exacerbated by virtual recruitment. A signaling preference pathway—proposed as a solution to application inflation—was piloted by otolaryngology in 2020-2021 through a voluntary, free preference-signaling program.24 A previous study demonstrated that two-thirds of internal medicine program directors were interested in having a mechanism for applicants to indicate a high level of interest.10 For the 2022 recruitment season, the Association of American Medical Colleges (AAMC) is piloting a voluntary supplemental ERAS application in surgery, dermatology, and internal medicine20 that will include preference signaling for up to 5 programs.
To our knowledge, this study is the first to describe internal medicine program director experiences with virtual recruitment and to study their opinions about program outcomes and future plans. However, our study has limitations. The 2020-2021 recruitment season was the first conducted virtually; practices and program directors opinions may change substantially over time. Additionally, not only were program directors modifying their recruitment practices, but they were also responding to a pandemic in which their trainees were on the front lines of care. This pressure may have significantly influenced their experience with and the implementation of virtual recruitment. Although the survey response rate was generally representative of the underlying population, possible nonresponse bias does not allow the results to be completely reflective of the perspectives and experiences of all internal medicine program directors at the time of the survey.
Now that there is more planning time and given that a hybrid recruitment approach may be possible in the future, further studies are needed as program directors refine their approach to recruitment and incorporate virtual and in-person elements. Further, virtual recruitment's impact on diversity and equity in internal medicine residency selection is not yet known.25, 26, 27
In summary, virtual recruitment offers significant cost savings for residency programs and applicants as well as reduced environmental impact, and most program directors reported satisfaction with the outcome. However, it may be more difficult with virtual recruitment to convey all program traits and to assess the extent of applicant interest. Future internal medicine recruitment will be shaped by the experiences of the 2020-2021 recruitment season. Like some other specialties,28, 29, 30, 31, 32, 33 many internal medicine program directors plan to incorporate virtual interviews into future recruitment cycles. This shift suggests that a hybrid approach may become the norm even after the COVID-19 pandemic ends. Application inflation remains a major challenge for both internal medicine programs and applicants and ostensibly was exacerbated by virtual recruitment. The upcoming ERAS supplemental application pilot is an important effort to address this challenging issue, although multiple strategies are likely needed.
Acknowledgments
We would like to thank the members of the Alliance for Academic Internal Medicine–Association of Program Directors in Internal Medicine Survey and Scholarship Committee for its support in reviewing and revising the survey instrument that informed this research, as well as the residency program directors who completed the survey.
Footnotes
Funding: None.
Conflicts of Interest: None. A summary of the results was presented during a freely available webinar for members of the Alliance for Academic Internal Medicine on August 5, 2021.
Authorship: All authors had access to the data and a role in writing this manuscript.
Supplementary data to this article can be found online at https://doi.org/10.1016/j.amjmed.2021.10.017.
Appendix A Reported Satisfaction Level with the Incoming Intern Class (2020-2021 Recruitment Cycle) by Residency Program Type
| University-based (n = 105) | Community-based (n = 37) | Community-based, university-affiliated (n = 120) | Total (n = 262) | P Value* | |
|---|---|---|---|---|---|
| Much more/somewhat more satisfied | 39 (37.1) | 14 (37.8) | 44 (36.7) | 97 (37.0) | .989 |
| About the same | 49 (46.7) | 12 (32.4) | 54 (45.0) | 115 (43.9) | .229 |
| Somewhat more/much more dissatisfied | 17 (16.2) | 11 (29.7) | 22 (18.3) | 50 (19.1) | .104 |
For 262 of 270 respondents (4 nonrespondents). An additional 4 respondents reported “Unsure/Unable to answer.” Excludes 4 military-based programs due to small cell sizes/data confidentiality.
Design-based Pearson χ2 goodness-of-fit test with 2 degrees of freedom
Appendix B Guiding Set of Principles Endorsed by Residency Program Directors
| n = 269 | |
|---|---|
| Recruitment should be only in person (if safe due to COVID-19) | 20 (7.4) |
| Virtual option should be offered even if in-person is the norm again | 77 (28.6) |
| Recruitment should remain virtual with option for subsequent onsite visit | 87 (32.3) |
| Recruitment should be only virtual | 15 (5.6) |
| Each program should be allowed to make its own determination | 55 (20.5) |
| Unsure | 15 (5.6) |
Notes: For 269 of 270 respondents; 1 nonrespondent.
The survey question asked “If there is general agreement in favor of a guiding set of principles, which of the following would you endorse?”
References
- 1.Association of American Medical Colleges (AAMC). Final Report and Recommendations for Medical Education Institutions of LCME-Accredited, U.S. Osteopathic, and Non-U.S. Medical School Applicants. Available at:https://www.aamc.org/system/files/2020-05/covid19_Final_Recommendations_05112020.pdf. Accessed August 2, 2021.
- 2.Huppert LA, Hsiao EC, Cho KC, et al. Virtual interviews at graduate medical education training programs: determining evidence-based best practices [e-pub ahead of print]. Acad Med. doi:10.1097/acm.0000000000003868. Accessed September 28, 2021. [DOI] [PubMed]
- 3.Wolff M, Burrows H. Planning for virtual interviews: residency recruitment during a pandemic. Acad Pediatr. 2021;21(1):24–31. doi: 10.1016/j.acap.2020.10.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.National Resident Matching Program. National Resident Matching Program, Data Release and Research Committee: Results of the 2019 NRMP Applicant Survey by Preferred Specialty and Applicant Type. Available at:https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2019/06/Applicant-Survey-Report-2019.pdf. Accessed August 2, 2021.
- 5.Accreditation Council for Graduate Medical Education (ACGME. Common Program Requirements (Residency). Available at:https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRResidency2020.pdf. Accessed August 2, 2021.
- 6.Brummond A, Sefcik S, Halvorsen AJ, et al. Resident recruitment costs: a national survey of internal medicine program directors. Am J Med. 2013;126(7):646–653. doi: 10.1016/j.amjmed.2013.03.018. [DOI] [PubMed] [Google Scholar]
- 7.Antono B, Willis J, Phillips RL, Jr, Bazemore A, Westfall JM. The price of fear: an ethical dilemma underscored in a virtual residency interview season. J Grad Med Educ. 2021;13(3):316–320. doi: 10.4300/jgme-d-20-01411.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Association of American Medical Colleges (AAMC). ERAS Statistics 2016-2020- Internal Medicine (Categorical). Available at: https://www.aamc.org/data-reports/interactive-data/eras-statistics-data. Accessed August 16, 2021.
- 9.Association of American Medical Colleges (AAMC). Fees for ERAS Residency Applications. Available at:https://students-residents.aamc.org/applying-residencies-eras/fees-eras-residency-applications. Accessed September 13, 2021.
- 10.Angus SV, Williams CM, Stewart EA, Sweet M, Kisielewski M, Willett LL. Internal medicine residency program directors' screening practices and perceptions about recruitment challenges. Acad Med. 2020;95(4):582–589. doi: 10.1097/ACM.0000000000003086. [DOI] [PubMed] [Google Scholar]
- 11.Sweet ML, Williams CM, Stewart E, et al. Internal medicine residency program responses to the increase of residency applications: differences by program type and characteristics. J Grad Med Educ. 2019;11(6):698–703. doi: 10.4300/JGME-D-19-00194.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Marshall AL, Halvorsen AJ, Kearns L, Reich LM, West CP, Oxentenko AS. Disparities in salary and work-life integration in internal medicine program directors are associated with gender and partner employment status. Am J Med. 2021;134(2):285–291.e1. doi: 10.1016/j.amjmed.2020.10.004. [DOI] [PubMed] [Google Scholar]
- 13.Qualtrics. Qualtrics Software. Version XM. Provo, UT: Qualtrics; 2020.
- 14.Stata Corp . StataCorp LLC; College Station, TX: 2019. Stata Statistical Software. Version Release 16. [Google Scholar]
- 15.US Census Bureau. Census Regions and Divisions of the United States. Available at: https://www2.census.gov/geo/pdfs/maps-data/maps/reference/us_regdiv.pdf. Accessed December 1, 2020.
- 16.Accreditation Council for Graduate Medical Education (ACGME). Accreditation Database System Online (Public). Available at: https://apps.acgme.org/ads/Public/Programs/Search. Accessed June 1, 2020.
- 17.American Medical Association (AMA). Fellowship and Residency Electronic Interactive Database Access System (FREIDA) Online. Available at: https://freida.ama-assn.org/search/list?spec=42771. Accessed September 1, 2020.
- 18.American Board of Internal Medicine (ABIM).Residency Program Pass Rates [2008-2019]. Available at:https://www.abim.org/Media/lhgmdidp/residency-program-pass-rates.pdf. Accessed June 2020.
- 19.National Resident Matching Program. 2021 Applicant and Program Director Survey Findings: Impact of the Virtual Experience on the Transition to Residency. Available at:https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2021/05/Research-Brief-Virtual-Experience-2021-FINAL.pdf. Accessed August 17, 2021.
- 20.Association of American Medical Colleges (AAMC). Supplemental ERAS application (for the ERAS 2022 cycle). Available at: https://students-residents.aamc.org/applying-residencies-eras/supplementalerasapplication. Accessed September 7, 2021.
- 21.Huppert LA, Santhosh L, Babik JM. Trends in US internal medicine residency and fellowship applications during the COVID-19 pandemic vs previous years. JAMA Netw Open. 2021;4(4) doi: 10.1001/jamanetworkopen.2021.8199. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.D'Angelo JD, D'Angelo AD, Mathis KL, Dozois EJ, Kelley SR. Program director opinions of virtual interviews: whatever makes my partners happy [e-pub ahead of print]. J Surg Educ. doi:10.1016/j.jsurg.2021.04.008. Accessed August 17, 2021. [DOI] [PubMed]
- 23.Donahue LM, Morgan HK, Peterson WJ, Williams JA. The carbon footprint of residency interview travel. J Grad Med Educ. 2021;13(1):89–94. doi: 10.4300/jgme-d-20-00418.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Society of University Otolaryngologists. Otolaryngology Preference Signaling. Available at: https://opdo-hns.org/mpage/signaling. Accessed August 18, 2021.
- 25.Hammoud MM, Standiford TC, Carmody JB. The 2020-2021 residency application cycle: Lessons learned and lingering problems. JAMA. 2021;325(22):2249–2250. doi: 10.1001/jama.2021.5708. [DOI] [PubMed] [Google Scholar]
- 26.Nwora C, Allred DB, Verduzco-Gutierrez M. Mitigating bias in virtual interviews for applicants who are underrepresented in medicine. J Natl Med Assoc. 2021;113(1):74–76. doi: 10.1016/j.jnma.2020.07.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Heitkamp NM, Snyder AN, Ramu A, et al. Lessons learned: applicant equity and the 2020-2021 virtual interview season. Acad Radiol. doi:10.1016/j.acra.2021.08.005. Accessed September 29, 2021. [DOI] [PMC free article] [PubMed]
- 28.Hamade N, Bhavsar-Burke I, Jansson-Knodell C, et al. Virtual gastroenterology fellowship recruitment during COVID-19 and its implications for the future [e-pub ahead of print]. Dig Dis Sci. doi:10.1007/s10620-021-07014-1. Accessed August 17, 2021. [DOI] [PMC free article] [PubMed]
- 29.Hill MV, Ross EA, Crawford D, et al. Program and candidate experience with virtual interviews for the 2020 Complex General Surgical Oncology interview season during the COVID pandemic. Am J Surg. 2021;222(1):99–103. doi: 10.1016/j.amjsurg.2020.11.007. [DOI] [PubMed] [Google Scholar]
- 30.Jimenez AE, Khalafallah AM, Romano RM, et al. Perceptions of the virtual neurosurgery application cycle during the coronavirus disease 2019 (COVID-19) pandemic: a program director survey. World Neurosurg. 2021;154:e590–e604. doi: 10.1016/j.wneu.2021.07.078. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Rajesh A, Asaad M, Elmorsi R, Ferry AM, Maricevich RS. The virtual interview experience for MATCH 2021: a pilot survey of general surgery residency program directors [e-pub ahead of print]. Am Surg. doi:10.1177/00031348211038555. Accessed September 30, 2021. [DOI] [PubMed]
- 32.Rhoades JS, Ramsey PS, Metz TD, Lewkowitz AK. Maternal-fetal medicine program director experience of exclusive virtual interviewing during the coronavirus disease 2019 pandemic. Am J Obstet Gynecol MFM. 2021;3(4) doi: 10.1016/j.ajogmf.2021.100344. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Geary AD, Wang TS, Lindeman B, et al. Perspectives on virtual interviews-a follow-up study of the Comprehensive Endocrine Surgery Fellowship interview process [e-pub ahead of print]. Surgery. doi:10.1016/j.surg.2021.03.069. Accessed September 30, 2021. [DOI] [PubMed]