Abstract
Purpose
With the United States Medical Licensing Examination Step 1 transition to pass/fail in 2022, uncertainty exists regarding how other residency application components, including research conducted during medical school, will inform interview and ranking decisions. The authors explore program director (PD) views on medical student research, the importance of disseminating that work, and the translatable skill set of research participation.
Method
Surveys were distributed to all U.S. residency PDs and remained open from August to November 2021 to query the importance of research participation in assessing applicants, whether certain types of research were more valued, productivity measures that reflect meaningful research participation, and traits for which research serves as a proxy. The survey also queried whether research would be more important without a numeric Step 1 score and the importance of research vs other application components.
Results
A total of 885 responses from 393 institutions were received. Ten PDs indicated that research is not considered when reviewing applicants, leaving 875 responses for analysis. Among 873 PDs (2 nonrespondents), 358 (41.0%) replied that meaningful research participation will be more important in offering interviews. A total of 164 of 304 most competitive specialties (53.9%) reported increased research importance compared with 99 of 282 competitive (35.1%) and 95 of 287 least competitive (33.1%) specialties. PDs reported that meaningful research participation demonstrated intellectual curiosity (545 [62.3%]), critical and analytical thinking skills (482 [55.1%]), and self-directed learning skills (455 [52.0%]). PDs from the most competitive specialties were significantly more likely to indicate that they value basic science research vs PDs from the least competitive specialties.
Conclusions
This study demonstrates how PDs value research in their review of applicants, what they perceive research represents in an applicant, and how these views are shifting as the Step 1 exam transitions to pass/fail.
Performance on the United States Medical Licensing Examination (USMLE) Step 1 has historically factored heavily into decisions to interview and rank applicants for graduate medical education residency programs, despite only moderate correlation with residency success.1–5 With the USMLE Step 1 transition to pass/fail in 2022, there is uncertainty as to how other application components will inform selection decisions, creating a challenging environment for both students and advisers.6–9 Several specialty-specific reports indicated that letters of recommendation, USMLE Step 2 scores, and the medical school performance evaluation would be highly valued going forward.10–14 However, several studies report conflicting predictive values of these and other criteria on trainee success.5,15–17
Student research participation is often a factor that program directors (PDs) consider when reviewing applicants. Aggregate data from the 2021 National Resident Matching Program (NRMP) Program Director Survey show that 41.1% of all PDs consider involvement and interest in research when extending interview offers (mean importance score of 3.6, with 1 indicating not at all important and 5 indicating very important), and 29.8% of PDs consider research for rank list decisions (mean importance score, 3.7).18 A majority of students already participate in research during medical school,19,20 although there is great variation in research opportunities and training.21 Students participating in dedicated research programs and with a higher number of publications22,23 are more likely to receive invitations to interview24 and to match23,25–28 in certain specialties. Although the NRMP Program Director Survey provides useful information about which specialties are more or less likely to consider research in interview and ranking decisions, there remains a lack of information on which types of research and what extent of dissemination are considered by PDs to be most salient in these decisions. Furthermore, it remains unclear how research may be used as a proxy for personal qualities, such as time management, critical thinking, and self-directed learning.
As residency application numbers continue to increase among U.S. senior medical students29 and medical schools look to foster research activities for students that meet both educational and application objectives, a better understanding of how PDs are using research participation to make decisions is needed. Transparency among residency programs regarding the interview and ranking process will better inform education practice and student advising regarding research endeavors and addresses a call to action recently made by the Coalition for Physician Accountability.30 As directors of medical student research programs at 14 institutions across the United States, we support students who engage in a broad array of research types, including clinical, basic science, community based, medical education, and quality improvement, among others. Students frequently seek our guidance in selecting projects and mentors, concerned about positioning themselves for the residency application process. With this study, we sought to explore PD views on the wide array of research conducted by medical students, the importance of disseminating that work in various ways, and the translatable skill set that research participation represents.
Method
We designed a survey for U.S. residency PDs by iterative consensus discussion. Survey domains were identified following the criteria explored by Green et al2 and supplemented by common student concerns brought to us over collective decades of research program administration. A modified nominal group technique31 via video conferencing platform was used to further refine domains and survey questions during 5 meetings. An expert consultant from the University of Chicago Survey Lab provided guidance and review of survey item and response formatting to follow best practice in the field. A final survey version was pilot tested with local PDs from the University of Chicago, resulting in minor edits for clarity and visualization on the Qualtrics platform.
The survey asked about program specialty and setting and queried the importance (high, moderate, low, or not considered) of participation in research or scholarly work when reviewing applicants. Respondents indicating that research was “not considered” exited the survey. We asked PDs to anticipate whether research will carry increased weight in offering interviews and ranking highly for the match with a pass/fail USMLE Step 1. We asked whether sending graduates into academic or physician–scientist careers was a program goal and how PDs consider various accomplishments in determining meaningful research participation (high, moderate, low, or not considered). We asked whether PDs value certain types of research more highly than others and whether PDs used meaningful research participation as a proxy for a variety of traits, including commitment to specialty and self-directed learning skills. Next, we paired meaningful research participation with other application components (e.g., USMLE Step 2 score, clerkship grades, letters of recommendation) and asked which was more important to garner an interview. At the conclusion of the survey, we asked participants, “Is there anything else that you would like to tell us about the role of medical school research experience in how likely prospective residents are to be invited to interview or in how those interviewed are ranked” to capture additional themes for future investigation. The complete survey is found in Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/B416.
The survey was distributed electronically using Qualtrics to PDs at all U.S. residency programs to which senior medical students can apply, excluding preliminary year programs. Program lists and emails were obtained from the Accreditation Council for Graduate Medical Education website, yielding 5424 unique emails. If an email was not specific to the PD, we attempted to locate the PD email using the American Medical Association FREIDA website or by using online search engines. The final distribution list included 5,242 deliverable emails (182 [3.4%] were undeliverable), with only 1 email per program. Of these, 4,450 (84.9%) were academic medical center or university-affiliated programs, 677 (12.9%) were community-based programs, and 115 (2.2%) were military or federal programs. The survey was open from August to November 2021, with intermittent reminders, and closed when reminders yielded fewer than 20 additional responses.
We followed the example of Green et al2 and divided residency specialties into thirds based on the percentage of entry-level spots filled by U.S. MD degree senior students in 2021. Specialties were ranked from the highest percentage of spots filled by U.S. MD degree senior students to the lowest percentage.32–34 We assigned the top third of our respondents to the most competitive group, the middle third to the competitive group, and the lowest third to the least competitive group (Table 1). These groupings were used to categorize programs for our analysis and do not attribute value to the mission, setting, or specialty of the program.
Table 1.
Data were analyzed using χ2 tests for frequency comparisons among groups and logistic regression for binary responses. To address possible differences among respondents based on program characteristics, we stratified results by either program setting (academic medical center or university affiliated, community based, or military or federal) or competitiveness in our univariable analyses. In multivariable analyses, we included a priori those adjustment variables that described program setting, specialty competitiveness, and the importance placed on research and scholarly work in applicant review. We further considered other variables potentially related to research importance, including whether sending graduates into academic or physician–scientist careers was a program goal. We examined collinearity among variables using a model examining the question, “When reviewing a student’s application for residency, of how much importance is participation in research/scholarly work during medical school?” We kept those variables with a variance inflation factor less than 5 and used backward selection to exclude variables with P > .10. Variables remaining in our final adjustment model included program setting, competitiveness grouping, importance of research and scholarly work to applicant review, and whether sending graduates into physician–scientist careers is a program goal. P < .05 was defined as statistically significant. Analyses were conducted using Stata/SE software, version 14 (StataCorp, College Station, Texas). This study received exempt determination from the University of Chicago Institutional Review Board.
Results
We received 885 responses from the 5,242 deliverable emails (16.9%), representing 885 unique programs at 393 institutions across the United States (range of responses per institution, 1–12; mean [SD], 2.25 [2.01]). Of the 885 PDs who responded, 663 (74.9%) were from academic medical center or university-affiliated programs (response rate, 14.9% [663/4450]), 208 (23.5%) were from community-based programs (response rate, 30.7% [208/677]), and 14 (1.6%) were from military or federal programs (response rate, 12.2% [14/115]). On the basis of their proportions among the PDs invited to participate, PDs from academic medical center or university-affiliated programs were relatively underrepresented among our respondents (84.9% of invitees vs 74.9% of respondents), whereas PDs from community-based programs were relatively overrepresented (12.9% of invitees vs 23.5% of respondents). On the basis of the specialty fill rate by U.S. MD degree senior students in 2021, of the 885 programs, 305 (34.4%) were categorized as most competitive, 286 (32.3%) as competitive, and 294 (33.2%) as least competitive (Table 1). Of the 305 responses categorized from the most competitive specialties, 256 (83.9%) were in academic medical center or university-affiliated settings, 48 (15.7%) were in community settings, and 1 (0.3%) was in a military setting. Of the 294 responses categorized from the least competitive specialties, 179 (60.9%) were in academic medical center or university-affiliated settings, 106 (36.1%) were in community settings, and 9 (3.1%) were in military or federal settings. Statistically significant differences in frequency of each setting by competitiveness category were seen (P < .001). The importance of research participation in reviewing applicants was rated as high by 168 PDs (19.0%), moderate by 452 PDs (51.1%), and low by 255 PDs (28.8%). Ten PDs (1.1%; 7 in an academic medical center or university-affiliated setting and 3 in a community-based setting) indicated that research is not considered when reviewing applicants, leaving 875 responses for analysis. Of 873 PDs, 546 (62.5%; 2 nonrespondents) indicated that training physician–scientists (or clinician–scientists) was a program goal: 446 of 654 (68.2%) in an academic medical center or university-affiliated setting, 90 of 205 (43.9%) in a community-based setting, and 10 of 14 (71.4%) in a military or federal setting (P < .001).
Among 873 PDs (2 nonrespondents), 358 (41%) replied that with a pass/fail USMLE Step 1 meaningful research participation will be more important in offering interviews. In univariate analysis, a significant difference was found among specialty competitiveness groups, with 164 of 304 (53.9%) from the most competitive specialties reporting increased importance of research compared with 99 of 282 (35.1%) (P < .001) and 95 of 287 (33.1%) (P < .001) of competitive and least competitive specialties, respectively. In a separate univariate analysis, a significant difference was found among program settings, with 289 of 654 academic medical center or university-affiliated programs (44.2%) reporting increased importance of research compared with 62 of 205 community-based programs (30.2%) (P < .001). Results describing increased importance of research participation in ranking applicants were similar (Figure 1).
We found significant differences regarding the importance of various accomplishments in defining meaningful research participation based on specialty competitiveness (Figure 2). The accomplishment most commonly indicated to be of high or moderate importance was completion of an original research project (545 of 874 [62.4%]). This was identified more often by PDs from the competitive (170 of 281 [60.5%], P = .04) and most competitive (232 of 304 [76.3%], P = .02) specialties compared with the least competitive specialties (143 of 289 [49.5%]) when adjusted for program setting, importance of research in evaluating applications, and whether sending graduates into physician–scientist careers was a program goal. Compared with PDs from the least competitive specialties, PDs from the most competitive specialties more often identified having 1 publication (69.8% [210/301] vs 48.1% [139/289], P = .02) as important in the adjusted model. Although there was no difference in the importance of publication in a high-impact journal based on specialty competitiveness, it is notable that the P for trend from least to most competitive specialties was significant (P = .049) and that 121 of 303 PDs (39.9%) from the most competitive specialties identified this accomplishment as important. Additional relationships are shown in Figure 2.
Of the 875 respondents, 624 (71.3%) replied that they do not value certain types of research more than others, whereas 251 (28.7%) indicated that they value some types of research more than others, without a significant difference based on competitiveness category. Incidentally, these 251 PDs were distributed across program settings in similar proportions to the total respondent pool (academic medical center or university-affiliated settings, 183 [72.9%]; community-based settings, 65 [25.9%]; and military or federal settings, 3 [1.2%]). These 251 PDs then indicated which types of research were more highly valued (Table 2). Clinical research was most often identified as highly valued (210 [83.7%]). Other types of research commonly identified as high value included quality improvement (156 [62.2%]), medical education (129 [51.4%]), translational science (126 [50.2%]), and public or community health (125 [50.2%]). Of note, basic science was only identified as being of high value by 91 respondents (36.3%).
Table 2.
We separately analyzed responses from the same 251 PDs who valued some types of research more than others to identify differences based on program setting (academic medical center or university affiliated, community based, or military or federal) (see Supplemental Digital Appendix 2 at http://links.lww.com/ACADMED/B416). Of the 183 PDs from academic medical center or university-affiliated programs, PDs valued clinical research (162 [88.5%]), basic science (79 [43.2%]), and translational science (103 [56.3%]) at significantly higher rates than the 65 PDs from community-based programs (clinical research, 46 [70.8%]; basic science, 65 [16.9%]; and translational science, 22 [33.9%]). PDs from community-based programs valued public or community health research at a significantly higher rate (42 of 65 [64.6%]) than PDs from academic medical center or university-affiliated programs (80 of 183 [43.7%]).
In logistic regression of the same 251 PDs who valued some types of research more than others, those from the most competitive specialties were significantly more likely to indicate that they value basic science research compared with PDs from the least competitive specialties, both unadjusted (odds ratio [OR], 4.89; 95% CI, 2.55–9.38; P < .001) and when adjusted for program setting, importance of research in evaluating applicants, and whether the program aims to send graduates to physician–scientist careers (OR, 2.61; 95% CI, 1.27–5.40; P = .009). Translational science (OR, 2.31; 95% CI, 1.26–4.23; P = .007) and clinical research (OR, 2.88; 95% CI, 1.21–6.84; P = .02) were also more highly valued by PDs from the most competitive specialties in the unadjusted model but not once adjusted as above. PDs categorized into the least competitive group were more likely to identify quality improvement, community health, global health, health systems science, and medical humanities as being of greater value compared with PDs from competitive or most competitive specialties. Specific relationships are given in Table 2.
Of the 873 respondents, 309 (35.4%) indicated that they weigh productivity differently based on research type, with PDs from the most competitive specialties (133 of 302 [44.0%]) indicating this significantly more often than PDs from the competitive (83 of 282 [29.4%], P < .001) and least competitive (93 of 289 [32.2%], P = .003) specialties. These PDs then indicated from which research types they expect greater productivity. Although basic science was a more valued research type among PDs from both competitive and most competitive specialties (Table 2), logistic regression models showed that increased productivity was not expected (see Supplemental Digital Appendix 3 at http://links.lww.com/ACADMED/B416). PDs most often expected greater productivity from clinical research (206 of 309 [66.7%]), which was also the most common highly valued research type. In the unadjusted model, PDs from the most competitive specialties expected greater productivity from students conducting clinical research compared with PDs from the least competitive specialties (OR, 1.84; 95% CI, 1.04–3.25; P = .04). Additional relationships are detailed in Supplemental Digital Appendix 3 at http://links.lww.com/ACADMED/B416.
PDs indicated traits for which meaningful research participation serves as a proxy, including commitment to specialty, critical thinking skills, intellectual curiosity, interest in an academic career, self-directed learning skills, and others (see Supplemental Digital Appendix 4 at http://links.lww.com/ACADMED/B416). Only 3 traits were indicated by more than half of the 875 PDs: intellectual curiosity (545 [62.3%]), critical and analytical thinking skills (482 [55.1%]), and self-directed learning skills (455 [52.0%]). We compared the frequency with which each trait was selected by competitiveness and program setting. Research participation was more often a proxy for commitment to specialty among the 305 most competitive specialties (108 [35.4%]) compared with the 294 least competitive specialties (73 [24.8%], P = .005) and among the 663 academic medical center or university-affiliated programs (211 [31.8%]) compared with the 208 community-based programs (48 [23.1%], P = .02), despite this trait being infrequently identified overall.
We asked PDs to consider applications with a pass/fail USMLE Step 1 and compare research participation with each of 7 applicant factors for importance in deciding whom to interview. Research participation was not more important than any other factor but often was of equal importance (Figure 3). Letters of recommendation, specialty elective performance, and clerkship grades were more important than research for more than 50% of all PDs regardless of specialty competitiveness grouping. However, research participation was of equal or greater importance than election to Alpha Omega Alpha (or a similar honor society) or service work among all PDs and of equal or more importance than leadership during medical school and the numeric USMLE Step 2 score among PDs from the most competitive specialties. PDs from the most competitive specialties responded that research is more important than service work, leadership, and clerkship grades more frequently than PDs from competitive or least competitive specialties (P < .001). PDs from the most competitive specialties state that research is more important than performance in electives in the specialty more frequently than PDs from the competitive specialties (P = .001) or least competitive specialties (P < .001). PDs from the most competitive specialties state that research is more important than high USMLE Step 2 score more frequently than PDs from the competitive and least competitive specialties (P = .001). PDs from the most competitive specialties state that research is more important than election to Alpha Omega Alpha (or similar societies) more frequently than PDs from the LC specialties (P = .02).
Our open-ended exploratory question yielded detailed comments from 19% of respondents. In the preliminary analysis, multiple aspects of research experiences not addressed in our survey were mentioned, including inequities in access to research opportunities, a perceived inverse relationship between research experiences and clinical skills, differing priorities among programs, and the degree of the student’s contribution to the project.
Discussion
This study explores how residency PDs anticipate evaluating applications with a pass/fail USMLE Step 1 and specifically how this change will impact the importance of research participation and productivity. With a pass/fail USMLE Step 1, 41.0% of PDs overall state they will be more reliant on meaningful participation in research to select applicants to interview, with 53.9% of PDs from the most competitive specialties indicating that research will carry more weight in offering interviews. On the basis of our broad experience counseling students seeking research opportunities at institutions across the country, the relevance of research highlighted in the annual Graduation Questionnaire20 and NRMP data,22 and the results of this study, the perception of the importance of research and the actual pursuit of research have increased in response to the recent USMLE Step 1 pass/fail transition.
Studies support that medical student research helps build self-efficacy,35,36 stimulate interests,37 and develop scholarly abilities.21,38 Students perceive research as a way to distinguish themselves39 and are anxious to determine the level of productivity they should achieve and which research types are most valued by PDs. Although clinical research was most frequently noted in this study to be highly valued, students pursuing clinical research are also expected to be more productive, a finding magnified in the most competitive specialties. In our collective experience advising students, we often discover reluctance to pursue basic science and translational research because of the length of time needed to generate findings for presentation or publication. This difference from clinical research is recognized by PDs, with corresponding lower expectations for productivity from basic and translational research.
Our study has implications for medical schools and for students considering pursuit of research. Our study reflected on the key considerations for schools when developing, redesigning, or investing in student research programs.21,35,37,40–44 Our findings suggest that a broad array of research types are valued by PDs (Table 2) and should be offered whenever possible. Although the term research often calls to mind basic, translational, or clinical research performed in an academic setting, a large majority of our respondents said that they value all types of research equally. Of those who value some types more than others, quality improvement, medical education, and public or community health are also frequently noted to be highly valued by PDs. Furthermore, 30% of respondents from community-based programs noted that with a pass/fail USMLE Step 1 score student research will be more important in evaluating applicants. Our results indicate the importance of all types of research to PDs from diverse program specialties and settings. Therefore, to maximize the impact of research in residency applications, schools should pair research opportunities with curricula that allow students to build skills in presentation at national meetings, collaboration, and manuscript preparation because PDs rely on such metrics to assess whether research participation has been meaningful (Figure 2). Provision of strong research support and mentorship can also benefit the pipeline of future physician–scientists and, in particular, can ameliorate disparities among students from underrepresented backgrounds.45 However, even for students who do not plan to pursue career-long research, scholarly work develops skills in curiosity, critical thinking, and self-directed learning, which are recognized by most PDs (see Supplemental Digital Appendix 4 at http://links.lww.com/ACADMED/B416).
The observation that some PDs will consider research of equal or greater importance than the Step 2 score is provocative. The changing significance of Step 2 is a topic that has been hotly debated of late, with multiple survey studies reporting and blog posts opining that PDs will consider Step 2 “the new Step 1.”46–57 The sentiment that Step 2 will be relied on more heavily, at least compared with student research activity, was not expressed by respondents in this study. Most likely, the process of selecting applicants for interviews and ranking without a numeric Step 1 score will take several years to refine, and the question of whether research continues to be of equal or greater importance to the Step 2 score will merit revisiting at a future date.
Finally, this study asks how faculty can guide medical students who seek to optimize their own residency applications in what is currently uncharted territory. Figure 3 highlights that our respondents often identify research as being of equal importance to other applicant factors. From this, it is reasonable to infer that PDs are transitioning to the use of aspects of holistic review, which should be reassuring. Students will likely be well served by conducting research that is meaningful to them, while also recognizing how PDs view research focus (Table 2) and corresponding expectations for productivity (see Supplemental Digital Appendix 3 at http://links.lww.com/ACADMED/B416). A frequent concern of students is that they need to conduct research in an area related to their intended specialty, but only 29.8% of respondents to this survey viewed research as an indicator of commitment to their specialty (see Supplemental Digital Appendix 4 at http://links.lww.com/ACADMED/B416). This view was significantly more common among the most competitive specialties, which include all surgical subspecialties, but was still held by only a minority (35.4%). Students applying to the most competitive specialties (and those advising them) should make particular note of the differential weight placed on research and accompanying productivity that is reported in this study, which is supported by specialty-specific surveys.25,28,58–64 All students should recognize that skills gained through research are broadly applicable to any career in medicine65 and that PDs consider meaningful participation in research a useful proxy for several positive applicant traits (see Supplemental Digital Appendix 4 at http://links.lww.com/ACADMED/B416). Ultimately, students can feel confident that a deep dive into a research project will be a unique feature of their application.
There are several limitations to this study. Interest in research was found among PDs from both academic medical centers and community-based programs, and broad generalizability of findings is not truly possible, given the low response rate. However, we were able to draw meaningful conclusions across program settings by presenting overall analyses coupled with both adjusted and stratified analyses. In doing so, we sought to avoid biasing our conclusions to allow those advising students to glean important information from our results independent of student aspirations. We also acknowledge that there are many ways to group programs by competitiveness, and these groupings will impact analysis. In this study, to allow for comparison, we emulated the method used in prior literature,2 understanding that this only accounts for program fill rates among U.S. senior students. There are limitations to this strategy. For example, it does not account for how competitive specialties are among osteopathic or international medical graduates. Finally, this study was based on reported perceptions and as such is subject to reporting bias, including social desirability. Reported perceptions may not reflect the actual approach PDs will take when reviewing applicants. Additional studies will be needed to review characteristics of matched applicants and not just perceptions. Additionally, responses to the open-ended question revealed themes that we did not explore in our survey. Evaluation and analysis of these evolving topics are warranted and necessary.
Conclusions
The transition of USMLE Step 1 to pass/fail is based, in part, on the recommendations from the Invitational Conference on USMLE Scoring summary report,66 which sought to “reduce the adverse impact of the current overemphasis on USMLE performance in residency screening and selection.”66 This change, not surprisingly, has introduced significant uncertainty into the residency match process for students, advisers, and PDs. It will be several years before it is clear which elements of a residency application are most influential in the screening and selection process. In the interim, this study provides insight into the anticipated role that student research is likely to play. Our findings suggest that PDs value meaningful participation in and completion of research, that research participation demonstrates specific desirable attributes (e.g., curiosity, critical thinking), and that the perceived value of research when evaluating a residency application varies depending on program competitiveness and emphasis. With a pass/fail Step 1 exam, medical student research may be an important differentiator for both medical students and PDs as residency programs seek to implement a more holistic evaluation of applicants. We encourage all allopathic and osteopathic medical schools to bolster a broad array of research opportunities, with particular attention to development of translatable skills honed through research, to optimize their students’ development as physicians and access to opportunities.
Acknowledgments:
The authors thank Martha Van Haitsma of the University of Chicago Survey Lab for her expert advice on survey design and Dr. Vineet Arora, dean for medical education, University of Chicago Pritzker School of Medicine, for support of the Scholarly Concentrations Collaborative.
Supplementary Material
Footnotes
Supplemental digital content for this article is available at http://links.lww.com/ACADMED/B416.
Funding/Support: None reported.
Other disclosures: None reported.
Ethical approval: This study was approved as exempt by the University of Chicago Institutional Review Board, June 24, 2021, #IRB21-0651.
Contributor Information
Paige C. Fairchild, Email: pfairc01@gmail.com.
Ingrid Bahner, Email: ibahner@usf.edu.
Dwayne M. Baxa, Email: baxa@oakland.edu.
Deborah R. Birnbaum, Email: debbirnb@iu.edu.
Sarwat I. Chaudhry, Email: sarwat.chaudhry@yale.edu.
Katherine C. Chretien, Email: kchretien@jhmi.edu.
Donald B. DeFranco, Email: dod1@pitt.edu.
Amber Z. Deptola, Email: aszimm01@gmail.com.
Leslie E.W. LaConte, Email: lacon001@vtc.vt.edu.
Jenny J. Lin, Email: Jenny.Lin@mountsinai.org.
Leslie Petch Lee, Email: lpetch@uga.edu.
Maureen A. Powers, Email: mpowers@emory.edu.
Ira J. Ropson, Email: iropson@psu.edu.
Saumya M. Sankaran, Email: saumyas1@uw.edu.
Kara E. Sawarynski, Email: sawaryns@oakland.edu.
Stephen M. Sozio, Email: ssozio@jhmi.edu.
References
- 1.Bowe SN, Laury AM, Gray ST. Associations between otolaryngology applicant characteristics and future performance in residency or practice: A systematic review. Otolaryngol Neck Surg. 2017;156:1011–1017. [DOI] [PubMed] [Google Scholar]
- 2.Green M, Jones P, Thomas JX. Selection criteria for residency: Results of a national program directors survey. Acad Med. 2009;84:362–367. [DOI] [PubMed] [Google Scholar]
- 3.Gauer JL, Jackson JB. The association between United States Medical Licensing Examination scores and clinical performance in medical students. Adv Med Educ Pract. 2019;10:209–216. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Hartman ND, Lefebvre CW, Manthey DE. A narrative review of the evidence supporting factors used by residency program directors to select applicants for interviews. J Grad Med Educ. 2019;11:268–273. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Yang A, Gilani C, Saadat S, Murphy L, Toohey S, Boysen‐Osborn M. Which applicant factors predict success in emergency medicine training programs? A scoping review. AEM Educ Train. 2020;4:191–201. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Ehrlich H, Sutherland M, McKenney M, Elkbuli A. Implications of the United States Medical Licensing Examination Step 1 examination transition to pass/fail on medical students education and future career opportunities. Am Surg. 2021;87:1196–1202. [DOI] [PubMed] [Google Scholar]
- 7.Irwin TJ, Friedrich JB, Nguyen VT, Ko JH, Phillips BT, Eberlin KR. Assessment of plastic surgery residency applications without United States Medical Licensing Examination Step 1 scores. Plast Reconstr Surg. 2021;148:219–223. [DOI] [PubMed] [Google Scholar]
- 8.Markham TH, de Haan JB, Guzman-Reyes S, Brollier LD, Campbell AN, Pivalizza EG. Potential harm of elimination of score reporting for the United States Medical Licensing Examination Step 1 examination to anesthesiology residency selection. Anesth Analg. 2021;132:275–279. [DOI] [PubMed] [Google Scholar]
- 9.Lin GL, Nwora C, Warton L. Pass/fail score reporting for USMLE Step 1: An opportunity to redefine the transition to residency together. Acad Med. 2020;95:1308–1311. [DOI] [PubMed] [Google Scholar]
- 10.Vinagre R, Tanaka P, Park YS, Macario A. Red flags, geography, exam scores, and other factors used by program directors in determining which applicants are offered an interview for anesthesiology residency. Cureus. 2020;12:e11550. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Bird JB, Friedman KA, Arayssi T, Olvet DM, Conigliaro RL, Brenner JM. Review of the medical student performance evaluation: Analysis of the end-users’ perspective across the specialties. Med Educ Online. 2021;26:1876315. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Tamakuwala S, Dean J, Kramer KJ, et al. Potential impact of pass/fail scores on USMLE Step 1: Predictors of excellence in obstetrics and gynecology residency training. J Med Educ Curric Dev. 2021;8:23821205211037444. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Russo RA, Hameed U, Ibrahim Y, et al. Psychiatry residency directors’ attitudes toward and uses of the medical student performance evaluation and other potential tools for residency selection. Acad Psychiatry. 2022;46:622–626. [DOI] [PubMed] [Google Scholar]
- 14.Goshtasbi K, Abouzari M, Tjoa T, Malekzadeh S, Bhandarkar ND. The effects of pass/fail USMLE Step 1 scoring on the otolaryngology residency application process. Laryngoscope. 2021;131:E738–E743. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Radabaugh CL, Hawkins RE, Welcher CM, et al. Beyond the United States Medical Licensing Examination score: Assessing competence for entering residency. Acad Med. 2019;94:983–989. [DOI] [PubMed] [Google Scholar]
- 16.Westerman ME, Boe C, Bole R, et al. Evaluation of medical school grading variability in the United States: Are all honors the same? Acad Med. 2019;94:1939–1945. [DOI] [PubMed] [Google Scholar]
- 17.Nemani VM, Park C, Nawabi DH. What makes a “great resident”: The resident perspective. Curr Rev Musculoskelet Med. 2014;7:164–167. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.National Resident Matching Program. Results of the 2021 NRMP Program Director Survey. https://www.nrmp.org/wp-content/uploads/2021/11/2021-PD-Survey-Report-for-WWW.pdf. Accessed April 1, 2022.
- 19.Ha TC, Ng S, Chen C, et al. Inclination towards research and the pursuit of a research career among medical students: An international cohort study. BMC Med Educ. 2018;18:86. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Association of American Medical Colleges. Medical School Graduation Questionnaire: 2021 All Schools Summary Report. https://www.aamc.org/media/55736/download. Accessed May 1, 2022.
- 21.Chang Y, Ramnanan CJ. A review of literature on medical students and scholarly research: Experiences, attitudes, and outcomes. Acad Med. 2015;90:1162–1173. [DOI] [PubMed] [Google Scholar]
- 22.National Resident Matching Program. Charting Outcomes in the Match: Senior Students of U.S. MD Medical Schools. 2nd ed. https://www.nrmp.org/wp-content/uploads/2021/08/Charting-Outcomes-in-the-Match-2020_MD-Senior_final.pdf. Accessed March 1, 2022. [Google Scholar]
- 23.Conroy MB, Shaffiey S, Jones S, et al. Scholarly research projects benefit medical students’ research productivity and residency choice: Outcomes from the University of Pittsburgh School of Medicine. Acad Med. 2018;93:1727–1731. [DOI] [PubMed] [Google Scholar]
- 24.Mehta K, Sinno S, Thanik V, Weichman K, Janis JE, Patel A. Matching into integrated plastic surgery: The value of research fellowships. Plast Reconstr Surg. 2019;143:640–645. [DOI] [PubMed] [Google Scholar]
- 25.Toci GR, Elsner JA, Bigelow BF, Bryant BR, LaPorte DM. Medical student research productivity: Which variables are associated with matching to a highly ranked orthopaedic residency program? J Surg Educ. 2021;78:512–518. [DOI] [PubMed] [Google Scholar]
- 26.Areephanthu CJ, Bole R, Stratton T, Kelly TH, Starnes CP, Sawaya BP. Impact of professional student mentored research fellowship on medical education and academic medicine career path: Professional student mentored research fellowship. Clin Transl Sci. 2015;8:479–483. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Radville L, Aldous A, Arnold J, Hall AK. Outcomes from an elective medical student research scholarly concentration program. J Investig Med. 2019;67:1018–1023. [DOI] [PubMed] [Google Scholar]
- 28.Weissbart SJ, Stock JA, Wein AJ. Program directors’ criteria for selection into urology residency. Urology. 2015;85:731–736. [DOI] [PubMed] [Google Scholar]
- 29.National Resident Matching Program. 2022 Main Residency Match by the Numbers. https://www.nrmp.org/wp-content/uploads/2022/03/2022-Match-by-the-Numbers-FINAL.pdf. Accessed May 1, 2022.
- 30.The Coalition for Physician Accountability’s Undergraduate Medical Education-Graduate Medical Education Review Committee (UGRC). Recommendations for comprehensive improvement of the UME-GME transition. https://physicianaccountability.org/wp-content/uploads/2021/08/UGRC-Coalition-Report-FINAL.pdf. Accessed June 1, 2022.
- 31.Manera K, Hanson CS, Gutman T, Tong A. Consensus methods: Nominal group technique. In: Liamputtong P, ed. Handbook of Research Methods in Health Social Sciences. Singapore: Springer Singapore; 2019:737–750. [Google Scholar]
- 32.National Resident Matching Program. Results and Data: 2021 Main Residency Match. https://www.nrmp.org/wp-content/uploads/2021/08/MRM-Results_and-Data_2021.pdf. Accessed September 1, 2021.
- 33.SF Match: Residency and Fellowship Matching Services. Ophthalmology Residency Match Summary Report 2021. https://059987482848-shared-prod.s3.amazonaws.com/Specialties/OPHTH-R/2021+Ophthalmology+Residency+Match+Summary+Report.pdf. Accessed April 3, 2023.
- 34.American Urological Association. Urology Residency Match Statistics. https://www.auanet.org/education/auauniversity/for-residents/urology-and-specialty-matches/urology-match-results. Accessed September 1, 2021. [No longer available.]
- 35.DiBiase RM, Beach MC, Carrese JA, et al. A medical student scholarly concentrations program: Scholarly self-efficacy and impact on future research activities. Med Educ Online. 2020;25:1786210. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Black ML, Curran MC, Golshan S, et al. Summer research training for medical students: Impact on research self-efficacy. Clin Transl Sci. 2013;6:487–489. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Wolfson RK, Alberson K, McGinty M, Schwanz K, Dickins K, Arora VM. The impact of a scholarly concentration program on student interest in career-long research: A longitudinal study. Acad Med. 2017;92:1196–1203. [DOI] [PubMed] [Google Scholar]
- 38.Cornett M, Palermo C, Wallace MJ, Diug B, Ward B. A realist review of scholarly experiences in medical education. Med Educ. 2021;55:159–166. [DOI] [PubMed] [Google Scholar]
- 39.Alberson K, Arora VM, Zier K, Wolfson RK. Goals of medical students participating in scholarly concentration programmes. Med Educ. 2017;51:852–860. [DOI] [PubMed] [Google Scholar]
- 40.Bierer SB, Chen HC. How to measure success: The impact of scholarly concentrations on students—A literature review. Acad Med. 2010;85:438–452. [DOI] [PubMed] [Google Scholar]
- 41.Bierer SB, Prayson RA, Dannefer EF. Association of research self-efficacy with medical student career interests, specialization, and scholarship: A case study. Adv Health Sci Educ. 2015;20:339–354. [DOI] [PubMed] [Google Scholar]
- 42.Corliss SB, Abrams S, Cox S, Nelson EA. Learning new skills in innovation, leadership, and discovery during a 9-month scholarly concentration: Results from the first cohort at a new medical school. Med Sci Educ. 2021;31:331–336. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Burk-Rafel J, Mullan PB, Wagenschutz H, Pulst-Korenberg A, Skye E, Davis MM. Scholarly concentration program development: A generalizable, data-driven approach. Acad Med. 2016;91(11 Suppl):S16–S23. [DOI] [PubMed] [Google Scholar]
- 44.Wolfson RK, Arora VM. More on promoting medical student scholarly research. Acad Med. 2016;91:159. [DOI] [PubMed] [Google Scholar]
- 45.Jeffe DB, Yan Y, Andriole DA. Do research activities during college, medical school, and residency mediate racial/ethnic disparities in full-time faculty appointments at U.S. medical schools? Acad Med. 2012;87:1582–1593. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Pontell ME, Makhoul AT, Ganesh Kumar N, Drolet BC. The change of USMLE Step 1 to pass/fail: Perspectives of the surgery program director. J Surg Educ. 2021;78:91–98. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Blamoun J, Hakemi A, Armstead T. Perspectives on transitioning Step 1 of the United States Medical Licensing Examination to a pass/fail scoring model: Defining new frameworks for medical students applying for residency. Adv Med Educ Pract. 2021;12:149–154. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Mun F, Scott AR, Cui D, Chisty A, Hennrikus WL, Hennrikus EF. Internal medicine residency program director perceptions of USMLE Step 1 pass/fail scoring: A cross-sectional survey. Medicine (Baltim). 2021;100:e25284. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Kay HG, Makhoul AT, Kumar NG, Pontell ME, Drolet BC, Whigham AS. Otolaryngology program director, house-staff, and student opinions: Step 1 pass/fail score reporting. Ann Otol Rhinol Laryngol. 2023;132:63–68. [DOI] [PubMed] [Google Scholar]
- 50.Woroniecka K. Bracing for change: The escalating importance of USMLE Step 2. https://www.kevinmd.com/2021/06/bracing-for-change-the-escalating-importance-of-usmle-step-2.html. Accessed May 29, 2022.
- 51.Delnegro D. Step 2 percentiles: How to understand & interpret your scores in a Step 1 pass-fail world. https://blog.blueprintprep.com/medical/a-new-look-at-your-usmle-step-2-score. Accessed May 29, 2022.
- 52.Chretien K. The Step 1 exam is going pass-fail: Now what? https://www.aamc.org/news-insights/step-1-exam-going-pass-fail-now-what. Accessed May 29, 2022.
- 53.White-Dzuro CG, Makhoul AT, Pontell ME, Stephens BF, Drolet BC, Abtahi AM. Perspectives of orthopedic surgery program directors on the USMLE Step 1 scoring change. Orthopedics. 2022;45:e257–e262. [DOI] [PubMed] [Google Scholar]
- 54.Glassman G, Black J, McCoin N, Drolet B. Emergency medicine program directors’ perspectives on changes to step 1 scoring: Does it help or hurt applicants? West J Emerg Med. 2021;23:15–19. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Ganesh Kumar N, Makhoul AT, Pontell ME, Drolet BC. Characterizing the effect of pass/fail U.S. Medical Licensing Examination Step 1 scoring in neurosurgery: Program directors’ perspectives. World Neurosurg. 2020;142:e440–e444. [DOI] [PubMed] [Google Scholar]
- 56.Chisholm LP, Drolet BC. USMLE Step 1 scoring changes and the urology residency application process: Program directors’ perspectives. Urology. 2020;145:79–82. [DOI] [PubMed] [Google Scholar]
- 57.Aziz H, Khan S, Rocque B, Javed MU, Sullivan ME, Cooper JT. Selecting the next generation of surgeons: General surgery program directors and coordinators perspective on USMLE changes and holistic approach. World J Surg. 2021;45:3258–3265. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Wadhwa H, Shah SS, Shan J, et al. The neurosurgery applicant’s “arms race”: Analysis of medical student publication in the Neurosurgery Residency Match. J Neurosurg. 2020;133:1913–1921. [DOI] [PubMed] [Google Scholar]
- 59.Hasley HL, Megalla M, Bukowiec LG, Carlson SR. Medical student research productivity and matching to orthopedic residency programs. J Surg Educ. 2022;79:1–2. [DOI] [PubMed] [Google Scholar]
- 60.Schrock JB, Kraeutler MJ, Dayton MR, McCarty EC. A comparison of matched and unmatched orthopaedic surgery residency applicants from 2006 to 2014: Data from the National Resident Matching Program. J Bone Jt Surg. 2017;99:e1. [DOI] [PubMed] [Google Scholar]
- 61.Bigelow BF, Siegel N, Toci GR, Elsner JA, Hicks CW, Abularrage CJ. Bibliometric review of medical student research before matching integrated vascular surgery. J Surg Res. 2021;263:251–257. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 62.Thangamathesvaran L, Patel N M, Siddiqui SH, et al. The otolaryngology match: A bibliometric analysis of 222 first-year residents: Bibliometric analysis of 222 ENT interns. Laryngoscope. 2019;129:1561–1566. [DOI] [PubMed] [Google Scholar]
- 63.Ngaage LM, Ge S, Gao C, et al. Trends in the academic credentials of matched dermatology residency applicants. Cureus. 2020;12:e12411. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 64.Ngaage LM, Elegbede A, McGlone KL, et al. Integrated plastic surgery match: Trends in research productivity of successful candidates. Plast Reconstr Surg. 2020;146:193–201. [DOI] [PubMed] [Google Scholar]
- 65.Sawarynski KE, Baxa DM, Folberg R. Embarking on a journey of discovery: Developing transitional skill sets through a scholarly concentration program. Teach Learn Med. 2019;31:195–206. [DOI] [PubMed] [Google Scholar]
- 66.The InCUS Planning Committee. Summary report and preliminary recommendations from the Invitational Conference on USMLE Scoring (InCUS), March 11-12, 2019. https://www.usmle.org/sites/default/files/2021-08/incus_summary_report.pdf. Accessed June 1, 2022.