Practical Tips for Undergraduate Medical Education Advisors in Residency Application Signaling

Abstract

Objectives

The residency application process has become increasingly complex for medical students and advisors to navigate. Program signaling was piloted to improve applicants’ abilities to obtain interview offers at programs they were strongly interested in. The initial positive results led to expansion of signaling to additional specialties over the next two application cycles. Despite the benefits of program signaling, the variation in signaling practices among specialties has presented challenges for both advisors and students when determining how to best allocate signals. The aim of this study is to identify students’ perceptions of the signaling process, how this may impact outcomes, and to guide future educational programming.

Methods

This is an exploratory original survey study of students in a US allopathic medical school applying in ERAS for the 2023 residency cycle. The survey was developed to determine students’ understanding of how programs would use signals in the application process and assess strategies students used to allocate signals. We compared program signals to student interview offers and match outcomes using descriptive statistics.

Results

57 of 96 eligible students completed the survey. 51% signaled a range of programs based on their perceived competitiveness for the program while 40% signaled programs of interest regardless of perceived competitiveness. 53% of students thought sending a signal would increase their chance of an interview, while 42% were unsure how the signal would be used by residency programs. Students received interviews at 49% of the programs signaled, which increased to 56.5% when specialties offering more than 7 signals were excluded. 35% of students matched at a signaled program.

Conclusions

Students’ perceptions and strategies related to the signaling process are varied and may impact interview offers. Advisors should monitor and review internal institutional trends to help inform future educational programming to optimize signal allocation for their students.

Introduction

The residency application process has become increasingly complex for medical students and advisors to navigate.^1-5 The introduction of USMLE Step 1 pass fail scoring in 2022 and virtual program interviews have led to changes in the residency application review and selection process for graduate medical education programs.^6-9 Factors including student uncertainty in matching, ease of applying, and virtual interviews have led to applicants increasing the number of programs they apply to.^10-14 Program signaling was initially piloted by Otolaryngology in 2020 to improve applicants’ abilities to obtain interview offers at programs in which they showed strong interest.^15-17 Initial results showed high student and program satisfaction with signaling, increased ability to receive interview offers at programs of interest and improved distribution of interview offers. ¹⁶ Additional studies demonstrated the increased likelihood of interview offers persisted among categories of gender and Under Represented in Medicine (URM) students. ¹⁷ Based on these positive results, signaling expanded the following year to include additional specialties both within Electronic Residency Application Service (ERAS) and outside of ERAS. In the third year of implementation, multiple specialties were included in ERAS signaling. Signals ranged between two to 30 with one specialty additionally offering a tiered approach with gold and silver signals. ¹⁸ Despite the benefits of program signaling, the variation in signaling practices among specialties has presented challenges for both advisors and students when determining how to best allocate signals. ¹⁹ While most data has been focused on program director perceptions of signaling, data is limited as to student interpretation of the signaling process. Therefore, we aimed to (1) explore students’ perceptions of how programs may use signals in the application process and (2) categorize the strategies students used to allocate signals.

Methods

We conducted an exploratory original survey study of fourth year medical students (MD4) at the Geisinger Commonwealth School of Medicine (GCSOM), a US allopathic medical school, who applied in the 2023 ERAS cycle. We used convenience sampling to identify study participants. Inclusion criteria were GCSOM MD4 students applying to specialties participating in signaling in the supplemental ERAS application and who authorized interview release information in ERAS. Students applying to specialties that did not participate in ERAS signaling or did not release interview information were excluded. The study period was August 29, 2022, through November 3, 2022.

The survey was created by the GCSOM career advising department and reviewed by a core faculty member from each of two different residency programs for face validity. The survey was administered in Qualtrics after students submitted their ERAS and supplemental ERAS applications. All eligible students received an email invitation to complete the survey. The survey introduction included that participation was voluntary, responses were confidential, and survey completion provided consent. This study was determined to be exempt by the Geisinger Institutional Review Board (#0000-8345).

We collected signaling data for students in participating specialties and student perception data. Students were asked to select their specialty and the programs they signaled. Program codes were obtained from ERAS and students selected the exact program name with unique code. Interviews offered outside ERAS were manually entered by each student. Match data was provided by the Office of Student Affairs. Descriptive statistics were used to describe the data. We compared signals sent with interview offers received and final match outcomes.

Results

Fifty-seven students (59%) completed the survey signaling perception questions. Fifty-one percent of students signaled a range of programs to include a combination of reach programs and others they felt they were competitive for, 9% only signaled programs they thought they were competitive for, and 40% signaled programs they were most interested in regardless of self-perceived competitiveness (Figure 1). Residency programs and applicants were educated about the goals of program signaling by ERAS, specifically the use of signals for deciding whom to interview and not post-interview ranking decisions. We aimed to assess student's perceived understanding of the signaling goals so any gaps in knowledge could be addressed through additional educational programming. Fifty-three percent of students thought signaling would increase their chance of receiving an interview offer; 42% were unsure how programs would interpret signaling. Only 5% of students thought signaling would increase their chance of being ranked highly by a signaled program (Figure 2).

Figure 1.

Strategies used by students when identifying programs to signal.

Figure 2.

Student interpretation of how residency programs will use signaling.

Complete signaling data were available for 48% (n = 46) of eligible students. Collectively, students received interview offers at 49% of the programs signaled. Interview offers increased to 56.5% when specialties offering more than 7 signals were excluded. Approximately 35% of participating students matched at a signaled program.

Discussion

Program signaling was piloted in 2020 to improve applicants abilities to obtain interview offers at programs they were strongly interested in.^15-17 The initial positive results led to the expansion of signaling to additional specialties over the next two application cycles with varying number of signals per specialty and the introduction of a tiered signaling approach. These ongoing changes present opportunities and challenges. Potential benefits include the specialties’ abilities to adjust their approaches in an evidence-based manner. This may also benefit students as they can signal more strategically based on their individual applicant profiles. Despite these potential benefits, specialty specific signaling may present challenges for undergraduate medical education (UME) advisors who are often responsible for career counseling for all specialties. Our results highlight these challenges and present areas of opportunity to guide future educational programming to help UME advisors counsel students. Based on our findings, we offer the following practical tips for advisors supporting students throughout the signaling process (Table 1).

Table 1.

Practical tips for advisors supporting students throughout the signaling process.

Practical Tips for Advisors
1. Ensure signaling education for advisors
2. Optimize signaling education for students
3. Track and review institutional data
4. Review specialties with high signal numbers

Practical tip #1: ensure signaling education for advisors

Specialty specific recommendations are updated annually as new resources become available. Advisors must remain up to date on specialty specific information to provide accurate guidance to students. Students must understand the competitiveness of individual programs in their specialty of interest and their competitiveness for both specialty and program match. Career advisors need to evaluate a student's competitiveness and understand the resources to use in determining their competitiveness. The resources utilized must be shared with everyone advising students to ensure consistent messaging. All individuals offering formal career advising must engage in continuous professional development around the signaling and residency application process, which is critical for students’ match outcomes.

Practical tip #2: optimize signaling education for students

Advisors should ensure that all students understand the signaling process and how it might impact interview offers. The Assistant Dean of Student Affairs held a large group didactic session to educate GCSOM students about the signaling process. Students also received invitations to join the ERAS supplemental application webinars hosted by ERAS. Lastly, students had the opportunity to meet with their career coach and specialty advisors to discuss their signal allocations. All of these were optional sessions. Despite these initiatives, students had widely varied strategies for signaling and different interpretations about how programs might use signals. These findings support the need for required programming or required career advising meetings to review student signals, regardless of specialty interest.

Practical tip #3: track and review institutional data

Specialty level data is collected and shared by ERAS and is important to guide future signaling practices. Medical schools should also track their individual signal to interview conversion for each specialty. This information can help inform advisors about variations in specialties and may guide future professional development activities. Tracking program specific signal to interview conversion trends can also be used to help guide students during future application cycles. For example, many of our students signaled geographically distanced programs that did not yield an interview. Having school specific data in addition to ERAS data is necessary to develop a comprehensive strategy for signal allocation.

Practical tip #4: review specialties with high signal numbers

Reviewing signaling outcomes from specialties with a high number of signals, ie, orthopedic surgery, dermatology, urology, otolaryngology, neurosurgery, and obstetrics and gynecology, is critically important. Typically, specialties with the highest numbers of allotted signals are among the most competitive. Prior studies have demonstrated a benefit to the high signal approach leading to a more even distribution of interview offers among applicants. ²⁰ Despite this potential benefit, students need to be strategic when allocating their signals as it has been suggested specialties may only offer interview invitations to those who signaled. Advisor-student pairs should collectively ensure students signal programs that closely align with their competitiveness, as determined by their advisor. Students may apply to programs based on institution name and reputation rather than professional goals, fit, and values alignment with the program. Thus, it is critical to ensure students signal programs that match their interests and suitability.

Limitations

This pilot study has several limitations. This was a single center study with a small sample size. Participants were identified through convenience sampling and a power analysis was not performed. Survey questions were not pilot tested for construct validity and were reviewed for face validity only with two core faculty members from different residency programs. In the 2023 ERAS cycle, UME career advisors did not have access to signaling information. Students selected program signals sent when completing the survey and reported interview offers they received outside of ERAS, potentially introducing recall bias or missing data. Another limitation is the lack of a comparison group for interview offer rates from non-signaled programs. Interpretation of signaling to interview conversion for home and away rotations was limited because all students received home interview offers, and away rotation signaling recommendations varied by program and specialty. Our results should be taken into context given these limitations and confounders.

Our work may inform future scholarship related to signal allocation and career advising. Survey question validation and study expansion to include data from a broad range of medical schools would help strengthen take away and action items. The ability to track medical school and specialty specific signaling data in ERAS may benefit advisors and students, particularly given resource variation among schools, which may limit institution specific data collection.

Conclusion

Since its initial pilot by otolaryngology, signaling research has shown clear benefits to both students and programs. Students are more likely to obtain interview offers at programs of interest.^15-17,20 Signaling improves distribution of interview offers providing a benefit to students who are least likely to obtain interview offers.^15,16 Additionally, a high-signal approach has led to a decrease in applications submitted, resulting in less financial burdens for students. ²¹ Based on these positive outcomes, program signaling is anticipated to continue and will be refined as additional data becomes available. Our results highlight the need to develop educational programming around signaling for both students and advisors and to develop processes to monitor institutional data which can inform development of future hypothesis driven research. In the interim, the practical tips shared can support UME career advisors in their efforts to ensure successful match outcomes.