Analysis of Surgery Residency Website Content: Implications during the COVID-19 Era

Abstract

Background

Surgery residency program websites (SRW) are an important source of information for prospective applicants. The COVID-19 pandemic spurred a pivot from the traditional in-person interview format to interviews via virtual platforms. Because of the inability to meet in person, the information provided on program websites takes on an increased relevance to applicants. We hypothesized that SRW may be missing content important to applicants. Our study aims to assess SRW for the content which impacts the applicant decision-making process.

Methods

An internal survey distributed to fourth-year medical students in 2020 at a single academic institution identified the website content most important to applicants. A list of ACGME-accredited SRW as of December 1, 2020 was obtained. Using the Fellowship and Residency Electronic and Interactive Database, websites were assessed for content parameters identified by the survey.

Results

Medical students applying to surgical specialties identified fellowship acquisition (94%), faculty information (88%), application contact information (82%), and resident wellness (77%) as the most important website content. Review of SRW websites identified content pertaining to fellowship acquisition and resident wellness in only 60% and 27% of cases respectively. Overall, the SRW of university programs included the most content parameters, followed by hybrid programs, then community programs.

Conclusions

Many SRW are missing information that applicants deem important in their decision-making process. Most notably, there is a relative deficiency in information pertaining to fellowship match results and resident wellness. University based programs tend to include more of this information on their websites. SRW should continue to adapt to meet the needs of applicants in an increasingly virtual age.

INTRODUCTION

General surgery residency program websites (SRW) are a vital tool for applicants to gain information regarding specific programs. A variety of studies have shown that general surgery residency websites lack integral pieces of information which would be important decision-making factors for applicants.1, 2, 3 Some programs may be unaware that their websites are deficient in needed information, while others may rely on the interview day to provide information such as facility tours, introduction to faculty, curriculum, and rotation schedules.¹ The realities of the COVID-19 pandemic have spurred the ACGME to recommend that interviews be conducted using virtual platforms.⁴ This change, which may be the preferred approach for the foreseeable future, poses challenges to applicants seeking information about surgical programs and further increases the importance of the content and accessibility of residency websites.

The information provided on SRW may influence applicants’ decisions on where to apply, interview, and how to formulate their rank list. Although SRW have been assessed in the past, the rapid virtualization of the application process driven by the coronavirus pandemic has reinforced the importance of comprehensive online program information. In addition, there are more SRW available each year, further necessitating a reassessment of content. We hypothesized that SRW may be missing content important to applicants. The purpose of our study is to assess various SRW for the content prospective applicants have identified as key factors in their decision-making process. Additionally, we assessed SRW content based on program type and geographical location.

METHODS

An internal survey distributed to fourth-year medical students in 2020 at a single academic institution identified the website content most important to residency applicants. The applicants’ preferences for the most important (Likert scale 4 and 5) and least important (Likert scale 1 and 2) residency website contents were analyzed. A list of Accreditation Council for Graduate Medical Education (ACGME) accredited SRW was generated using the ERAS directory as of December 1, 2020. Using the Fellowship and Residency Electronic and Interactive Database (FREIDA), matching SRW were identified, accessed, and assessed for content parameters identified by the survey.

Websites were assessed for presence of contact information: address, phone, and email, program description, program director name, list of current residents, and alumni destinations/fellowship match list. Other content assessed included information on didactics, journal club, rotation schedules, educational support, wellness opportunities, research requirements, research output, diversity and inclusion, underrepresented minorities, and gender. Lastly, recruitment information was assessed such as inclusion of an informational program video, virtual tours, and whether a USMLE STEP 2 score is required at time of application.

Program type was based on the FREIDA website's program listing. University-based was designated as academic, community-based was designated as community, and community-based university affiliated was designated as mixed. Military-based SRW were not assessed by our study. Student t-test/ANOVA and Chi-square/Fisher's exact tests were used to compare website content based on program type (academic, community, mixed).

RESULTS

Of 169 medical students, 91 (54%) participated in the survey. Medical students identified resident wellness (88%), fellowship acquisition (83%), residency location (83%), faculty information (82%), and board pass rates (82%) as the top 5 most important aspects of website content. Subgroup analysis of applicants to surgical specialties (Table 1 ) identified fellowship acquisition (91%), faculty information (91%), resident information including names and photos (81%), application contact information (76%), residency location (76%), and resident wellness (76%) as their most important content areas (Table 1).

TABLE 1.

Surgical Specialties’ Top 6 Most Important Categories Compared to Nonsurgical Specialties

Categories	Surgical Specialties (n = 21)	All Other Specialties (n =70)
Fellowship Acquisition	91%	83%
Faculty Information	91%	82%
Resident Information - Names and Photos	81%	71%
Resident Wellness	76%	88%
Application Contact	76%	82%
Residency Location	76%	83%

Of the 319 programs listed in the ERAS directory, 315 (99%) websites were accessible by links provided by FREIDA. The southern region had the most surgical residency programs (100), as compared to 90 in the Northeast (NE), 78 in the Midwest (MW), and 46 in the West. Website content criteria is summarized in Table 2 . The list of content parameters was created by combining the most common parameters from several prior studies. We also included content parameters addressed by our survey which were not previously addressed in prior studies.1, 2, 3 An average of 12.8 (49%) of 26 items were found among SRW. Program description (100%) and faculty information such as program director name (95%) were the most readily available. Alumni destinations/fellowship match lists were listed on 60% of websites. One hundred and thirty-four SRW (43%) had a program overview video with an average video length of 5 minutes 25 seconds. We also assessed websites for a few content parameters which were not included in our survey but have been addressed by past studies including information on underrepresented minorities (16%), gender diversity (14%), and international opportunities (13%), sub-internship description (13%), virtual tour (10%), and virtual sub-internship availability (1%). These parameters have been included in prior articles and were included in our study for comparison's sake.

TABLE 2.

Overall SRW Content Summarized

Program Website Detail	Number of websites (n,%) N = 315
Residency Location	249 (79%)
Application Contact	268 (85%)
Program Description	315 (100%)
Faculty Information	298 (95%)
Resident Names/Photos	269 (85%)
Fellowship Acquisition	188 (60%)
Didactics	247 (78%)
Journal Club	159 (51%)
Rotation Schedule	227 (72%)
Clinic Responsibilities	109 (34%)
Call Responsibilities	56 (18%)
Educational Support	128 (41%)
Meetings/Conferences/Courses	243 (77%)
International Opportunities	41 (13%)
Address URM	50 (16%)
Address Gender	45 (14%)
Wellness	84 (27%)
Research Requirement	234 (74%)
Research Output	71 (23%)
Research Support	126 (40%)
Program Video	134 (43%)
Video length (min:sec)	5:25
Virtual Tour	32 (10%)
Virtual Sub-I	2 (1%)
Sub-I description	41 (13%)
STEP 2	167 (53%)

When accounting for the most important factors identified by surgical applicants, 188 (60%) SRW had information on alumni destinations and/or fellowship acquisition, 84 (27%) had information on resident wellness, 298 SRW had faculty information (95%), 269 had resident names and photos (85%), 268 had application contact (85%), and 249 had residency location (79%).

Direct comparisons of SRW content based on program type is summarized in Table 3 . When comparing Academic programs and Community programs, Academic programs had more instances of alumni destinations/fellowship match list (79% vs 33%, p < 0.001), wellness (37% vs 18%, p < 0.001), current residents (92% vs 79%, p = 0.02), application contact (90% vs 73%, p = 0.003), addressing under-represented minorities (URM) in medicine (35% vs 3%, p < 0.001), addressing gender (33% vs 3%, p < 0.001), clinic responsibilities (48% vs 21%, p < 0.001), international opportunities (27% vs 2%, p < 0.001), research requirements (90% vs 29%, p < 0.001), and research support (61% vs 21%, p < 0.001). When comparing Academic programs and Hybrid programs, Academic programs had more instances of alumni destinations/fellowship match list (79% vs 57%, p < 0.001), wellness (37% vs 22%, p = 0.012), current residents (92% vs 83%, p = 0.04), addressing URM (35% vs 6%, p < 0.001), addressing gender (33% vs 4%, p < 0.001), residency location (88% vs 73%, p = 0.007), clinic responsibilities (48% vs 30%, p = 0.006), international opportunities (27% vs 7%, p < 0.001), research requirements (90% vs 68%, p < 0.001), research support (61% vs 32%, p < 0.001), program video (50% vs 37%, p = 0.02). When comparing Hybrid programs and Community programs, Hybrid programs had more instances of application contact (87% vs 73%, p = 0.03), and alumni destinations/fellowship match list (57% vs 33%, p = 0.002).

TABLE 3.

Comparison of SRW Content Based on Program Type Summarized

				Chi-square/Fisher's Exact Test Tables
Program Website Detail	Number of Academic websites (n,%) N = 113	Number of Community websites (n,%) N = 67	Number of Hybrid Websites (n,%) N = 135	Academic vs Community p-value	Academic vs Hybrid p-value	Community vs Hybrid p-value
Residency Location	99 (88%)	51 (76%)	99 (73%)	0.061	0.0066	0.734
Application Contact	102 (90%)	49 (73%)	117 (87%)	0.003	0.431	0.03
Program Description	113 (100%)	67 (100%)	135 (100%)	1	1	1
Faculty Information	110 (97%)	61 (91%)	127 (94%)	0.08	0.354	0.557
Resident Names/Photos	104 (92%)	53 (79%)	112 (83%)	0.019	0.037	0.563
Fellowship Acquisition	89 (79%)	22 (33%)	77 (57%)	<0.001	<0.001	0.002
Didactics	94 (83%)	47 (70%)	106 (79%)	0.062	0.425	0.223
Journal Club	45 (40%)	36 (54%)	78 (58%)	0.088	0.005	0.652
Rotation Schedule	78 (69%)	48 (72%)	101 (75%)	0.74	0.323	0.734
Clinic Responsibilities	54 (48%)	14 (21%)	41 (30%)	<0.001	0.006	0.181
Call Responsibilities	26 (23%)	9 (13%)	21 (16%)	0.125	0.146	0.834
Educational Support	45 (40%)	28 (42%)	55 (41%)	0.875	0.897	1
Meetings/ Conferences/Courses	88 (78%)	48 (72%)	107 (79%)	0.373	0.877	0.289
International Opportunities	30 (27%)	1 (2%)	10 (7%)	<0.001	<0.001	0.105
Address URM	40 (35%)	2 (3%)	8 (6%)	<0.001	<0.001	0.502
Address Gender	37 (33%)	2 (3%)	6 (4%)	<0.001	<0.001	1
Wellness	42 (37%)	12 (18%)	30 (22%)	0.007	0.012	0.582
Research Requirement	102 (90%)	38 (29%)	92 (68%)	<0.001	<0.001	0.121
Research Output	29 (26%)	12 (18%)	30 (22%)	0.272	0.552	0.582
Research Support	69 (61%)	14 (21%)	43 (32%)	<0.001	<0.001	0.135
Program Video	57 (50%)	27 (40%)	50 (37%)	0.218	0.04	0.759
Virtual Tour	17 (15%)	7 (10%)	8 (6%)	0.5	0.02	0.264
Virtual Sub-I	0 (0%)	0 (0%)	2 (1%)	1	0.502	1
Sub-I description	14 (12%)	12 (18%)	15 (11%)	0.381	0.844	0.193
STEP 2	57 (50%)	34 (51%)	76 (56%)	1	0.373	0.549

Bold website details in the first column indicate the top website content but residency applicants.

DISCUSSION

SRW are historically the most important source of information for prospective applicants.1, 2, 3, 4, 5 The COVID-19 pandemic has increased the importance of SRW because of the inability to gain information during in-person interviews, tours, or second-look opportunities. We found that many SRW were lacking content important to applicants. An average of 12.9 of 26 content items (49%), were available to applicants accessing SRW.

Fellowship acquisition and resident wellness were identified as among the most important content parameters by applicants to both surgical and non-surgical specialties. Our results are similar to a national survey on factors and influences that determine choices of surgery residency applicants who identified fellowship acquisition as being an important factor for applicants.⁵ Despite the importance placed on these parameters by applicants, information pertaining to alumni destination/fellowship acquisition and resident wellness were present in only 60% and 27% of SRW, respectively. Our finding that 40% of SRW do not contain information about the activities of their matriculating residents represents a missed opportunity to provide content that is desired by applicants. Previous studies have shown that content pertaining to resident wellness was provided on fewer than 25% of websites.^{1 , 6} In comparison, we found that about a third of SRW have any information available pertaining to resident wellness. Our finding serves to reinforce the existing literature which has described the dearth of information available on the topic. This trend continues to highlight the need for programs to update their content as we move towards increased reliance on digital content.

Previous studies have also demonstrated the link between diversity and inclusion, and resident wellness.⁷ The vast majority of SRW lack specific statements addressing diversity and inclusion, especially under-represented minorities, and gender. We found that few SRW feature statements on gender (14%) or URM diversity (16%), consistent with the study by Dreisen et al. which found that of 242 non-military SRW, 19% included a diversity and inclusion message.⁸ Our study included 73 more SRW so it is possible that the newer websites do not feature an inclusion message or statement, which would explain a slightly decreased percentage in gender/URM diversity statements. Our data also suggests that SRW have remained stagnant in expanding upon inclusion possibilities. Further study is needed on the importance of these messages to applicants and whether this content influences their decisions on how to choose to apply, interview, and rank programs.

We found that academic SRW contained a higher average number of content parameters and were significantly more likely to contain information pertaining to alumni destinations and/or fellowship match lists and resident wellness than community or mixed programs. A previous survey by Jarman et al. found that applicants give preference to university programs rather than community programs when forming their rank lists.⁵ Since fellowship and resident wellness information is important to applicants, this discrepancy may further accentuate the bias toward university programs. Hybrid programs published more alumni destinations and/or fellowship match lists than community programs but still significantly less than academic programs. While not comprehensive, the list of content parameters provided in Tables 1 and 2 may serve as guide for program directors and administrators working to update SRW content.

There are several limitations to our study. From June 2019 to January 2020, there were 242 accessible SRW listed as members on the APDS website out of 304 programs participating in The Match.⁸ Our study found 317 SRW, out of 319 programs, showing a large increase in the amount of SRW in just 1 year. This surge in SRW makes comparing data and drawing conclusions between studies difficult since it is not possible to discern between newly created websites in our study from the pre-COVID-19 websites in older studies. The increased number of available SRW may be related to programs identifying the need for an online presence. Website design and difficulty in navigation was not considered, which may have caused an unequal distribution of time spent on 1 website over another. It is possible that SRW contents were missed on review despite thorough search. SRW usability and accessibility may be a future area for research. Another potential limitation is the small sample size of our survey of medical students. Of the class of 161 students, 91 participated in the survey with 21 of them planning to apply to a surgical specialty. Of those 21 students, only 6 were planning to apply to General Surgery. The opinions of students from a single institution may not be representative of the entire applicant pool across all regions and training programs. Finally, no questions addressing the importance of diversity and inclusion were included in the survey of medical students. Including specific questions about diversity and inclusion would have helped clarify whether this content influences how applicants choose to rank programs. A survey of applicants and program directors from multiple institutions, regions, and intended specialties would help to answer many of these questions and provide insight into any discrepancy in content importance that may exist between program leadership and applicants.

CONCLUSION

SRW are inconsistently providing the information deemed important by applicants. There is a deficiency in the important content areas of fellowship match data and resident wellness. Furthermore, linked to wellness, there is a paucity of URM, diversity, and gender content on SRW. Academic SRW tend to provide the most information to applicants, followed by hybrid programs, then community programs. Both the increased number of training programs available to applicants and the inability to interview in-person, serve to heighten the importance of the role played by SRW. SRW should adapt to meet the needs of applicants in an increasingly virtual age.