Assessment of US Radiology Residency Program Websites in the COVID-19 Era

Abstract

Objective

To provide an updated evaluation of radiology residency program websites in light of virtual interviewing during the COVID-19 pandemic and encourage programs to improve the quality of their online website presence.

Methods

We evaluated the websites of 197 US radiology residency programs between November and December 2021 for the presence or absence of 30 metrics. The metrics chosen are those considered important by applicants when choosing a program and have been used in other similar papers.

Results

Of the 197 programs, 192 (97.5%) had working websites. The average radiology residency website had 16 of 30 (54%) metrics listed on their websites. Five programs did not have accessible websites and were not included in the analysis. The most comprehensive website had 29 of 30 (97%) of metrics listed and the least comprehensive website had 2 of 30 (7%). There is a statistically significant difference in website comprehensiveness between top 20 and non–top 20 radiology program websites.

Conclusion

Although radiology residency program websites have generally become more comprehensive over time, there is still room for improvement, especially in times of virtual interviews when residency applicants are becoming more and more reliant on program websites to gain essential information about a program. Some key areas to include are diversity and inclusion initiatives, resident wellness, applicant information, program benefits, and showcase of people in the program.

Visual Abstract

Introduction

The National Resident Matching Program is a 6-month process starting in September of the final year of medical school in which medical students attempt to match into a US residency program of their choice. It is vital for medical students to have complete and nuanced information about residency programs for them to make the best-informed decision about their future career. Given that the current generation of medical students significantly interfaces with online social media platforms (84% of adults aged 18-29 utilize social media), it is in the best interest for residency programs to provide thorough information on an online medium [1]. Moreover, in 2020, it was found that emergency medicine residency websites’ peak visits occurred immediately preceding their interview dates [2]. These website visits play a vital role in influencing the applicant’s perception of a program, as 78% of surveyed residency applicants reported that a program’s website had an impact on their decision to apply to a program [3]. Radiology residency leaders say that they believed the residency websites to be the most useful tool to showcase their programs [4]. However, the most recent study assessing radiology residency websites in 2016 found that only 7% of websites were deemed to have comprehensive information [5].

The urgency to improve online access to information regarding radiology residencies is further increased because of the COVID-19 pandemic. With the switch from in-person to virtual interviewing, radiology applicants have diminished in-person interactions with program residents, directors, and faculty [6]. These interactions play a vital role in shaping the medical student’s rank list [7]. Other aspects that are lost with virtual interviews include the campus tour, feel of the city, noon conference experience, and pre-interview informal dinner [8]. Radiology program directors have encouraged their programs to think of novel ways to brand themselves in this new virtual world by revamping their online presence [9,10]. Virtual interviews are likely to remain the norm given the financial benefit as well as time saved on travel for applicants [11,12]. Therefore, radiology programs must work on improving their online presence in the COVID-19 era to provide applicants with more thorough information about their future programs to ensure an optimal fit between program and applicant [13].

Two prior studies assessing radiology websites were done almost 6 years ago, both using 2016 data [5,14]. This study aims to build on prior work by (1) including new radiology residency programs since 2016, (2) tracking improvements in information comprehensiveness in radiology websites in the past 6 years, (3) specifically evaluating content that mimics the interpersonal experiences lost with the prior in-person interviewing method, and (4) providing suggestions to improve online engagement of radiology residencies. We hypothesize that with the switch to virtual interviewing, radiology program websites now have more comprehensive information for applicants as compared with 2016.

Methods

A list of 197 radiology residency programs was obtained from the Fellowship and Residency Electronic Interactive Database on November 23, 2021. The website for each program was accessed using the Fellowship and Residency Electronic Interactive Database link or through a Google search. Between November and December 2021, the websites were then evaluated on the presence or absence of 30 metrics (Table 1 ) by four independent reviewers (D.S.L., K.A.-S., H.R.S., F.K.). These 30 metrics were determined through two steps. First, we analyzed what metrics were included by prior publications that assessed website comprehensiveness in various medical specialties, including radiology [5,14, 15, 16, 17, 18]. Second, we cross-referenced articles that surveyed medical school students on what factors they look for in a program website, putting more weight on surveys done on radiology applicants [3,7,9,19,20]. Only information that can be accessed through the site was included. Information on some of the listed metrics which may be found elsewhere on online forums (eg, www.studentdoctor.net, www.doximity.com, www.reddit.com, www.auntminnie.com) were not included in our study. All 30 metrics were then classified under one of the following categories: program overview, applicant information, resident life, curriculum, or videos. We also created a sixth super-category called “our people,” which includes resident-focused videos, resident’s extracurricular interests, faculty listing, and the message from the program director or chairperson. This super-category was created due to multiple recent studies emphasizing the weight of the personability of a program (happiness of residents, interactions with residents and faculty, and program culture) when an applicant ranks a program [7,19,20]. This category is showcasing the people of the program; an experience that is notably lost in virtual interviews. Program ranking based on the 2021 to 2022 Doximity Residency Navigator was noted. t Tests of unequal variance were then performed to determine any significant differences in the comprehensiveness of the websites between the top 20 ranked versus non–top 20 programs. All statistical analyses were conducted using SPSS 23 software (IBM Corp, Armonk, New York) [21]. P values < .05 were considered statistically significant. Institutional review board approval was waived as the project’s data was based on publicly available information online.

Table 1.

Presence of 30 metrics used to evaluate radiology residency websites

Category	Metric	Number Reported Out of 192, n (%)
Program overview	Contact e-mail (PD, APD, PC)	173 (90)
	Hospital facility description	158 (82)
	Faculty listing∗	149 (78)
	Message from program director or chairperson	143 (74)
	Geographical description and recreational activities	136 (71)
	Radiology program mission statement	129 (67)
	Ongoing research or recent publications	80 (42)
	Diversity statement	65 (34)
Applicant information	Selection criteria for applicants	71 (37)
	Interview day information	71 (37)
	Q&A forum	57 (30)
Resident life	Current residents∗	169 (88)
	Academic courses (AIRP or physics)	151 (79)
	Benefits	142 (74)
	Fellowship placement	138 (72)
	Vacation	118 (61)
	Salary	114 (59)
	Meal allowance	79 (41)
	Parking and transportation	77 (40)
	Resident wellness initiatives∗	73 (38)
	Current residents’ extracurricular interest∗	68 (35)
	Moonlighting information	54 (28)
	Current residents’ research and medical interests∗	34 (18)
	Current residents’ contact information∗	22 (11)
Curriculum	Rotation schedule	146 (76)
	Support for research	144 (75)
	Call schedule	102 (53)
Videos	Residency overview tour∗	96 (50)
	Resident-focused video∗	70 (36)
	Virtual tour	46 (24)

AIRP: ACR Institute of Radiologic Pathology, APD: Associate Program Director, PC: Program Coordinator, PD: Program Director, Q&A: question and answer.

^∗“Our people” super-category, showcasing the people of the program.

Results

Of 197 radiology programs, 192 (97.5%) had accessible websites, so website analysis was done only with the 192 programs. In this analysis, we reported the mean and 1 standard deviation for the actual number and percentage value. Overall, the 192 programs had an average of 16.2 of 30 (54%) ± 6.0 (34%-74%) metrics on their website. Metric comprehensiveness ranged from 7% to 97%. Of 192 websites, 129 (67.2%) and 19 (9.9%) program websites met 50% and 75% of criteria, respectively. The three most included metrics were contact e-mail (90%), current residents (88%), and the hospital facility description (82%) (Table 1). The three least included metrics were current resident contact information (11%), resident research or medical interests (18%), and a virtual tour of the department (24%). The University of Washington, Washington University, and Duke University had the most comprehensive program websites, displaying 29 of 30 (97%), 27 of 30 (90%), and 27 of 30 (90%) metrics respectively (Table 2 ).

Table 2.

Top 10 most comprehensive radiology program websites

Program Name	Number of Present Metrics Out of 30, n (%)
University of Washington	29 (97)
Washington University, Barnes-Jewish Hospital, St Louis Children’s Hospital Consortium	27 (90)
Duke University	27 (90)
University of Texas Southwestern	26 (87)
Yale	25 (83)
University of Rochester	25 (83)
University of Pennsylvania	25 (83)
University of North Carolina	25 (83)
Brown University	25 (83)
Massachusetts General Hospital, Harvard Medical School∗	24 (80)

^∗Tied with four other programs: University of Arkansas, University of Colorado, Boston University, and Cooper University.

All 30 metrics were analyzed under one of the following categories: program overview, applicant information, resident life, curriculum, or videos. The mean number of programs reporting information in these categories were 129.1 of 192 (67.2%) ± 37.6 (47.6%-86.8%), 66.3 of 192 (34.5%) ± 8.1 (30.3%-38.7%), 95.3 of 192 (49.6%) ± 46.6 (25.3%-73.9%), 130.7 of 192 (68.1%) ± 24.8 (55.2%-81.1%), and 68.7 of 192 (35.8%) ± 22.0 (24.3%-47.3%), respectively. This indicates that curriculum and program overview metrics were most commonly available among radiology program websites, whereas applicant information and video metrics were least commonly included. For the additional “our people” super-category, the average comprehensiveness was 91.6 of 192 (47.7%) ± 51.8 (20.7%-74.7%) for program websites.

The mean number of metrics present in the radiology websites from the top 20 Doximity ranked programs were 21.4 of 30 (71.3%) ± 4.1 (57.6%-85%) compared with 15.7 of 30 (52.3%) ± 5.1 (35.3%-69.3%) metrics available from the non–top 20 Doximity programs (P < .001) (Table 3 ). Top 20 ranked program websites also differed significantly from non–top 20 program websites in the categories of program overview (P < .001), resident life (P < .05), curriculum (P < .05), and the “our people” category (P < .001). There was no significant difference between the top 20 and the non–top 20 programs in the inclusion of applicant information and any website videos.

Table 3.

Comprehensiveness of radiology program websites by Doximity ranking

Category	Metrics Present in Websites of Top 20 Programs		Metrics Present in Websites of Non–Top 20 Programs		P Value
	Mean	SD	Mean	SD
Total comprehensiveness	21.4 of 30	4.1	15.7 of 30	5.1	<.001
Program overview	6.7 of 8	1.3	5.2 of 8	1.7	<.001
Applicant info	1.0 of 3	1.0	1.0 of 3	0.9	.95
Resident life	7.8 of 13	2.7	6.3 of 13	3.0	.03
Curriculum	2.5 of 3	0.8	2.0 of 3	0.9	.03
Videos	1.5 of 3	1.1	1.1 of 3	1.1	.11
“Our people”	6.0 of 8	1.4	4.1 of 8	1.8	<.001

SD, standard deviation.

Discussion

In the era of virtual residency interviews, applicants rely on other sources of information such as online media and websites to make decisions about applying to and ranking programs. Our goal with this study is to highlight the comprehensiveness of radiology residency program websites. Our hypothesis is supported with our data. Compared with previous studies assessing website comprehensiveness in radiology, programs have shown an increase in comprehensiveness [5,14]. Nonetheless, we found that improvements can be made, specifically with diversity and inclusion initiatives, resident wellness, applicant information, program benefits, and showcasing the people in the program. We hope that this will encourage programs to update their websites and bridge the gap that is lost during virtual interviews.

Two studies have analyzed radiology residency program websites from 2016. The studies, published in March 2016 and December 2016, included 180 and 189 program websites, respectively [5,14]. As of January 2022, our study included 197 radiology program websites. Compared with 2016, there has been an increase in the percentage of programs reporting messages from the program director (74% versus 46%), fellowship placement (72% versus 55%), faculty listing (78% versus 63%), and current residents (88% versus 78%). Even with these improvements, there are still some categories in which a gap in website comprehensiveness remains. For example, for the categories related to resident benefits such as salary, meal allowance, and parking, the mean increase in comprehensiveness from 2016 to 2021 was only 4.6%. This is in line with prior survey research showing that radiology applicants value having a good fit with the program faculty and residents, as well as their future fellowship prospects, compared with resident benefits when choosing a program [7]. Programs thus might be more perceptive to changing their website to better include factors more heavily weighted by the applicant. Nonetheless, in the era when more websites become more comprehensive, lack of inclusion of program benefits on the website could be an anomaly for a radiology program.

Our research also demonstrates that the top 20 radiology programs had more comprehensive websites compared to non–top 20 programs. Specifically, top 20 programs were found to have a better showcase of the people of the program as determined by the “resident life” and the “our people” categories. This could be an advantage for the top 20 programs in attracting potential applicants in the new online era.

In a survey of 188 residency applicants, 41% of respondents decided to not apply to at least one program based on the quality of its website [3]. Other specialties also realized the need to re-evaluate their program website comprehensiveness in light of COVID-19 changes to the application process [15, 16, 17, 18]. A study analyzing the online presence of neurosurgery programs demonstrated that many programs have increased their virtual presence amidst the pandemic, with 57% of program social media accounts being created in 2020 [16]. Another study on otolaryngology websites saw similar results to our study, with overall improvement in online presence with the continued need to bridge the gaps that still exist during a virtual interview cycle [15]. In line with other specialties, our results also demonstrate that radiology residency websites should improve in comprehensiveness.

Diversity initiatives and implementations that are showcased by programs are a crucial factor for applicants in applying to and ranking residency programs [22]. According to our data, only 34% of programs included any diversity initiative or statement on their website. All applicants should have the opportunity to interact with and learn from a gender, racial, and ethnically diverse group of faculty and coresidents. Similarly, residents who are underrepresented in medicine should feel that they belong in their learning environments by working with colleagues of similar backgrounds. In fact, the ACGME requires programs to provide a copy of the institution’s diversity and nondiscriminatory policy to applicants, which can be provided on the program website [22]. Moreover, a recent survey reported that applicants underrepresented in medicine and female radiology applicants placed more value on program diversity when selecting programs compared with those that are not underrepresented in medicine and their male applicant counterparts [19]. Radiology residency programs thus should include more transparent statements about diversity as well as showcase initiatives on their website to promote inclusiveness.

According to a study of 622 radiology residency applicants, the most important resources for learning about a radiology residency program were interactions with residents and faculty [7]. The “our people” super-category aims to assess the comprehensiveness of radiology websites regarding the showcase of the people of the program. Although 88% of websites listed their current residents, only 35% listed resident extracurricular activities, and 18% listed residents’ medical or research interests. Furthermore, only 11% of websites listed resident contact information, which is important information to include so that prospective applicants can connect with a resident to learn about the program. A better showcase of the people of the radiology program would close the gap between virtual and in-person interviewing.

Additionally, the most important factor in ranking a program for radiology residency applicants was the perceived happiness of residents [7]. A program that values resident wellness and addresses burnout in an evidence-based manner is likely to foster happiness between its residents [23]. Our analysis showed that only 38% of programs listed any resident wellness or mental health initiatives on their websites. Moreover, radiology residents report that the COVID-19 pandemic has had a marked impact on their well-being [24]. It is likely that radiology residency programs that demonstrate wellness and burnout-reduction initiatives will be looked upon favorably by applicants.

Although the COVID-19 pandemic has altered the residency interview process, programs can use this opportunity to utilize alternative platforms to inform applicants about their program. After websites, the most helpful platforms that showcased residency programs in 2021 were Twitter and Instagram [4]. Furthermore, an analysis of radiology Twitter hashtags over a 30-day period in 2021 showed that #RadRes—short for “radiology residents”—was used 7,864 times [25]. Besides social media, additional resources for programs to effectively promote their program in the online era could include YouTube videos, virtual away rotations, and virtual open houses [20,26, 27, 28].

Limitations

One limitation of this study was that metrics were recorded simply with a present-or-absent binary criterion; thus, the quality of the information was not included in the analysis. Moreover, although the authors expanded on the number of metrics to evaluate a program website, there may still be other factors about residency programs that applicants also take into consideration that we failed to include in our analysis. Nonetheless, our study highlights potential areas of improvement for radiology residency websites to advertise their programs more comprehensively and accurately to applicants. Further research should analyze other media such as Twitter, Instagram, Facebook, YouTube, and so on to better capture a program’s online comprehensiveness and resources.

Take-Home Points

• ▪Due to the COVID-19 pandemic and subsequent virtual interview cycles, the radiology residency application process has shifted toward reliance on online resources.
• ▪Although radiology program websites were determined to be the most important medium for showcasing a program, they are still lacking in several key areas.
• ▪Diversity initiatives, resident wellness, videos, applicant information, and a showcase of the people of the program are some key areas of improvement for these websites.
• ▪Alternative online engagement efforts such as Twitter, Instagram, and virtual away rotations provide radiology residency programs with novel ways to reach applicants and advertise themselves in the COVID-19 era.