Abstract
Background
Prospective orthopedic residency applicants commonly use one of three databases to identify potential programs: Accreditation Council of Graduate Medical Education (ACGME), American Medical Association (FREIDA), or Orthogate. org. In addition, institutional websites are typically the primary source of information once programs are identified. We sought to evaluate the databases and websites used by prospective orthopedic surgery applicants for content and accessibility. We hypothesized that information would be more available in comparison to previous studies but would still fail to provide complete, up to date program information for the prospective applicant.
Methods
Three online databases were queried in December 2014 to compile a list of orthopedic residency programs in the United States. This combined list was used as a basis for evaluating individual institution websites. Previously described criteria were used to evaluate the availability of information contained within orthopedic surgery residency websites.
Results
At the time of online review, 157 programs were identified. Depending on the database in question, up to 33% of programs either did not provide a link or listed a non-functioning link. Among the variety of evaluated criteria, inclusion of the information varied between 12% and 97% for the individual program websites.
Conclusions
Online databases are useful in listing programs, but individual program details and direct functional links are lacking. Most program websites contain varying degrees of desired information; however, not all programs maintain websites which consistently provide information to satisfy the evaluated criteria in this study. Improved online accessibility and availability of information for residency programs would increase their visibility and utility for prospective applicants.
1. Introduction
Each year, more medical students apply for orthopedic residency. With this increase in number of applicants, the competition for a position continues to increase as well, making it one of the most competitive specialties.1 With these trends, the importance of maintaining an informative and accessible website continues to grow. The importance of web-based information has been evaluated for multiple orthopedic fellowships2-4 as well as various other surgical residencies.5-7 Rozental et al. performed a similar study for orthopedic residencies in 2001.8 Their study revealed at that time many academic orthopedic departments underutilized the Internet with subpar websites or lack of an Internet presence. Although the Internet has been established as a useful communication tool for quite some time, utilization has significantly increased since 2001.9 Increased utilization brings more up-to-date and accurate information, however not all academic departments take advantage of this useful communication tool.2-7
Medical students frequently rely on online databases to identify available residency programs. Three commonly used databases are maintained by the Accreditation Council of Graduate Medical Education (ACGME), 10 American Medical Association (Fellowship and Residency Electronic Interactive Database - FREIDA),11 and the open-source website Orthogate (http://www.orthogate.org).12 The purpose of this study was to determine the availability and accessibility of information on orthopedic residency programs obtainable through the three databases. We analyzed the information available on various program websites through the links provided by the three databases and from the results provided by a Google search. In addition, previous research by Rozental et al. allowed a comparison to gauge the improvement in several key categories over the past decade. We hypothesized that the ease of accessing individual program websites from databases and discovering relevant program information contained within independent residency websites does not fully meet the needs of current orthopedic surgery applicants.
Methods
Identification of orthopedic residency programs in the United States was accomplished with the use of the ACGME database, the AMA’s FREIDA online database, and Orthogate’s online database.10-12 The database search only included allopathic orthopedic residencies, as there is not currently a combined process for osteopathic and allopathic residencies. The three databases were queried between December 21 and 23, 2014. Each database was assessed for availability and functionality of website links to each program by placing them in one of five categories: no link provided, a non-functional link, a link to the sponsoring institution requiring multiple clicks to navigate to the residency website, a link to the orthopedic department requiring multiple clicks to navigate to the residency website, and a link which led directly to the residency website. The databases were also evaluated for congruency of information, including programs listed, program director, and contact information.
A Google search (Mountain View, CA, USA)13 was also performed to evaluate website accessibility for each program as an alternative to searching the three online databases. Google was selected because it is the most popular search engine worldwide.14 A search was performed for each program using the phrase “program name + orthopedic surgery residency.” Each search evaluated the first page of results (first 10 listings) for direct links to the residency program website.
Each orthopedic residency program’s website was then evaluated for content using previously described areas of interest2-4 with examination of resident education details, resident recruitment details, and contact information. In addition to criteria described in similar papers, a study by Deloney et al, which performed a survey of radiology interviewees at a single institution, was used to compile a list of relevant details. The Deloney et al study characterized details as necessary, desirable, or superfluous.15 Resident education details included rotation schedule, didactic schedule, conference descriptions, research curriculum, and call schedules. Resident recruitment details included program description or director’s letter, application requirements, faculty education, current residents, resident education information, career placement, and salary. Results were then analyzed as a proportion of programs containing the information compared to previous studies.
Results
Database Information
The three databases revealed a varying number of total programs – 156 programs were listed in the ACGME database, 157 programs were listed in the FREIDA database, and 153 programs were listed in the Orthogate database. The databases provided either no link or a link that was non-functioning in 12% (FREIDA), 21% (ACGME), and 33% (Orthogate) of the program listings. A majority of programs provided a functioning link that, at a minimum, directed the user to an institutional website. A direct link to the unique residency website was provided by a small percentage of programs: ACGME listed 24 (15%), FREIDA listed 34 (22%), and Orthogate listed 26 (16%) (Table I).
Table I.
Evaluation of Links Provided by Databases
| Database | Programs | No Link | Non-functioning | Institution | Department | Residency |
|---|---|---|---|---|---|---|
| ACGME | 156 | 4 (3%) | 28 (18%) | 41 (26%) | 60 (38%) | 24 (15%) |
| FREIDA | 157 | 9 (6%) | 9 (6%) | 43 (27%) | 62 (39%) | 34 (22%) |
| Orthogate | 153 | 13 (9%) | 37 (24%) | 28 (18%) | 49 (32%) | 26 (17%) |
Combining the search results of the three databases, 157 unique orthopedic residency programs were identified, including 149 civilian programs and 8 military programs. This combined list served as the basis for evaluation of institutional websites. All programs were found using a Google search that included “program name + orthopedic surgery residency.”
Most of the contact information, including phone number, email, name of the program director, was congruent across the ACGME and FREIDA databases. Orthogate did not provide any contact information. However, 64 (41%) programs had different email addresses and 36 (23%) programs had different phone numbers listed in comparing the ACGME and FREIDA databases
Resident Education
With respect to resident education, most programs included the evaluated criteria. A rotation schedule was provided by 118 (75%) programs. The majority of programs included information detailing their didactic schedules, research requirements, and meetings or courses attended by the residents. However, only a small number of programs presented information describing the resident call schedule (Table II).
Table II.
Number (%) of websites with information pertaining to resident education
| Education (n = 157) | No. (%) |
| Didactic Schedule | 106 (67%) |
| Rotation Schedule | 118 (75%) |
| Research Curriculum | 93 (59%) |
| Conference Descriptions | 94 (60%) |
| Call Schedules | 19 (12%) |
Resident Recruitment
In regards to resident recruitment, the majority of programs covered the evaluated criteria. Nearly all programs provided a description of the program. A list of current residents could be found on the websites of 129 (82%) programs while only 109 (69%) provided detailed educational background for those residents. Career placement was supplied by half of the programs (Table III).
Table III.
Number (%) of programs with information pertaining to recruitment.
| Recruitment (n = 157) | No. (%) |
| Program Description | 153 (97%) |
| Application Requirements | 129 (82%) |
| Current Residents | 129 (82%) |
| Resident Education Information | 109 (69%) |
| Alumni Career Placement | 79 (50%) |
| Faculty Education Information | 109 (69%) |
| Salary | 55 (35%) |
Contact Information
Although contact information was listed for all 157 programs, the type of this information varied among programs. Eighty-one of the programs (52%) provided a telephone number and/or email for both the program director and residency coordinator, 70 (45%) listed information for only the coordinator, and 6 (3%) had only the director’s information available.
Discussion
When researching residency programs, medical students typically begin with a search of available programs using one of the publicly available databases and then progressing to evaluation of individual programs. Multiple studies have examined the quality of information available for various surgical sub-specialties and orthopedic fellowships.2-7 In a comprehensive review of orthopedic programs in 20018, Rozental et al. found that most orthopedic programs under-utilize the Internet as a tool for dissemination of information.
Our current research reveals improvement in utilization, both in accessibility and content, although room for improvement continues to exist. It appears academic departments are realizing the importance of an Internet presence in reaching potential applicants. Having multiple steps needed to access the website and out of date information reflects poorly on the individual program. Orthopedic residency websites compare favorably to websites for orthopedic fellowships; the shared criteria reveal similar proportions of inclusion.2-4 This does not serve as surprise as many of the same individuals are responsible for both residency and fellowship websites. Expanding the comparison to other surgical specialties shows similar proportions as well.6-7
In 2014, seventy students applying to a radiology residency returned a survey prepared by Deloney et al.15 More than half agreed with a long list of elements necessary for a residency website (many of the same elements evaluated by this project), with another 30% to 40% responding that those elements were desirable. They suggested that websites are an important recruiting tool, maintaining them with current information is important to the recruitment process, and site navigation needs to be intuitive and efficient. A survey of orthopedic residency applicants would serve as an important future research avenue to more effectively determine what matters most to students pursuing a position in orthopedics.
Evaluation of the three available databases highlighted programs that did not provide a direct link to the residency homepage - 12% (FREIDA), 21% (ACGME), and 33% (Orthogate) of programs. . Although the lack of functioning links is not necessarily reflective of the program, as the databases are maintained by the AMA, ACGME, or are open-sourced, it does reflect a shortcoming in providing ease of access for applicants. Additionally, the FREIDA database included one extra program not listed by ACGME; the reason for this remains unclear. Concerning database congruency, most programs had the same information provided. Although many of the numbers and addresses appear to be similar (e.g. likely would reach someone within the orthopedic department), the discrepancy makes contacting the program involve unnecessary additional steps.
Since Rozental et al.8 published their findings, the importance of having a useful web presence has increased significantly. As expected, each of the shared criteria between our studies shows an increased percentage of programs publishing the desired information. The improvement is likely tied to both an increased awareness of shortcomings as well as more individuals with a clearer understanding of the Internet’s importance with today’s students. Importantly, in 2001, only 73% of orthopedic programs maintained websites while in 2014, all orthopedic programs were noted to have a website. The elements demonstrating the largest increases between the 2001 study and ours are contact information listed, 43% to 100%; rotation schedules, 21% to 75%; current resident listing, 45% to 82%; and career placement of alumni, 12% to 50%. These numbers reflect critical improvement in providing a clear description of what the program has to offer.
Many of the orthopedic programs provided information in the areas evaluated in this study. In only two areas did fewer than half of the programs report the desired information – call schedule (12%) and salary details (35%). Also of note, 79 (50%) programs included information concerning career placement of their alumni. This information provides an opportunity for the program to showcase the success of previous graduates and allows the applicant insight into post-residency opportunities based on these trends. Another criterion to note was the medical school attended by current residents; 69% of programs reported this information. This information could potentially be important to prospective applicants, as the educational background highlights connections between prospective applicants and current residents. Previous studies have not included this criterion, however this information serves as an important tool in networking.
This study has several limitations. Although multiple publications have arrived at a consensus concerning important criteria in the application process, individual investigators determine these elements. A survey of residents, applicants, and interested medical students would be beneficial in directing future studies as to which criteria are truly important. In addition, the determination of whether the information was included in the website was a binary decision – there was no consideration as to the varying degrees of quality of information. Also, some programs maintain more than one website as they are affiliated with multiple entities. We only evaluated the top result on Google and did not continue to search for additional websites. Another important understanding is that many programs do not have direct control in updating their pages; as most academic centers have a central website, changes must go through other departments prior to publication. Most importantly, we realize that the Internet is a dynamic entity. These websites were evaluated in December 2014, and programs could have added or subtracted information, which may change the reported results
In conclusion, orthopedic residency programs can evaluate their improvement in disseminating information based off two studies separated by thirteen years. The overall trend shows improved utilization of the Internet; however, there are still areas in which individual programs can increase their appeal to applicants. Ensuring that information is up to date on the centralized databases is one avenue. More directly under the program control is the information contained on their unique website. Most programs contain varying degrees of desired information, however, not all programs maintain up to date websites consistently including the same evaluated criteria. As this information is lacking, it is difficult for the applicant to perform head to head comparisons. Residency programs would benefit from routine analysis of their website to ensure the information is up to date and serving as a positive representation of what they have to offer to potential applicants. The Internet already has established itself as the primary source for information, and program websites serve as the initial impression for many prospective applicants.
References
- 1.National Resident Matching Program (US). Charting Outcomes in the Match, 2014. [Internet]. Washington, DC. [cited 2015 January 6]. Available from: http://www.nrmp.org/wp-content/uploads/2014/09/Charting-Outcomes-2014-Final.pdf.
- 2.Silvestre J, Guzman JZ, Skovrlj B, Overley SC, Cho SK, Qureshi SA, Hecht AC. The Internet as a communication tool for orthopaedic spine fellowships in the United States. Spine J. 2014;(14):01770–7. doi: 10.1016/j.spinee.2014.11.024. Nov29pii: S1529-9430 doi: 10.1016/j.spinee.2014.11.024. [DOI] [PubMed] [Google Scholar]
- 3.Davidson AR, Murphy RF, Spence DD, Kelly DM, Warner WC, Jr, Sawyer JR. Accessibility and quality of online information for pediatric orthopaedic surgery fellowships. J Pediatr Orthop. 2014 Dec;34(8):831–4. doi: 10.1097/BPO.0000000000000217. [DOI] [PubMed] [Google Scholar]
- 4.Mulcahey MK, Gosselin MM, Fadale PD. Evaluation of the content and accessibility for accredited orthopaedic sports medicine fellowships. J Bone Joint Surg Am. 2013 Jun 19;95(12):e85. doi: 10.2106/JBJS.L.00785. [DOI] [PubMed] [Google Scholar]
- 5.Reilly EF, Leibrandt TJ, Zonno AJ, Simpson MC, Morris JB. General surgery residency program websites: usefulness and usability for resident applicants. Curr Surg. 2004 Mar-Apr;61(2):236–40. doi: 10.1016/j.cursur.2003.10.006. [DOI] [PubMed] [Google Scholar]
- 6.Svider PF, Gupta A, Johnson AP, Zuliani G, Shkoukani MA, Eloy JA, Folbe AJ. Evaluation of otolaryngology residency program websites. JAMA Otolaryngol Head Neck Surg. 2014 Oct;140(10):956–60. doi: 10.1001/jamaoto.2014.1714. [DOI] [PubMed] [Google Scholar]
- 7.Silvestre J, Tomlinson-Hansen S, Fosnot J, Taylor JA. Plastic surgery residency websites: a critical analysis of accessibility and content. Ann Plast Surg. 2014 Mar;72(3):265–9. doi: 10.1097/SAP.0000000000000125. [DOI] [PubMed] [Google Scholar]
- 8.Rozental TD, Lonner JH, Parekh SG. The Internet as a communication tool for academic orthopaedic surgery departments in the United States. J Bone Joint Surg Am. 2001 Jul;83-A(7):987–91. doi: 10.2106/00004623-200107000-00002. [DOI] [PubMed] [Google Scholar]
- 9. United States Department of Commerce (US). Computer and Internet Use in the United States: Population Characteristics, May 2013 [Internet]. Washington, DC. [cited 22 December 2014]. Available from: https://www.census.gov/prod/2013pubs/p20-569.pdf.
- 10. ACGME (US). ACGME program search [Internet] Chicago, IL [cited 2014 December 21-23]. Available from: https://apps.acgme.org/ads/Public/Programs/Search.
- 11. American Medical Association (US). FREIDA online [Internet]. Chicago, IL. [cited 2014 December 21- 23] Available from: http://www.ama-assn.org/ama/pub/education-careers/graduate-medical-education/freida-online.page.
- 12. Orthogate (CA). Orthopaedic surgery residency programs [Internet]. Toronto, Canada. [cited 2014 December 21-23]. Available from http://www.orthogate.org/residency-programs.
- 13. Google (US). Google search engine [Internet]. Mountain View, CA. [cited 2014 December 21-23] Available from: http://www.google.com.
- 14. Alexa Internet (US). Top sites [Internet]. San Francisco, CA. [cited 2014 December 20] Available from: www.alexa.com/topsites.
- 15.Deloney LA, Perrot LJ, Lensing SY, Jambhekar K. Radiology resident recruitment: A study of the impact of web-based information and interview day activities. Acad Radiol. 2014 Jul;21(7):931–7. doi: 10.1016/j.acra.2014.03.009. [DOI] [PubMed] [Google Scholar]