Abstract
Background
Residency applicants have the right to see letters of recommendation written on their behalf. It is not known whether applicants are affected by waiving this right.
Objectives
Our multicenter study assessed how frequently residency applicants waived their FERPA rights to view their letters of recommendation, and whether this affected the ratings they were given by faculty.
Methods
We reviewed all ERAS-submitted letters of recommendation to 14 ACGME-accredited programs in 2006–2007. We collected ERAS ID, program name, FERPA declaration, standardized letter of recommendation (SLOR) use, and SLOR Global Assessment ranking. The percentage of applicants who waived their FERPA rights was determined. Chi-square tests of independence assessed whether applicants' decision to waive their FERPA rights was associated with their SLOR Global Assessment.
Results
We examined 1776 applications containing 6424 letters of recommendations. Of 2736 letters that specified a Global Assessment, 2550 (93%) applicants waived their FERPA rights, while 186 did not. Of the applicants who chose not to waive their rights, 45.6% received a ranking of Outstanding, 35.5% Excellent, 18.3% Very Good, and 1.6% Good. Of applicants who waived their FERPA rights, 35.1% received a ranking of Outstanding, 49.6% Excellent, 13.7% Very Good, and 1.6% Good. Applicants who did not waive their FERPA rights were more likely to receive an Outstanding Assessment (P = .003).
Conclusions
The majority (93%) of residency applicants waived their FERPA rights. Those who did not waive their rights had a statistically higher chance of receiving an Outstanding Assessment than those who did.
What was known
Letters of recommendation are an important factor in applicants' successful match in their chosen specialty. Applicants receive conflicting advice on whether to waive or retain their right to see these letters.
What is new
Applicants' decisions to waive or not waive their right to see letters of recommendation affected faculty scoring.
Limitations
Retrospective study identifies associations but cannot infer causality; other factors may influence the differences in scores.
Bottom line
Applicants who did not waive their rights had a statistically higher chance of receiving an Outstanding Assessment than those who did.
Introduction
Each year, more than 35 000 applicants seek acceptance to US residency programs.1 After evaluating applications, residency programs must decide whether or not to grant interviews. Letters of recommendation are an important part of these applications. For US students, these letters of recommendation are protected by the Family Educational Rights and Privacy Act (FERPA), a federal law enacted in 1974 to protect the confidentiality of student records, including letters of recommendation.2 Under section 99.1, “a parent or eligible student must be given the opportunity to inspect and review student education records.” However, letters of recommendation as part of residency applications have traditionally been considered confidential. Thus, when medical students request a letter of recommendation, they are commonly given the option to waive their rights under FERPA to access the letter. Such waivers are not required as a condition of residency acceptance, and there is no consensus on whether applicants are affected positively or negatively based on their decision to waive or not waive their FERPA rights. Although some authorities recommend that students waive their FERPA rights, others do not.3–6
As letters of recommendation remain a key component of prospective residents' applications, it is important to maximize their objectivity and reliability. In response to these concerns, the Council of Residency Directors of Emergency Medicine (CORD) developed a standardized letter of recommendation (SLOR) for emergency medicine residency applicants.7 The SLOR is intended to address concerns over perceived inflation of applicant abilities, to establish a universal language among letter writers, and to provide a quantitative assessment of applicants. First released in the 1990s, the SLOR continues to be commonly used in emergency medicine today. It contains 4 sections: (1) background information on the applicant and letter writer, (2) personal characteristics, (3) a Global Assessment Score, and (4) an open narrative.
The Global Assessment Score section asks the faculty letter writer to rank the applicant in comparison to other emergency medicine residency candidates and to estimate how highly that candidate would be ranked by their own particular program. The form also requires faculty members to reveal how he or she has ranked other candidates in the past. Finally, the faculty member must indicate whether or not the student has waived his or her FERPA right to review this letter of recommendation. A preliminary evaluation of the SLOR compared to traditional letters of recommendation suggested that the SLOR was favored for its readability, objectivity, and consistency.7
A pilot study from a single institution suggested that waiving FERPA rights did not impact faculty's Global Assessment of them.8 We conducted a multicenter study examining letters of recommendation to determine (1) how frequently residency program applicants waived their FERPA rights to view their letters of recommendation; and (2) when applicants' retained their FERPA rights, whether and how it affected their Global Assessment rating. Our hypothesis was that students not waiving their FERPA rights would have an impact on faculty letter writing behavior.
Methods
We conducted a retrospective, observational study that reviewed all Electronic Residency Application Service (ERAS)–submitted applications and letters of recommendation to 14 Accreditation Council for Graduate Medical Education–accredited emergency medicine residency programs (figure). Data were compiled by each participating institution on its own applications. Data collected included program name, applicant medical school, applicant ERAS ID number, applicant's FERPA declaration (if specified), whether or not the SLOR was used, faculty initials, and the SLOR Global Assessment ranking. Non-US medical school data were excluded. Data from duplicate applications and recommendation letters were eliminated. We determined the percentage of applicants who waived their FERPA rights. Chi-square test was used to test the overall association between SLOR Global Assessment Score and waiving of FERPA rights. The overall χ2 test result was significant, and separate χ2 tests were used to test the proportions of Outstanding, Excellent, Very Good, and Good SLOR Global Assessment Scores by FERPA rights waived or not waived. R statistical software was used in the analysis, with a level of significance equal to .05. We maintained applicant and faculty confidentiality through the use of ERAS ID numbers and faculty initials.
FIGURE.
Participating Residency Programs' Geographic Distribution
The study was reviewed and approved by the Borgess Medical Center Institutional Review Board.
Results
After eliminating duplicate applications and letters of recommendation, we reviewed 1776 unique applicants to the 14 participating programs and 6424 unique letters of recommendation. Of these letters, 2808 used the SLOR format; 72 of the SLORs did not include a Global Assessment. Of the 2736 SLORs that provided a Global Assessment, 2550 (93.2%) specified that the applicant had waived his or her FERPA rights, whereas 186 (6.8%) did not waive FERPA rights. Among those applicants who chose not to waive FERPA rights, 44.6% were given a Global Assessment ranking of Outstanding, 35.5% a rating of Excellent, 18.3% were rated Very Good, and 1.6% were rated Good. Among applicants who waived their FERPA rights, 35.1% received a Global Assessment ranking of Outstanding, 49.6% were Excellent, 13.7% were rated Very Good, and 1.6% were rated Good. Applicants who retained their FERPA rights were statistically more likely to receive an Outstanding Global Assessment (P = .011; table).
TABLE.
Global Assessment Ranking
Discussion
Our current study provides additional insight into the resident application process. This is the largest study looking at FERPA rights and its possible effect on letters of recommendation. Applicants who retained their FERPA rights were statically more likely to receive an Outstanding Global Assessment than those who did waive their rights. The fact that applicants are able to read their recommendations may influence some faculty to write better letters.
Although many emergency medicine program directors believe the SLOR has higher interrater reliability than traditional narrative letters of recommendation,9 our study suggests that FERPA may influence the objectivity of the SLOR and be a source of variability. Oyama et al10 postulated that SLOR writers might feel pressured to advocate on behalf of applicants and overstate students' competiveness in the Global Assessment Score section. This phenomenon may be exaggerated when applicants do not waive their FERPA rights and the SLOR author knows the applicant may read the letter of recommendation. Our findings suggest that knowledge of students' ability to view their recommendations may influence faculty behavior when preparing letters of recommendation and questions the objectivity of the SLOR.
This is a retrospective study of associations and, thus, limits out ability to make formal conclusions. Our results revealed an association between Outstanding scores and FERPA rights being waived but do not suggest that the lack of a waiver is the reason for those Outstanding scores. Some applicants who did not waive their FERPA rights may have reviewed their SLORs and then chosen to submit via ERAS only those SLORs with Outstanding scores, potentially biasing our results. It is also possible that some letter writers, despite indicating that the applicant waived FERPA rights, voluntarily allowed the applicant to view the letter. This would permit students who had waived their rights to preferentially submit only those letters with a higher ranking. There also is the possibility that higher performing students in general may wish to read their letters of recommendation and not waive their rights. An analysis of additional applicant characteristics, such as class rank, United States Medical Licensing Examination scores, or sex, may yield additional insight into applicant behavior. However, collection of this level of detail might threaten applicant confidentiality.
This study investigated only emergency medicine residency applications, who may differ from those of other disciplines. For instance, applicants to disciplines that are less competitive may feel less compelled to waive their FERPA rights, as fully 93% did in our study.
Conclusion
The overwhelming majority (93%) of emergency medicine residency applicants waived their FERPA rights to view their letters of recommendation, which remained consistent with the original pilot study.8 Those who did not waive their rights had a statistically higher chance of receiving an Outstanding Global Assessment ranking than those who did.
Footnotes
Jessica Diab, MD, is Attending Physician, Emergency Department, Alaska Native MedicalCenter;Stephanie Riley,MD, is Attending Physician,Department of Emergency Medicine, Community Hospital South; Andrew Downes,MD, is Attending Physician, Sarasota Emergency Associate,and Clinical Instructor,Florida State College of Medicine; Theodore Gaeta, DO, is Program Director, Emergency Medicine, New York Methodist Hospital, and Clinical Associate Professor of Emergency Medicine, Department of Medicine,WeillMedical College of Cornell University; H. Gene Hern, MD, is Residency Director, Department of Emergency Medicine, and Associate Clinical Professor of Emergency Medicine, University of California, San Francisco; Eric Hwang, DO, is Emergency Medicine Resident, St. Joseph’s Regional Medical Center; Lawrence Kass, MD, is Chair for Education, Residency Program Director, Department of Emergency Medicine, Penn State HersheyMedical Center; Michael Kelly, MD, is Assistant Medical Director, Orlando Regional Medical Center, Assistant Professor, Department of Emergency Medicine, University of Florida College of Medicine, and Clinical Assistant Professor of Medicine, Department of Clinical Sciences, Florida State University College of Medicine; Samuel D. Luber, MD, MPH, is Assistant Professor and Residency Program Director, Department of Emergency Medicine, University of Texas Medical School at Houston; Marc Martel, MD, is Program Director, Emergency Medicine Residency, Hennepin County Medical Center, and Associate Professor, Department of Emergency Medicine, University of Minnesota; Alicia Minns, MD, is Assistant Clinical Professor, Department of Emergency Medicine, University of California San Diego; Leigh Patterson, MD, is Assistant Professor, Residency Program Director, Department of EmergencyMedicine, East Carolina University Brody School of Medicine; Philip Pazderka,MD, is Associate Residency Director, Department of Emergency Medicine, and Assistant Professor of Emergency Medicine,Western Michigan University School of Medicine; Osman Sayan, MD, is Assistant Residency Director of EmergencyMedicine, Columbia University, New York-Presbyterian, and Assistant Clinical Professor of Medicine, Columbia University; Jason Thurman, MD, is Associate Residency Director, Department of Emergency, and Associate Professorof Emergency Medicine,Vanderbilt University Medical Center; Phyllis Vallee, MD, is Associate ProgramDirector for EmergencyMedicine, Henry Ford Hospital, and Clinical Associate Professor of EmergencyMedicine,Wayne State University School of Medicine; and David Overton, MD, is Residency Director, Department of Emergency Medicine, and Chairman of Emergency Medicine, Western Michigan University School of Medicine.
Funding: The authors report no external funding source for this study.
Conflict of interest: The authors declare they have no competing interests.
References
- 1.Association of American Medical Colleges. Highest match rate for U.S. medical school seniors in 30 years. https://www.aamc.org/newsroom/newsreleases/276900/120316.html. Accessed January 22, 2013. [Google Scholar]
- 2.United States Department of Education. Family Educational Rights and Privacy Act, 1974, 34 CFR part 99. http://www.ed.gov/policy/gen/guid/fpco/ferpa/index.html. Accessed March 12, 2014. [Google Scholar]
- 3.JABSOM Office of Student Affairs. Guidelines for preparing letters of recommendation for JABSOM students. http://jabsom.hawaii.edu/facdev/FacHandbook/doc/Guidelines%20for%20Letters%20of%20Recommendation%20for%20Medical%20Students.pdf. Accessed January 22, 2013. [Google Scholar]
- 4.Michigan State University College of Medicine. What you need to know about letters of recommendation. http://www.echt.chm.msu.edu/BlockIII/Docs/CoreComp/B3CCCareerDevModLettersOfRecommendationForResidencyApplication.pdf. Accessed January 22, 2013. [Google Scholar]
- 5.University of Illinois College of Medicine at Chicago. Guidelines for requesting letters of recommendation for residency applications. http://chicago.medicine.uic.edu/departments_programs/departments/fammed/residency/application_information/. Accessed January 22, 2013. [Google Scholar]
- 6.Western Michigan University School of Medicine. Advice for emergency medicine applicants: should I waive my right to read my letters. http://med.wmich.edu/node/103. Accessed January 22, 2013. [Google Scholar]
- 7.Keim SM, Rein JA, Chisholm C, Dyne PL, Hendey GW, Jouriles NJ, et al. A standardized letter of recommendation for residency application. Acad Emerg Med. 1999;6(11):1141–1146. doi: 10.1111/j.1553-2712.1999.tb00117.x. [DOI] [PubMed] [Google Scholar]
- 8.Diab J, Riley S, Overton D. The Family Education Rights and Privacy Act's impact on residency application behavior and recommendations: a pilot study. J Emerg Med. 2011;40(1):72–75. doi: 10.1016/j.jemermed.2009.09.017. [DOI] [PubMed] [Google Scholar]
- 9.Girzadas DV, Harwood RC, Dearie J, Garrett S. A comparison of standardized and narrative letter of recommendation. Acad Emerg Med. 1998;5(11):1101–1104. doi: 10.1111/j.1553-2712.1998.tb02670.x. [DOI] [PubMed] [Google Scholar]
- 10.Oyama LC, Kwon M, Fernandez JA, Fernández-Frackelton M, Campagne DD, Castillo EM, et al. Inaccuracy of the global assessment score in the emergency medicine standard letter of recommendation. Acad Emerg Med. 2010;17(suppl 2):38–41. doi: 10.1111/j.1553-2712.2010.00882.x. [DOI] [PubMed] [Google Scholar]