Improving Site Visit Reports for High-Performing Residency Programs Without Citations

Introduction

As the Accreditation Council for Graduate Medical Education (ACGME) transitions to the Next Accreditation System (NAS), there is value in exploring the elements of the current accreditation process for opportunities for improvement. A key function of the accreditation process, both in the current system and in the NAS, is to assess compliance with the ACGME program and institutional requirements.

In the current accreditation system, the ACGME's review of individual residency education programs is based on the Program Information Form prepared prior to the site visit. In the NAS, the review will be based on data annually submitted to the ACGME, and brief documentation submitted in preparation for a 10-year self-study visit. After each site visit, the ACGME field representative completes an objective, narrative site visit report relating his or her findings. This report is submitted to the Residency Review Committee (RRC), and together with the information provided by the program and other data (such as the ACGME Resident Survey, and in many specialties, operative and experience logs), it is used to make the accreditation decision. Areas of noncompliance receive citations by the RRC. The best programs are granted Full Accreditation without citations on the basis of substantial compliance with all standards.

The Site Visit Report

During the past 10 years the site visit report has changed significantly. The reports started as open-ended narrative documents. To ensure that compliance with all program requirements was addressed in the report, the ACGME developed a consistent format that followed the order of the program requirements and developed standard forms for each specialty and subspecialty. The most recent versions of those forms provided forced-choice responses and areas where the field representative could add narrative to provide clarification or added details. This format facilitated the identification of all areas in which a program did not meet the accreditation requirements, yet it was less effective in allowing the field representative to identify program strengths or innovative practices. The heavily formatted reports made it possible for a site visit report to indicate full compliance with all program requirements yet appear rather sterile and devoid of the specific information about the program that would allow an RRC to feel comfortable giving a program the longest possible cycle length.

Planning for the Next Accreditation System

In the 2010 ACGME Annual Report, ACGME Chief Executive Officer Thomas Nasca, MD, described preparations for the NAS.¹ Two of the characteristics of the new system were “continued (rather than episodic) oversight” by the ACGME and “a continuous focus on improving program quality for the Review Committees, sponsors, and programs.”¹ With this change there would be a greater expectation of ongoing oversight of the individual programs by the sponsoring institution. The ACGME accreditation site visit for programs would be focused to a greater degree on ongoing quality improvement and efforts to innovate. This new approach was described in 2012 in the New England Journal of Medicine paper, “The next GME accreditation system–rationale and benefits.”²

Study Aims

Both the perceived deficiencies of the current site visit reporting format for stable, high-performing programs without citations, and the shift in site visit focus in the NAS created a need to explore a new site visit report format that would allow a focus on program achievements and innovation. The first Nathan Blank Fellowship project, begun in 2011, sought to address the perceived deficiencies of the current site visit reports for this type of program. The second aim was to contribute to the design of a new site visit report format that would be better suited to (1) conveying positive aspects of a program that go beyond compliance with the requirements, (2) describing program foci that may be important to accreditation in the NAS, and (3) highlighting successful efforts to innovate in graduate medical education.

Methods

The study population for the 2011 Nathan Blank Fellowship Project consisted of programs that during the 2010–2011 academic year received Full Accreditation and did not receive any citations. These programs were independently reviewed by the primary author, a field representative, and the second author, a former RRC executive director.

Both authors compiled a list of comments that the field representatives had written in the report demonstrating that the program exceeded the minimum requirements for the specialty. Any site visit report that contained 1 or more comments of this nature was classified as Positive. The researchers compared the 2 comment lists. Differences were reconciled through consensus, and the researchers developed a final list of comments. They then analyzed this list of Positive reports to explore associations with the field representatives, content, and any other patterns in these positive comments, and potential relationships with the site visit report format.

Results

Of the 956 core residency programs that received accreditation letters between July 1, 2010, and June 30, 2011, 92 gained Full Accreditation without citation. In this analysis, Independent and Integrated Plastic Surgery were combined into a single group, as were Independent Integrated Thoracic Surgery programs. Combined Internal Medicine–Pediatrics programs were considered separately.

figure 1 compares the total number of core programs reviewed for the study period and a subgroup of high-performing programs. figure 2 shows the percentage of high-performing programs among the programs reviewed during the study year, showing that the distribution of high-performing programs without citations was not evenly spread across specialties or field representatives. A total of 20 of the 31 field representatives had completed reports for high-performing programs, with individual field representatives completing between 1 and 12 reports for these programs.

FIGURE 1.

Total Number of Letters Generated by the Various Residency Review Committees (RRCs) During the 2010–2011 Academic Year

Total number of letters is represented by the top bars on this logarithmic scale (range, 10–101). The number of “High Performing” programs for the various RRCs is represented by the bottom bars (range, 0–11). Four of the RRCs identified no high-performing programs without citations. Abbreviations: OB/GYN, obstetrics-gynecology; Neuro Surg, neurological surgery; PM&R, physical medicine and rehabilitation; Med/Peds, internal medicine–pediatrics; Nuc Med, nuclear medicine; CRS, colorectal surgery.

FIGURE 2.

Percentage of High-Performing Programs 2010–2011 Reviewed by Various Residency Review Committees That Received Highest Ratings (ie, “High Performing”) Compared With the Total Number of Programs Reviewed in the Year

Abbreviations: Nuc Med, nuclear medicine; CRS, colorectal surgery; OB/GYN, obstetrics-gynecology; Med/Peds, internal medicine–pediatrics; PM&R, physical medicine and rehabilitation; Neuro Surg, neurological surgery.

Of the 92 programs in the study, the consensus Positive site visitor report list included 52 reports completed by 13 different field representatives. Individual field representatives completed between 1 and 8 of these reports. The reports for 40 programs (completed by 7 field representatives) contained no Positive findings. The 40 high-performing programs without Positive comments were not distinguishable in any significant way from the 52 programs with comments. In both groups there was a random distribution of performance from the most recent previous RRC review in terms of length of accreditation cycle and the number of previous citations. figure 3 shows the number of high-performing programs site visited and the number of reports with Positive comments completed by each field representative in the study. There was a strong correlation between the number of Positive comments contained in the individual site visitor's report and the RRC's finding of high-performing programs (correlation coefficient  =  0.876).

FIGURE 3.

Reports for High-Performing Programs Completed by Various Field Representatives (N  =  92) Compared With the Number of Reports That had Positive Comments

There was a 0.876 correlation between the presence of Positive comments and the “best” rating by the Residency Review Committee.

The positive comments found in the site visit reports were classified by area of the program requirements that was mentioned. This showed that a range of program requirements received positive comments. There were 2 added areas not reflected in the program requirements: improvement and resident satisfaction. figure 4 presents the number of Positive comments in each category.

FIGURE 4.

Percentage of Positive Comments by Category, Categorized by Program Requirements Area

Abbreviations: Ed program, educational program; G&O/curriculum, goals and objectives curriculum; SBP, systems-based practice; PBLI, practice-based learning and improvement.

The sections of the site visit report that contained most of the Positive comments were the Notable Information and Summary sections. Both sections require the field representative to enter free text that is not bound by pull-down menu choices and only optional free-text entries. Another section that included a number of Positive comments was Educational Program and Competencies. Prior to August 2010, that section required text entry. After that date the form was changed to make text entry concerning the various competency areas optional. Making the text entry optional diminished the use of this section for Positive comments. Approximately one-half of the site visit reports reviewed for this study were completed after this change in format. A total of 7 site visit reports contained Positive comments on one or more of the competency areas in the old format, whereas only 3 reports in the newer format contained comments on the competency areas.

Discussion

The accreditation of educational programs by assessment of compliance with established standards is common in all areas of education, including the Liaison Committee on Medical Education^3,4 and the Accrediting Council for Independent Colleges and Schools.⁵

With the 1997 introduction of ORYX,⁶ The Joint Commission began to move from a standards-based approach to an emphasis on outcomes of care. The introduction of the 6 general competencies by the ACGME in 1999 signaled the transition of residency program accreditation toward an outcomes-based model.⁷ The RRC for Internal Medicine further described a more outcomes-based accreditation approach in 2004,⁸ and in 2008 the ACGME began the journey to the NAS.

As the accreditation process moved to more outcomes-based approaches, site visit reports have evolved to provide more consistent review and documentation of a program's compliance with the program requirements. This approach satisfied the needs for standards-based accreditation process, yet did little to allow a focus on program outcomes.

Some field representatives consistently included Positive comments in their report. The format of the study made it impossible to ascertain whether the remaining field representatives did not discover the positive aspects in their programs or did not report those activities in the context of an accreditation approach focused on assessing compliance with the accreditation standards. Moving toward the NAS will require added professional development of the field representatives concerning the importance of including Positive comments, when warranted, in site visit reports.

Although there was a positive correlation between the presence of Positive comments in the reports and RRCs issuing no citations to the program, there was no evidence that the presence of the comments guaranteed or unduly influenced that outcome. A total of 40 of the 92 programs in the study received reviews with no citations and the longest possible cycle length without positive comments, suggesting that RRCs were able to make a decision of high performance in the absence of this information. In the NAS, close communication between the field representatives and the RRCs will be beneficial in highlighting the important factors in residency education. This will improve the identification of programs that consistently provide excellent education, and it may assist in the collection and dissemination of best practices for adoption or adaptation by other programs.

This study has several limitations. Site visit reports encompassed only core programs that received RRC decision notices during a 1-year period, with limited ability to generalize to other years or to subspecialty programs. The evaluation of what constituted a Positive comment was subjective. Other reviewers might have found additional Positive comments in the 92 reports reviewed. The assignment of field representatives to program reviews is not random. Assignments are made on the basis of the field representatives' specialty expertise, experience, geographic location, and other factors. The most experienced field representatives are more likely to be assigned to the most challenging programs that are likely to receive citations by the RRC. Finally, because the study focused on reports for programs for which the RRCs issued no citations, it is not possible to ascertain the degree to which reports for programs with citations may or may not contain Positive comments.

Conclusions

Although the current format of the site visit report does not preclude a field representative from including Positive comments, indicating to the RRC that a program is exceeding the accreditation requirements, this format probably is not optimal to encourage such added descriptions. The finding after the elimination of required text entries for the competencies clearly demonstrates this point. For the NAS, site visit reports will need to be modified to encourage documentation of Positive aspects of the program as well as areas of noncompliance and opportunities for improvement.