Testing a Site Visit Approach for the Next Accreditation System

Editor's Note: The ACGME News and Views section of JGME includes data reports, updates, and perspectives from the ACGME and its review committees. The decision to publish the article is made by the JGME.

Introduction

In 2009, the Accreditation Council for Graduate Medical Education (ACGME) began work on a Next Accreditation System (NAS), a more outcomes-based and improvement-focused approach to the accreditation of programs and sponsoring institutions. The transition to the NAS is scheduled for phased implementation, beginning July 2013 with 7 Phase I specialties and their subspecialties (emergency medicine, internal medicine, neurological surgery, orthopaedic surgery, pediatrics, diagnostic radiology, and urology).¹ The remaining specialties, their subspecialties, and the institutional review will implement NAS data collection and review starting in July 2014.

A key objective of the NAS entails reducing the burden of accreditation, and one readily identifiable source of burden in the current process was the preparation of the Program Information Form (PIF). To reduce the onus of site visit preparation for programs and to enhance the focus of the visit on the “real” program (as contrasted with a written description), the ACGME decided to explore whether site visits in the NAS could be conducted without a PIF. Types of site visits considered for a PIF-less approach included Full visits, to assess compliance with all pertinent program requirements (eg, the core and detailed requirements for new programs or solely the core requirements for programs on continued accreditation), and Focused site visits, to assess a complaint and/or a potential problem suggested by data from the continuous accreditation process, as defined in the ACGME Manual of Policies and Procedures effective July 1, 2013.²

Here we describe a test of a site visit without the traditional PIF (a “PIF-less” visit) to validate the model that will be used beginning in July 2013 for Full and Focused site visits of NAS Phase I and Phase II programs.

The Test of the PIF-Less Site Visit

Testing PIF-less site visits began in August 2011. A total of 3 sponsoring institutions and 14 individual programs were included in the test, planned as a “proof of concept” pilot to confirm that a site visit to assess compliance with the ACGME accreditation requirements could occur without a PIF. The first set of visits resulted from a timely offer by the designated institutional official of a large university teaching hospital to test a site visit without a PIF, using that institution's GME database. During the planning for the first visit, a second university teaching hospital volunteered for a test visit, and a third institution, an urban public hospital affiliated with a major university, was recruited to increase the diversity of institutions in the test. The aims of these PIF-less site visits were to (1) assess compliance with the ACGME program requirements, (2) include a focus on educational outcomes and physician competencies, and (3) test a process by which the site visitor(s) would provide practical feedback to the program, highlighting areas of strength and suggestions for improvement, as appropriate.

It was decided that the information from these test visits would not be shared with the Residency Review Committees (RRCs). The 14 participating programs would use the information from these visits for local program improvement—either as data gathering for the internal review or as an input into that process.

All PIF-less visits were conducted by teams made up of ACGME field staff and Department of Field Activities leadership. Program and institutional participants included the program director, selected faculty, residents, the coordinator, and the designated institutional official; their respective time commitments were comparable to those of established ACGME site visits. Comprehensive field notes were taken during the visits. Data were also collected during debriefing sessions with the site visitors and program participants.

During the visits, the site visit team used interviews and review of data existing at the program to assess the following program dimensions:

1. Compliance with ACGME program and relevant institutional requirements, including duty hours, supervision, teaching and learning, and scholarly activities, among others;

2. Current status and improvements in areas previously cited; and

3. Availability of educational, technological, and financial resources.

Data and documentation reviewed included the institutional and program information (box), and a consensus list of program strengths and opportunities for improvement aggregated by the residents (and, for some programs, the faculty) and sent directly to the site visit team. At the end of the visit, the team offered preliminary feedback, highlighting key strengths and opportunities for improvement. After the site visit, the team prepared a written report for sharing with the program and sponsoring institution and collected feedback on the process from participants.

Box High-Value Data for Ongoing Program Assessment and Improvement

• Responses to “active” citations (citations not deemed “inactive” by subsequent annual data submissions by the program)

• Major changes since the last site visit

• Resident and faculty consensus lists of program strengths and opportunities for improvement

• Accreditation Data System annual data

• Resident survey results

• Faculty survey results (starting in early 2013 for Phase I programs)

• Block diagram and rotation schedule

• Resident files, including graduates, problem residents, upper-level entrants (systematic nature of evaluations, attrition, final letter)

• Goals and objectives (and stimulated conversations with residents)

• Examples of resident involvement in quality improvement and safety projects (and stimulated conversations with residents)

• Evaluations of the faculty and program

• Conversations about uses of data in decisions

• Improvements from internal review and required annual review

• Duty hours and related policies

• Duty hour compliance data, responses to noncompliance

• Supervision policies (and “policy-stimulated” conversations with residents)

• Board certification data

Data shown in italics were prepared or updated for the test site visit.

Results of the PIF-Less Site Visit Test

Across a diverse group of programs with variable performance in the accreditation process, the PIF-less site visits were well received by site visitors and program participants. PIF-less test visits showed no apparent disadvantages over a visit based on a PIF. Areas that participants considered an improvement over a PIF-based approach included the enhanced focus on the actual program (contrasted with a prior focus on verifying, and sometimes correcting, information in the documentation prepared for the site visit). For the program director interview, the more open format used for these visits highlighted important dimensions of the program, both strengths and areas in need of improvement that may have been less readily identified in a visit with a focus on reviewing the PIF. The PIF-less approach was deemed superior for several excellent programs and one program with potentially significant problems. In one high-performing program, a PIF would not have fully identified a truly learner-centered approach, which was explored through in-depth interviews with residents that fully illustrated the program's ongoing and nuanced adjustments to learner needs. For a program with significant potential deficiencies, the PIF-less approach allowed for an in-depth exploration of a broad range of issues identified in the ACGME resident survey.

One potential disadvantage of a PIF-less site visit, voiced by a minority of participants during meetings and in feedback after the visit, concerned the inability of the program director to “present” the program in a formal manner to the RRC. In one instance where this was voiced during a visit, the site visitor's verbal summary of the program's strengths and issues still being worked on offered assurance that a PIF-less visit could provide an accurate description of key dimensions relevant to an accreditation review.

Adding Value: Reshaping the Site Visit Debriefing and Other Improvements

An added aim of the PIF-less visits entailed exploring whether site visit teams could offer reliable and useful feedback to programs at the conclusion of the visit. This “end of visit” feedback has been requested for many years, but the approach used would need to be sensitive to the unique role of the ACGME site visitor as “fact finder” and the RRC's prerogative to make the accreditation decision. Feedback thus could not constitute, or be viewed as, a prediction of the program's likely accreditation outcome or a summary of areas the RRC might cite. Instead, it was decided that the debriefing would focus on a small number of strengths and opportunities for improvement, with the intent of offering actionable feedback with perceived high utility (and, ideally, low financial and opportunity costs). A focus on strengths was deemed important for balancing the feedback, and in a time- and resource-constrained environment, to avoid a program discontinuing an initiative that benefits resident education for lack of an explicit mention during the site visit.

The PIF-less test visits also resulted in some immediate improvements to site visits that have already been implemented in the current system, including enhanced approaches for engaging learners and faculty through expanded interviews. The visits also further confirmed the utility of placing the resident interviews first in the schedule, and interviewing residents in 2 or more groups by level of training. Finally, the visits produced significant changes in how site visit reports are designed and completed to highlight key observations and items relevant to accreditation, which are then described in added detail.

Discussion

The first test confirmed that the PIF-less visits as envisioned by the ACGME for use in the NAS could be conducted without a PIF, including Full and Focused site visits to assess a complaint or scheduled in response to accreditation data showing a potential problem in a program. Limitations of the test include the relatively small sample size, the fact that it was not possible to do a comparison of a PIF-less visit and a review based on a PIF for a given program to assess whether the 2 approaches would reach similar conclusions, and the use of the site visit information solely for program improvement (not accreditation).

For the site visit team, it was gratifying to see that participating programs and the residents, faculty, and others interviewed approached the PIF-less test visits as though they were “a real site visit,” with active and full participation, providing the data requested for review on site, and a bit of the nervousness that often accompanies an accreditation visit. The visits also were characterized by openness and a focus on improvement, although some of this may have been due to use of the information for local improvement, instead of a “high-stakes” accreditation decision, which may have contributed to a more receptive and less critical audience.

The test of the PIF-less site visit is the first step in a larger exploration of new approaches to the accreditation site visit needed in the NAS. Objectives in the NAS include promoting a minimum level of quality (through compliance with accreditation requirements), ongoing improvement in all programs, and innovation in high-performing programs.¹ The added focus on ongoing improvement will be carried out through (1) a self-study approach in which programs will collect and analyze improvement data, and (2) a self-study site visit that will allow programs to describe their improvement processes and the outcomes achieved. Self-study visits for the first group of programs in Phase I will occur in 2015, and the guidance for the self-study process is currently under development. In a future article, we will present the results of the ongoing test of a self-study site visit, with a focus on program aims, assessment of the program's environment, and a review of ongoing improvement activities and the results achieved.

Conclusion

The test of the new site visits confirmed the feasibility of a PIF-less visit and acceptance of the approach by program and institutional participants and the members of the ACGME field staff who participated. At the same time, some feedback from participants suggests that the PIF, despite complaints about the burden it created, was considered of value as a means of communicating information to the RRC. Further refinements in a PIF-less approach thus will need to overcome this potential disadvantage of the new approach—the lack of a traditional PIF as a primary source of information for the RRC. To compensate for this, the ACGME will need to enhance guidance to programs about how to convey important information to the RRC, using the Accreditation Data System, particularly the sections describing the current status of areas previously cited and major changes to the program. Finally, site visit reports will need to provide added detailed information to compensate for the lack of a PIF, such as program attributes and the environment in which the program operates, along with data on compliance. The aim is to offer RRC reviewers context for their review and, ideally, to meaningfully connect them to the information exchange during the site visit.