Editor's Note: The ACGME News and Views section of JGME includes data reports, updates, and perspectives from the ACGME and its review committees. The decision to publish the article is made by the ACGME.
On July 1, 2014, the second group of programs accredited by the Accreditation Council for Graduate Medical Education (ACGME) entered the Next Accreditation System (NAS), and all programs are now functioning under the principles of NAS, which include continuous accreditation via annual reviews of data, lengthening the interval between scheduled accreditation visits, and a focus on educational outcomes. The aims are to reduce the burden of accreditation, contribute to ongoing program improvement, and allow high-performing programs to innovate.1 The focus on improvement calls for a new approach to self-assessment, and the model is a Self-Study undertaken by the program, producing a record of improvements and areas still being worked on, followed by Self-Study Visit (SSV). The SSV has 2 objectives: assess the program's current status and compliance with the relevant requirements, and review the record of improvements made since the last scheduled accreditation review.2
This practical guide to the ACGME Self-Study and the SSV is based on the deliberations of ACGME senior staff who designed elements of the NAS, and on test visits to nearly 40 accredited programs to pilot the new elements of the Self-Study and associated site visit. These visits explored the utility and value of the new dimensions of the Self-Study: use of longitudinal data from the Annual Program Evaluation to record programs' ongoing improvements and a new focus on program aims, and opportunities and threats. The guidance is organized around 8 steps for conducting a Self-Study, shown in box 1.
box 1 Self-Study Steps
Assemble the Self-Study Committee
Aggregate and analyze data from successive Annual Program Evaluations to generate a longitudinal assessment of the program's improvement
Engage leaders and constituents in a discussion of program aims
Identify program strengths and areas for improvement
Examine program opportunities and threats facing the program
Aggregate the Self-Study findings
Discuss the findings with program leadership and constituents to set improvement priorities
Develop a succinct Self-Study document for use in further program improvement and as documentation for the Self-Study Visit
1. The Self-Study Committee
Given its role in the Annual Program Evaluation, the Program Evaluation Committee (PEC), or a slightly expanded group with additional faculty and residents will be ideally suited to serve as the core body for the Self-Study. The members of the PEC are already involved in planning and evaluating educational activities, reviewing and revising curricula, addressing areas of noncompliance, and conducting the Annual Program Evaluation. In addition, given the focus on educational outcomes, it may be useful to have a representative from the Clinical Competency Committee on the Self-Study Committee.
Subspecialty programs will appoint their own PEC, but it is important to note that the SSV for a subspecialty program will be coordinated with that for its core program, and there may be benefit in coordinating the Self-Study. The reason for the coordinated approach is that in NAS, the ACGME places more responsibility for the oversight of subspecialty programs on the core program, and there are dimensions such as faculty development, scholarly activity, or access to educational resources such as simulation, where the core programs and its subspecialty programs may benefit from a coordinated approach.
2. Longitudinal Annual Program Evaluation Data
The Self-Study constitutes an in-depth assessment of the program that examines longitudinal data from all annual evaluations since the program's last scheduled accreditation review. Depending on the timing of the Self-Study, it may also incorporate the Annual Program Evaluation for the current year. For visits scheduled in 2015 and 2016, programs are expected to track improvements since the last scheduled review, generally the prior 5 to 6 years. The information collected should highlight strengths, areas where improvement has been achieved, and areas still in need of improvement. For areas for improvement, the focus should be on each year's action plans (V.C.3 (Core)), records of progress on the previous year's action plans (V.C.2.e (Core)), and documentation in PEC meeting minutes that relate to the action plans (V.C.3.a (Detail)). This information can be entered into a simple table or spreadsheet to create a longitudinal record of the improvements achieved.
In addition to data from the Annual Program Evaluations, the Self-Study Committee may explore what other existing data (such as information collected by the sponsoring institution) may be of value to the Self-Study.
3. Program Aims
Program aims are a new dimension for the Self-Study. They offer added context for a program's improvement efforts by focusing on program and institutional leaders' key expectations for the program, and are elicited through responses to the question, “What types of residents is the program educating; what are their future roles and practice settings?” Aims may focus on some or all of these dimensions: types of trainees accepted into the program, training for particular career options (clinical practice, academics, research, primary/generalist care), and added objectives, such as care for underserved patients, health policy or advocacy, population health, or generating new knowledge.
4. Strengths and Areas for Improvement
The longitudinal data from the Annual Program Evaluations will offer information on Strengths and Areas for Improvement that, at minimum, will address the 4 dimensions of program evaluation stipulated by the Common Program Requirements (V.C.2) (Core) as shown in box 2. These dimensions can serve as the basis for the assessment of strengths and areas for improvement. In addition, programs may wish to broaden the assessment by surveying residents and faculty about strengths and areas for improvement as part of their Self-Study.
box 2 Required Dimensions of the Annual Program Evaluation
• Fellow performance, V.C.2.a) (Core)
• Faculty development, V.C.2.b) (Core)
• Graduate performance, including performance of program graduates on the certification examination, V.C.2.c) (Core)
• Program quality, V.C.2.d) (Core)
The Self-Study test visits revealed a wide range of program strengths, including faculty experience, capabilities and dedication, and resident excellence and caring. Common areas for improvement related to the need to enhance the capabilities of vendor-provided residency management suites to generate useful data for Clinical Competency Committees, and faculty development needs, particularly needs related to faculty members' expanded role in making Milestone assessments.
5. Opportunities and Threats
A new dimension of the Self-Study is the focus on opportunities and threats. Opportunities are factors beyond the immediate control of the program that, if acted upon, contribute to enhanced success, while threats are factors that could have a negative effect. Assessing the environment for opportunities and threats could be an ongoing activity, but at minimum, the Self-Study Committee and program leaders should examine these dimensions as part of the Self-Study.
Exploring opportunities and threats is relevant to program sustainability by considering factors that may assist with, or detract from, the program succeeding and achieving its aims in the future. For example, the Self-Study test visits to several primary care programs revealed common threats for programs that depend on community settings for resident experiences. Added pressure for clinical productivity on community practitioners, and practices being assumed into delivery networks under the Affordable Care Act reduced community faculty ability and willingness to serve as preceptors for residents. A benefit of identifying potential threats is that it facilitates the development of contingency plans for dealing with them.
6. Aggregating the Self-Study Findings
The next step entails aggregating the data collected and examined during the Self-Study process. The documentation of the findings should ensure the program has considered how strengths will be maintained, has prioritized the areas for improvement, and has developed plans for what improvements will be made and how success will be tracked. The discussion should ensure that program leadership takes advantage of opportunities, and that threats are considered and addressed.
7. Discussion of the Findings by Program Leaders and Constituents
The primary aim of the Self-Study is to assist in ongoing efforts at program improvement. In this context, its value is greater when there is input from the faculty, residents and fellows, and any other constituents relevant to the program. When discussing the data from the Self-Study with these stakeholders, program leadership should ask them to reflect on program improvement and challenges that still face the program, using the information collected during the Self-Study.
8. The Self-Study Document
In the final step, the Self-Study Committee should synthesize the information into a succinct Self-Study report that focuses on strengths, areas for improvement, opportunities and threats, and an action plan for addressing priority areas for improvement. Consistent with an improvement model, programs are expected to indicate both strengths and areas still in need of improvement in the document. An outline showing the elements of the Self-Study document is shown in box 3.
box 3 Elements of the Self-Study Document
Introduction (which should also provide a brief summary of the how the Self-Study was conducted, and the participation of program leadership, residents, faculty, and any other stakeholders)
Program Overview
Program Aims
Aggregated lists of Strengths and Areas for Improvement for the period since the last scheduled accreditation review
Opportunities and Threats
Action Plans for maintaining Strengths, addressing Areas for Improvement, capitalizing on Opportunities, and mitigating Threats
The Self-Study document should offer insight into how prior areas for improvement were addressed, as well as any high-level information about current areas still in need of improvement. The document should not exceed 2500 words (about 7 pages of single-spaced narrative). Programs are expected to upload their final updated report through the Accreditation Data System (ADS) 10 days prior to the SSV.
The Self-Study Site Visit
The first group of Phase I programs is scheduled for an SSV in late spring and summer of 2015. Programs scheduled for an SSV in the last months of academic year 2014–2015 will need to initiate their Self-Study in the fall of 2014.
The ACGME will give programs 90 to 120 days of advance notice for the exact date of the SSV. Programs should update their Self-Study document to reflect any recent improvements in high-priority areas identified during the Self-Study, and also update the narrative sections in ADS.
A survey team made up of 2 or more accreditation field representatives will visit the program, or the core program and its associated subspecialty programs. Visits for a core program will generally be a full day, while visits for a subspecialty program may involve a half-day or less for a smaller subspecialty. Prior to the visit, the team members will review the materials submitted by the program in detail. At the time of the SSV, program leaders should be able to describe action plans for addressing key areas for improvement, and might be able to provide early data on the success of these efforts.
During the visit, the survey team will develop a list of strengths, areas for improvement, and opportunities and threats based on the program's report and information gathered during interviews and review of documentation. At the conclusion of the site visit, selected strengths and areas for improvement will be shared as verbal feedback with the program leadership. This feedback does not include any recommendations about compliance with standards or the program's accreditation status; these decisions are the prerogative of the Review Committee.
Conclusions
An effective Self-Study requires participation from program leadership, residents and fellows, faculty members, and any other constituents, with a frank assessment of the program's strengths and areas for improvement. There also is benefit in an approach that provides transparency through sharing the findings with constituents and eliciting their reaction. Finally, programs may benefit from exploring other uses of the data from their Self-Study. The effort of conducting a Self-Study can be offset by the uses and value of the information gathered for the given program.3
Footnotes
All authors are with the Accreditation Council for Graduate Medical Education. Ingrid Philibert, PhD, MBA, is Senior Vice President, Field Activities; and Mary Lieh-Lai, MD, is Senior Vice President, Medical Accreditation.
The authors would like to thank the leadership, faculty, and residents of the nearly 40 accredited programs that participated in the Self-Study test visits held between 2012 and 2014 for their willingness to take the time to participate, and for their frank discussion and feedback on the Self-Study process. We would also like to recognize the contributions of the members of the field staff who participated in the Self-Study test visits, particularly William W. Robertson Jr, MD, MBA; Serge Martinez, MD, JD; Donna Caniano, MD; John Coyle, MD; Judith Rubin, MD, MPH; and Barbara Bush, PhD, and those members of the ACGME senior leadership team who helped refine the approach.
References
- 1.Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—rationale and benefits. N Engl J Med. 2012;366(11):1051–1056. doi: 10.1056/NEJMsr1200117. [DOI] [PubMed] [Google Scholar]
- 2.Coyle J, Martinez S, Robertson WW, Jr, Philibert I. Testing a site visit approach for the next accreditation system. J Grad Med Educ. 2013;5(2):349–351. doi: 10.4300/JGME-05-03-43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Simpson DE, Golden DL, Rehm JM, Kochar MS, Simons KB. The costs versus the perceived benefits of an LCME institutional Self Study. Acad Med. 1998;73(9):1009–1012. doi: 10.1097/00001888-199809000-00024. [DOI] [PubMed] [Google Scholar]