CORR^® Curriculum—Orthopaedic Education: The ACGME Annual Program Evaluation: Program Improvement or Unnecessary Burden?

When the Accreditation Council on Graduate Medical Education (ACGME) transitioned from periodic to continuous assessment of residency programs in 2013, it did so knowing residency programs would have to assess their own programs on an annual basis as well. Enter the Annual Program Evaluation (APE), which is a yearly assessment of a residency program’s progress towards meeting certain education requirements. In particular, the APE focuses on resident performance, faculty development, graduate performance, and program quality [1].

The APE is critical for developing longitudinal residency program performance data, and this information forms the basis of a 10-year self-study and ACGME site visit. In fact, the ACGME views the APE as “the cornerstone of program improvement” [3]. Although programs have gained some experience on how to conduct this yearly evaluation, the APE requirements can feel burdensome; the art is to make them useful to the program itself.

Forming Your Program Evaluation Committee

Before an APE can be completed, a Program Evaluation Committee (PEC) should be formed. The ACGME requires every residency program to have a PEC, regardless of specialty. A PEC consists of a program director, at least two faculty members, and at least one resident. All programs are required to collect multiple assessments—from resident and faculty performance to program quality. The PEC’s job is to track all of these assessments periodically, addressing extensive program citations and areas for improvement [2, 4].

Participation in the PEC requires substantial time and effort and is best served by faculty with a broad range of experience and focus on the program’s overall best interest. We believe programs should offer incentives to encourage faculty to get involved. For example, PECs can be venues for faculty development, where institutions could identify future program directors by evaluating their contributions to the committee. Academic departments should also compensate participating faculty financially, as well as consider participation in a PEC as a form of good citizenship when faculty members are considered for promotion.

Once the PEC is solidified, the question becomes how can the committee make the most of their time? The PEC should assign its members a specific area or topic to review and present at the yearly APE meeting where the residency program’s progress over the year is discussed. Assigning topics based on the previous year’s ACGME citations or areas for improvement should be the first priority. If, for example, faculty attendance to didactic educational sessions was viewed to be poor on an ACGME annual faculty survey, a constructive response would include how many faculty were present for each session over a period of time (usually from when the concern or citations was known to the program), with supporting data about attendance kept by the program.

Making the APE Worthwhile

While sometimes burdensome, completing the APE each year has improved our programs in measurable ways. Because of the structured approach, it allows for more faculty engagement, more coherent systematic improvements, and better resident feedback than previous editions of the annual residency updates. The APE forces residency programs to continuously address ACGME citations or areas of concern that may have previously gone unnoticed by faculty back when evaluations were periodic or less structured.

A yearly review of faculty and resident surveys, alumni feedback, board certification, procedure logs, resident performance, research and scholarly activity, and duty hours are standard and important measures. The PEC should assign faculty to review each of these parameters for the APE, comparing them to the previous years for trends, as well as national data, if available. Each faculty member should then give a brief verbal report for his or her topic. This allows more faculty to have an active part in the APE, allows for more reflection on a topic rather than overburdening one or few faculty.

Because of the more organized review of the APE, measured improvement of the program is possible. After addressing any citations, potential problems can be reviewed by the PEC in order to address them over the next year. For example, the APE might identify a downward trend in graduate procedure logs, which might be addressed for improvement by rotation changes over the following academic year.

Resident representatives must be on the committee, and fulfill the role of providing feedback from a resident’s perspective. Resident participation can be improved by teaming a resident up with a faculty member to review one of the areas covered by the APE. This fosters mentorship, and allows for obvious resident input on the residency program. A good experience by residents on the committee raises resident morale as well as provides valuable feedback for the residency program.

ACGME Annual Program Update

The annual program update is a condensed version of the APE’s findings, which generally consist of a small number of proposed action plans for improvement for the coming year, citations, or areas for concern.

The annual program update is uploaded to the ACGME website in a standard format, ensuring that all areas are covered for the yearly update. One of the problems with the annual program update is that certain requirements do not necessarily reflect the quality of the program. For example, faculty and resident surveys do not always paint the complete picture of a program. The anonymous ACGME annual faculty and resident surveys are reviewed each year for trends, and may have a major impact on how the residency program is reviewed by the Residency Review Committee. Comparisons are made to other programs across the country (national average) in all areas. Topics include duty hours, faculty participation, resident evaluations, the program’s educational content, resources for the program, and education tools for patient safety and team training. The survey questions are not available to programs in advance, and despite attempts to make the survey precise, the questions are open to interpretation. Throughout the year, it is important for the program leadership to discuss these areas with both residents and faculty, emphasizing the areas of compliance, and areas for improvement, if needed.

Despite these challenges, a yearly annual program update does allow for earlier recognition of problems and a timely way for providing feedback to the program. Continuous improvement of the residency program is more likely under the present system, which benefits those residents in the program as well as the patients they later serve.

In putting forth the effort for the APE and annual program update every year, a compilation of these efforts will provide the data for a successful self-study and ACGME site visit. Involving more faculty in the oversight and administration of the residency program will make the quality of the APE and annual updates more comprehensive. Furthermore, engaging as many faculty and residents as possible in this process will allow for a more complete and realistic recommendations.