The Challenges of Program Accreditation Decisions in 2021 for the ACGME Review Committee for Surgery

Abstract

Through only a few months, the COVID-19 pandemic has greatly impacted the daily activities and education of surgical residents and fellows and the programs in which they are enrolled. The pandemic has also forced many changes for the Accreditation Council for Graduate Medical Education and its Review Committee for Surgery. This article details some of those changes and their effect on the process of conferring 2021 accreditation decisions by the Review Committee.

The first case of 2019-nCoV in the United States was diagnosed on 20 January 2020.¹ At this writing (mid-August 2020), more than 5 million cases have been confirmed in the United States and more than 163,000 Americans have died of the disease.² Mitigation efforts, including social distancing and the closure of schools, restaurants, theaters, and other businesses has had an enormous economic impact on individuals and the nation and have radically altered the day-to-day lifestyles of Americans.

The pandemic has also had a major impact on surgery programs in the United States. Perhaps most significantly, elective operations and other procedures were cancelled in almost every site beginning in mid-March following recommendations of the American College of Surgeons,³ the Centers for Medicare and Medicaid,⁴ the American Hospital Association,⁵ and others. The ability to continue even medically necessary, time-sensitive procedures was adversely affected by a lack of viral testing, limited supplies of personal protective equipment, and the need to reserve beds in critical care units for actual and potential COVID patients. Many locations also experienced disruption in perioperative staffing because of concerns regarding public transportation, the need to provide home childcare and other issues. The cancellation of operations resulted in a diminution of preoperative and postoperative appointments and outpatient visits were further diminished by both patients and staff having to deal with childcare and transportation issues as well as clinical facilities which limited their ability to socially distance. Social distancing recommendations/requirements forced residency and fellowship programs to cancel educational conferences or rapidly convert them to virtual formats. The planned rotations and other clinical experiences of many surgical residents/fellows were altered by re-assignment to Emergency Departments, COVID units or intensive care units because of their procedural ability and prior ICU experience. The daily schedules of other residents/fellows were altered due a lack of surgical work to be done, prophylaxis against infection, or both.

Since 1950, the Review Committee for Surgery (RC; although it has had other titles in the past) has been responsible for the decisions on the accreditation status for each general surgery program and, over time, programs in the subspecialties of pediatric surgery, vascular surgery, hand surgery, surgical critical care, and complex general surgical oncology have been added. Accreditation decisions include the program status, the resident/fellow complement, citations and areas for improvement. The Committee is currently comprised of 13 practicing surgeons who were nominated by the American Board of Surgery (4), the American College of Surgeons (4), the American Medical Association (4), and the American Osteopathic Association (1) as well as one surgical resident and one public member, both of whom were nominated at large. The RC normally uses a specific set informational tools to render accreditation decisions. The pandemic has caused or will cause disruption in the availability and application of several of those informational tools which are important in accreditation decisions and that disruption is somewhat variable between the specialties and subspecialties accredited by the RC Surgery. This article will set forth the informational tools typically used by the RC in making accreditation decisions, describe the anticipated disruptions in those tools for use in the 2021 accreditation decisions and the approach that the RC will take toward making optimal accreditation decisions with less than optimal information.

TOOLS NORMALLY USED IN ACCREDITATION DECISIONS

Since the implementation of the Next Accreditation System,⁶ the RC has used a specific set of informational tools in order to confer accreditation decisions (Table 1 ). The absence of a program's substantial compliance with the Requirements can be identified and cited based on the information contained in any of the listed tools. Most citations, though, derive from information contained in resident/fellow surveys, the board certification examination pass rates of program graduates, the graduate case logs and program site visit reports, particularly those prompted by complaints or by data from the listed sources that raise significant concern about the program (“data-driven site visits”).

TABLE 1.

Informational Tools Normally Used by the Review Committee in Rendering Program Accreditation Decisions

Faculty Roster

Scholarly Activity

Faculty

Resident

Milestones reporting*

Annual ACGME Surveys

Resident/Fellow

Faculty

Board examination pass rate

Case logs

Site Visits

Data-driven

Full

Focused

10-year regular scheduled (full)

Information in “Major Changes and Other Updates” section of Annual Update

^⁎The Review Committee knows only whether the program reported or failed to report Milestones, as required. The Committee does not have access to the Milestones actually reported by any program nor to the Milestones reported for any individual resident/fellow.

Surveys

The items on the resident/fellow and faculty surveys are derived from the Common Program Requirements and designed to assess the resident, fellow, and faculty perceptions of program compliance with those requirements. The Accreditation Council for Graduate Medical Education (ACGME) does not mandate of the use of logs to document work hours and thus compliance with work hour requirements is assessed using resident/fellow survey responses as the principal data source. The survey also serves as a gauge of the program's compliance with requirements that delineate essential elements of any good educational program, including the engagement of faculty members, appropriate supervision, the evaluation system, the availability of essential resources, and the balance between education and other clinical duties. It also probes a program's compliance with requirements regarding patient safety and quality improvement.

Certification Examination Pass Rates

Perhaps the best currently available measure of a program's educational outcome is the ability of the graduates to successfully navigate the examinations that lead to board certification. For decades, the ACGME and the RC Surgery have used the rate of success of the first-time takers among a program's graduates on board certification examinations as an important indicator of the ability of that program to educate surgeons.

Case Logs

Surgical programs are not simply accredited or not accredited in a binary fashion. Along with any of the accredited statuses (i.e., Continued Accreditation, Accreditation with Warning, Probationary Accreditation) comes designation of the maximum complement of residents/fellows that the program can have at any given time. The Review Committee must determine whether a program has an adequate volume and variety of operative cases to provide opportunities for that approved complement of residents/fellows to develop the technical proficiency to practice the specialty. For applicant programs this is assessed by review of number of cases of each relevant type done in the clinical sites of the proposed program. For an accredited program, it is assessed by review of the operative cases recorded by the most recent graduates of that program.

Site Visits

Several forms of site visits exist in the Next Accreditation System. Every program is to undergo a full site visit once every ten years. At any point in-between, though, a site visit may be requested by the Review Committee based on a complaint brought by an individual(s) familiar with the program. Much more commonly, though, interim site visits are requested by the Review Committee because the annual review of data from a program has triggered one or more concerns regarding the program's ongoing compliance with the Requirements. Those “data-driven” site visits can be requested to examine a single aspect of the program (“focused site visit”) or the program's compliance with each of the Requirements (“full site visit”).

DISRUPTIONS IN THE ACCREDITATION SYSTEM DUE TO THE COVD-19 PANDEMIC

Like the programs that it accredits, the ACGME has experienced significant disruption as a result of the pandemic. The offices closed on 17 March and all employees have worked from home since that time.⁷ In addition to carrying on all the normal activities of accreditation, ACGME staff have fielded hundreds of questions from individual residents/fellows, individual faculty members, programs and institutions. In response, they have created and posted multiple sets of Frequently Asked Questions. The ACGME Board of Directors, which normally meets 3 times each year, has held virtual meetings at least monthly since March. The Council of Review Committee Chairs, the Council of Review Committee Residents and the Council of Public Members have also met virtually on a much more frequent that normal basis. Numerous policies, procedures and guidance statements have been crafted and posted specifically in response to the pandemic and the disruption it has caused in accredited programs and institutions as well in the tools normally used in accreditation. The in-person meeting of the RC Surgery scheduled for April was held by teleconference.⁸ The RC has “met” by conference call on many occasions in the past but only when agendas were light, typically with specific questions that needed to be answered and few, if any, accreditation decisions to be made. Using audiovisual communication, the Committee in its 23 April meeting was able to reach accreditation decisions on almost all the programs it was set to review. On the postmeeting survey, all members rated the meeting as either “extremely effective” or “very effective” but it was also noted that an in-person meeting would have enhanced discussion of the programs.

Faculty rosters should not be affected by the pandemic. The scholarly activity of faculty members and residents/fellows that will be included in program reviews for 2021 accreditation decisions reflects activity in the 2018-2019 year and so will not be affected by the pandemic. Program reporting of Milestones for the second half of the 2019-2020 academic year has been made optional.⁹ However, there will be substantial alteration in several other tools used by the Review Committee in making accreditation decisions.

Surveys

The ACGME Resident/Fellow and Faculty Surveys are administered annually in 3 tranches. The second tranche of Surveys this year included general surgery and all the subspecialties for which accreditation decisions are made by the Review Committee for Surgery. That survey “window” closed on 15 March. As of that date, fewer than 3500 cases of COVID-19 had been diagnosed in the country. Of the nearly 700 programs under the purview of the RC Surgery, only 5 failed to meet the 70% compliance threshold. However, the receipt of the survey results by programs and the survey data reviewed by the RC Surgery in their 2021 meetings will be impacted by the pandemic, nevertheless. On 18 March, the ACGME announced that the surveys in the third tranche (originally scheduled as 9 March to 12 April) would be suspended.¹⁰ Later announcements clarified that the surveys would remain available until 26 June but that their completion would be optional.^{11 , 12} Programs receive their survey reports after the return and analysis of all completed surveys. Historically, this has occurred in late April or early May such that the data is available for review and use by the program before the end of the academic year. Because of the very late survey close date due to the pandemic, the 2020 survey results will not be released to programs until at least late July. Furthermore, approximately 60 specialties and subspecialties were due to be surveyed in the third tranche. Because of the immediate effects of the pandemic on residents/fellows and faculty members in those specialties and because their surveys were deemed optional, it is anticipated that the response rate will be lower than normal. It is understandable, too, that the responses to those surveys may be skewed by the unprecedented conditions imposed by the pandemic. The survey results for each program have historically been scaled to the national results (i.e., those from every resident/fellow and faculty member in every specialty and subspecialty). Due to the low response rate anticipated in the third tranche (as well as possible skewing), the 2020 reports which are shared with the programs and used by the Review Committee will use for comparison only the other programs in the specialty/subspecialty.¹² Thus, there will be challenges for both programs and the RC in understanding results and trends. Those challenges will be magnified by the fact that the 2020 surveys contained many items that were altered from prior years or entirely new because of new Common Program Requirements having taken effect at the beginning of the 2019-2020 academic year.

Certification Examination Pass Rates

The American Board of Surgery Certifying Examinations (CE) in Complex General Surgical Oncology and Pediatric Surgery were successfully administered in February and March (respectively) before a pandemic was declared. The CE in Vascular Surgery that had been scheduled for May 2020 was cancelled¹³ and on April 22, American Board of Surgery announced cancellation of the June CE in Surgery.¹⁴ However, in a valiant attempt to provide a timely written examination for the candidates (at great expense to the organization in dollars and in time and effort on the part of the staff), the ABS announced on 7 May that the 2020 Qualifying Examination(QE) in Surgery would be administered in July as a web-based, at home examination.¹⁵ Furthermore, they announced on 10 July that web-based QEs in Pediatric Surgery, Vascular Surgery and Complex General Surgical Oncology as well as the Certifying Examination in Surgical Critical Care would be administered in September.¹⁶ Unfortunately, the administration of the Qualifying Examination in Surgery was an abject failure.^{17 , 18} Subsequently, the ABS postponed the QEs in Pediatric Surgery, Vascular Surgery and Complex General Surgical Oncology as well as the CE in Surgical Critical Care that had been planned for September.¹⁹ The American Osteopathic Association website indicates only that Initial Certification examinations will resume when testing facilities allow.²⁰ The certification examination pass rates utilized by the Review Committee are comprised of the number of individuals who passed an examination on first attempt divided by the number of first-time takers. The inability of program graduates to take an examination will not adversely affect a program's pass rate for this year. Unfortunately, the inability of a program's graduates to take examinations may deny the program the ability to improve an extant low pass rate. The full implications of the cancellation of certification examinations in 2020 (including the resulting enormous backlog of eligible candidates in future years) for the examinees, the programs and the Review Committee are yet unknown. But, clearly, the lack of available certification examination results will stand as an obstacle to the RC's ability to confer equitable and consistent accreditation decisions in 2021.

Case Logs

Even though there is no universal agreement on what constitutes an elective operation and the fact that some surgical facilities do not even classify operations in that way, most surgical operations performed by residents/fellows are “electively” scheduled. “Elective” operations were drastically reduced or completely stopped in most sites following guidance of national organizations,3, 4, 5 lack of sufficient testing for COVID-19, lack of personal protective equipment, and other obstacles. The return to performance of elective cases and normal operative volumes has been and will be dictated by the confluence of several local conditions (political, legal, social, and financial) and public health policy. However, it is fair to assume that the number of operations performed by residents/fellows in most programs since March will be much lower than it would have been absent a pandemic and much lower than occurred in those programs in prior years. Remembering that the Review Committee uses only the case logs of the most recent year's graduates for accreditation purposes, the impact of the pandemic on those case logs will be inversely proportional to the duration of the program. If, hypothetically, the period of significant diminution in procedural activity lasted from April through June, that would constitute only 5% of the time that a resident spends in a 5-year general surgery or integrated vascular surgery program. It would be 12.5% of the time that a fellow spends in pediatric surgery, vascular surgery or complex general surgical oncology. However, it would constitute 25% of the very important final year in each of those programs and 25% of the only year in surgical critical care and hand surgery. Furthermore, it can reasonably be assumed that the residents/fellows would have been doing both more and more complex operations during the final 3 months of the academic year. Thus, even those residents/fellows who had relatively small reductions in total case volume likely missed important opportunities to fine-tune their technical skills. In addition, depending on the curricular structure of a program, it is possible that residents/fellows missed their only opportunity to gain substantial operative experience in a specific area. For example, if the residents in a general surgery program perform most of their endocrine operations during a 2-month rotation in the PGY-5 year, a resident scheduled to be on endocrine surgery during May and June may have had the opportunity to do few, if any, endocrine operations.

Site Visits

In-person accreditation site visits for the remainder of the academic year were cancelled as of 9 March⁸ and, at this point, there is no plan to resume in-person site visits before 2021. The ACGME contemplated development of the capacity for virtual site visits long before the pandemic. Preparation and planning for remote site visits have rapidly accelerated since March. Protocols have been written and the Accreditation Field Representatives (“site visitors”) have undergone training in those protocols as well as in the use of multiple audiovisual communications formats. The testing of virtual accreditation site visits will begin this summer and it is anticipated that as many as 200 such visits per month will be possible this fall. In the interim, though, the RC Surgery was unable to reach 2020 accreditation decisions on a small number of programs for which site visit results were unavailable. It remains to be seen whether site visits can be accomplished in all the programs for which they are requested by the RC Surgery for 2021 accreditation decisions. It also remains to be seen whether the virtual site visits will provide the same, improved or diminished quality of evidence as do in-person site visits for the accreditation decisions of the Review Committee.

2021. ACCREDITATION DECISIONS

The mission of the Review Committee is to help programs improve. It does so primarily through feedback to the programs, which takes the form of citations and areas for improvement. However, with resident/fellow and faculty surveys that are different both in content and comparison group, predictably diminished graduate case logs, a paucity of board certification examination results, and site visits that will be accomplished in an unprecedented fashion, if at all, the Review Committee will be faced with unprecedented challenges to confer equitable and consistent accreditation decisions in 2021.

When examining a program that appears to have one or more deficits in its substantial compliance with the Requirements, the Review Committee will have 2 lenses through which to further examine that program. The first lens for supplementary examination is the history of that program. For example, if 3 of the 6 2019-2020 graduates of the program failed to meet all case log minima but the history revealed that the graduates of that program had routinely far exceeded all case log minima in the past, it is unlikely that the Review Committee would view that program as falling short of accreditation standards. That historic lens is always available to the RC (at least for well-established programs) and is used with regularity. The second lens has been designed specifically for use in the 2021 accreditation decisions as a result of the COVID pandemic. The 2020 ADS Annual Update will contain a series of questions that can help the RC understand the magnitude and duration of the effects of the pandemic on a program.²¹ For surgical programs, questions designed to characterize the impact of the pandemic on elective operations performed by residents/fellows are included. Beyond responding to the prescribed questions, the program may, of course, provide greater detail on the impact of the pandemic in the “Major Changes and Other Updates” section of the update.

SUMMARY

The Review Committee for Surgery is responsible to the residents/fellows and the programs but, most importantly, to the public to confer accreditation decisions that are honest, accurate, equitable and consistent. Accreditation decisions in 2021 will necessarily be made with information based on surveys that contain changed or altogether new items and scaled in a never before used manner; a small fraction of the board certification pass rate data that is usually available; site visits performed using novel methodology, if done at all; and, graduate case logs that presumably accurately represent the full experience of residents/fellows but with that full experience diminished by the pandemic. Those decisions will be tempered by general knowledge of the impact of the pandemic, personal knowledge of the effects of the pandemic on the RC members’ home institutions, and specific knowledge of the pandemic's effects on the reviewed programs and will be guided by the knowledge that a program cannot be held responsible for a pandemic that is far beyond its control.