An Early Assessment of the ACGME CLER Program: A National Survey of Designated Institutional Officials

Abstract

Background

The Accreditation Council for Graduate Medical Education (ACGME) Clinical Learning Environment Review (CLER) is an innovative element of the ACGME's new accreditation system. To date, little information has been collected regarding the value of CLER.

Objective

The purpose of this study is to collect information on designated institutional officials' (DIOs') perspectives about the initial CLER visits conducted at their institutions.

Methods

The authors created and distributed a survey to DIOs about their initial CLER visits. Demographic data were compared across survey responses with Spearman's rank correlation and the Kruskal-Wallis test.

Results

The authors received responses from 63% of DIOs (186 of 297) at institutions that participated in the initial CLER visits, with 88% (164 of 186) having served as DIO during the visit. Seventy-two percent (114 of 158) reported institutional changes to address CLER focus areas prior to the visit, yet only 32% (51 of 157) reported that additional resources were allocated to these areas after the site visit. Sixty-five percent (102 of 156) reported institutional executive leadership was positive about participating in CLER; 85% (134 of 158) reported that ACGME conducted the visits efficiently; 84% (133 of 158) reported that the site visit accurately assessed the institution's performance in the CLER focus areas; and 60% (93 of 156) reported CLER provided high-value information.

Conclusion

Survey results from DIOs suggest that CLER is an effective mechanism to improve the learning environment. Common concerns included limited advance notice for the site visit and disruptions of clinical practice.

Introduction

The Accreditation Council for Graduate Medical Education (ACGME) developed and implemented the Clinical Learning Environment Review (CLER) as an innovative process to evaluate the clinical learning environment at institutions that sponsor ACGME-accredited residency and fellowship programs as part of their new accreditation system.¹ The CLER program goals include the following: (1) to provide periodic feedback on patient safety, health care quality, care transitions, supervision, duty hours and fatigue management/mitigation, and professionalism to ACGME-accredited institutions, and (2) to encourage the clinical sites to use the feedback received from the CLER program to improve trainee engagement in learning.² A key aspect of CLER is extending responsibility for graduate medical education (GME) more directly to executive leaders who control sponsoring institution (SI) resource distribution. During its first phase, ACGME completed CLER visits at 297 SIs and published national comparative data.³

The ACGME requires appointment of a designated institutional official (DIO) at each SI, and assigns the DIO responsibility to oversee and govern ACGME-accredited programs the institution sponsors.⁴ The DIO plays a key role in CLER and during the CLER site visit.

The primary aim of this study is to obtain feedback from DIOs about the initial CLER visits. The study addresses (1) institutional changes stimulated by CLER prior to the initial site visit; (2) the institutional impact of the CLER visit on clinical workflow; (3) institutional changes in response to the CLER visit; (4) institutional changes stimulated by release of national CLER data; and (5) DIO perceptions of the value of CLER. Study results could inform efforts to improve CLER value in a continuous quality improvement cycle.

Methods

The authors identified SIs on the ACGME website and developed a survey (provided as online supplemental material) seeking feedback from DIOs about their CLER visit.⁵ An author (K.R.F.) developed a draft survey that all coauthors independently reviewed and edited. All authors conducted a second review of the revised survey, and the survey was subsequently finalized. The authors distributed the survey to the DIO of each SI with 3 consecutive weekly reminders to nonresponders. The survey included questions about demographics, the CLER process, and the impact of CLER. The authors selected the research electronic data capture (REDCap) tool hosted at Mayo Clinic for survey distribution, and used a 4-point Likert scale to score responses.⁶ Likert scale options chosen were strongly agree, agree, disagree, and strongly disagree. The authors selected a 4-point Likert scale to mandate participants commit to a positive or negative response.

Demographic comparisons between DIO tenure, the number of ACGME-accredited programs at each institution, and CLER survey responses were assessed using Spearman's rank correlation. The authors compared survey responses between regions using the Kruskal-Wallis test and used the Wilcoxon rank sum test for pairwise comparison of questions that varied significantly by region. A P value ≤.05 was considered statistically significant. Template analysis was used for the thematic analysis of open-ended comments provided by respondents.⁷ The analytical process was led by an author (K.R.F.) with reiterative analysis and input from coauthors.

The Mayo Clinic Institutional Review Board deemed this study exempt.

Results

At the time of our survey, 812 DIOs were identified. Survey results were considered from only the 297 DIOs at SIs where initial CLER visits were conducted.³ Responses were received from 63% of these DIOs (186 of 297), 88% of whom (164 of 186) served as DIO during the initial CLER visit. Demographic data for the DIOs are in Table 1.

Table 1.

Demographics of DIOs Who Participated in CLER Visit (N = 164)

The CLER visit preparation data are in Table 2. Minor differences are present in the denominators of these data as not all respondents answered each question. A total of 72% of respondents (114 of 158) agreed or strongly agreed that their institution made changes to address CLER prior to the visit. However, 68% (106 of 157) agreed or strongly agreed that their SI allocated no additional resources to do so. Of the respondents, 44% (68 of 156) agreed or strongly agreed that participants in the CLER visit were coached in advance. Furthermore, 65% of DIOs (102 of 156) agreed or strongly agreed that their executive leadership was positive about participating in CLER. The size of the SI (the number of ACGME-accredited programs sponsored) was negatively correlated with the statement, “The C-suite of my SI was enthusiastic about the opportunity to participate in the CLER visit” (r = –0.16, P = .05).

Table 2.

Sponsoring Institution CLER Visit Preparation

The CLER visit process data are presented in Table 3. A total of 85% of DIOs (134 of 158) agreed or strongly agreed that the ACGME conducted the CLER visit efficiently; 84% (133 of 158) agreed or strongly agreed that CLER accurately assessed performance in the CLER focus areas; and 60% (93 of 156) agreed or strongly agreed that CLER provided high-value information. Of the respondents, 45% (70 of 157) agreed or strongly agreed that the CLER visit significantly disrupted clinical activities. Larger SI size was negatively correlated with the statement, “The CLER visit was conducted efficiently” (r = –0.18, P = .025) and positively correlated with the statement, “The CLER visit significantly disrupted clinical activities at my institution” (r = 0.24, P = .003). Responses to the question about clinical disruption also varied by region (P = .027). The rank sum test for each pairwise comparison demonstrated that the West region varied significantly from the South (P = .008) and Northeast regions (P = .028), with the West region more likely to report disruption.

Table 3.

Designated Institutional Official CLER Visit Feedback

The CLER visit feedback data are in Table 4. Of the DIOs, 71% (109 of 154) agreed or strongly agreed that the CLER report was easy to interpret, and 74% (112 of 152) agreed or strongly agreed that the report provided meaningful information. Additionally 44% of DIOs (68 of 154) agreed or strongly agreed that the CLER report identified previously unknown areas for improvement. Of the DIOs, 70% (108 of 154) agreed or strongly agreed that the CLER report motivated SI clinical learning environment improvement, 59% (90 of 152) agreed or strongly agreed that the report motivated exploration of new SI initiatives, and 60% (92 of 154) agreed or strongly agreed that the CLER report motivated increased GME and executive leadership communication. However, 71% of DIOs (109 of 154) agreed or strongly agreed that no additional resource allocation was stimulated by CLER. A total of 83% of DIOs (128 of 154) agreed or strongly agreed that adding a review of institutional quality and safety outcome data would improve the value of CLER, and 58% of DIOs (88 of 153) agreed or strongly agreed that they were confident national CLER data provided valid comparisons. Overall, 73% of respondents (113 of 155) agreed or strongly agreed that their initial CLER visit positively impacted the clinical learning environment.

Table 4.

Designated Institutional Official Impressions After CLER Visit

Tenure of the DIO was negatively correlated with agreement with the statements, “The CLER report motivated my SI to improve the clinical learning environment” (r = –0.22, P = .007) and “The CLER process increased communication/cooperation between GME and the C-suite” (r = –0.22, P = .005). Size of the SI was negatively correlated with the statement, “The CLER process positively impacted the clinical learning environment at my institution” (r = –0.19, P = .019).

A total of 41% (68 of 164) of respondents provided comments, with 29% of commenting DIOs (20 of 68) expressing concern that the short notice of the CLER visit disrupted patient care (Box 1). Representative comments on the impact of CLER are included in Box 2. Several DIOs noted that CLER is early in development and should be viewed as an iterative process. One reported the “disruption of clinical services (eg, clinics being canceled on short notice) to be in contrast to the patient-centered approach espoused by the ACGME.” One respondent suggested that short notice was a particular concern at Veterans Affairs sites due to scheduling constraints. Several respondents strongly supported CLER despite reporting disruption to patient care. For example, a respondent stated, “Good basic concept and long overdue. Needs continuous refining as each SI is a little different in needs and resources but overall positive.” Some comments distinguished between CLER content and process. For example, a respondent was positive about the goals of CLER but described the visit process as “a step backward.” A similar comment stated that the “goals of CLER are outstanding and deserve everyone's support. If the process of the CLER visit could be made less disruptive . . . the information would be greatly appreciated.”

What was known and gap

The Clinical Learning Environment Review (CLER) program has conducted nonaccreditation visits with feedback for approximately 3 years, yet little is known about designated institutional officials' (DIOs') perceptions of the program.

What is new

A survey assessed DIOs' perceptions of CLER.

Limitations

Data relied on DIO self-report and may be subject to response and recall bias.

Bottom line

DIOs reported that CLER is an effective mechanism to improve the learning environment. Limited advance notice and disruptions of clinical practice were common concerns.

Box 1 Representative Comments Addressing Preparation Time

• CLER visits are short notice and very disruptive.

• Stop short notice visits! This is about culture. No amount of prep can change culture!

• Our institution found the disruption of clinical services (eg, clinics being canceled on short notice) to be in contrast to the patient-centered approach espoused by the ACGME.

• Not worth the time and resources invested in the process; the 2-week notification before visit is way too short a time to arrange all C-suite schedules for availability.

Box 2 Representative Comments on the Impact of CLER

• Our C-suite has become involved in and acknowledges that the feedback from CLER might assist us in making global improvements for the learning environment but also for patient care.

• SI created a new department of quality post-CLER.

• The CLER program has highlighted the importance of resident participation in quality and patient safety activities.

• The CLER visit highlighted the focus areas for the C-suite. This has been important for increased incorporation of training programs and trainees in policy/administration/quality.

• The CLER process was valuable in integrating GME and C-suite more quickly.

• The single most important element has been the conversation and increased collaboration between GME and the hospital patient safety and quality divisions to improve patient care and education of our trainees.

• I think the CLER visit is valuable, and a tool that has helped the C-suite understand the value of GME in a completely different fashion than the historical “workforce” value perspective.

• The best part of CLER is the expectations and goals that drive the integration of residents into our health care system.

• The CLER is a positive process, but we need better feedback.

• I feel the review awakened important awareness of many aspects of our learning environment and affirmed how aligned our organizational infrastructure is with GME activity.

• It has directed residency training to focus on patient-centered care, and patient safety and quality.

Discussion

Many DIOs in our survey reported that their initial CLER site visit effectively assessed and often improved the clinical learning environment at their SI. The most frequent concerns identified were disruption to clinical practice and challenges related to short visit notification. Improvements in the clinical learning environment related to initial CLER site visits were often accomplished without additional allocation of resources.

These findings represent a small sample of the national DIO cohort (297 of 812) since the initial round of CLER site visits was limited to larger SIs. Consequently, our findings may be biased and may not represent the total population of DIOs. The results of our survey are largely consistent with those of Koh et al,⁸ who reported increased focus on patient safety and quality improvement, increased focus on resident supervision, and improved mechanisms for procedural competency assessment as results of CLER. Positive CLER outcomes included provision of formative feedback, raised awareness of GME's role in patient care, increased attention to CLER focus areas, and promotion of GME and executive leadership engagement. The primary suggestion for improvement was similar, as DIOs expressed a need for more time to prepare for the visit.

Demographic comparisons suggest that DIOs with shorter tenures and those at smaller SIs reported a more positive CLER impact. For example, DIOs with shorter tenures were more likely to agree or strongly agree that CLER motivated their SI to improve the clinical learning environment and that CLER increased communication between GME and executive leadership. The DIOs at larger SIs were less likely to report that CLER was conducted efficiently and more likely to report disruption of clinical activities.

This study has several limitations. The survey was reviewed by those with content expertise, but it has no other evidence of validity. As a result, interpretation of survey questions may have differed from the authors' expectations. Given the low percentage of the total population of DIOs who responded, response bias is likely. There were minor differences in response rates among survey questions. Our findings are limited to feedback from DIOs. Program directors, faculty, residents, fellows, nurses, allied health staff, executive leaders, and quality and safety officers may have different perspectives.

Next research steps should include correlating SI CLER data with widely available SI safety and outcome data. Integrating data from CLER with other datasets, including ACGME anonymous resident and faculty survey data, could also provide a more complete and instructive assessment of the SI clinical learning environment.

Conclusion

Many of those who responded to this anonymous DIO survey support CLER as an effective mechanism to assess and improve the clinical learning environment, to stimulate continuous improvement, and to increase communication among GME and executive leaders. Concerns identified included disruption to the clinical practice and logistical challenges related to the short notice before visits. Our data suggest that improvements in the clinical learning environment were often not associated with additional SI resource allocation.