A Novel, Web-Based Quality Improvement Platform to Address ACGME CLER Requirements

Abstract

Background

In 2014, the Accreditation Council for Graduate Medical Education (ACGME) formally mandated trainee (resident and fellow) participation in health care quality improvement (QI) projects as one of the Clinical Learning Environment Review (CLER) Pathways to Excellence. Subsequent national reviews showed large variations in how QI education is conducted, as well as a significant mismatch between educational and organizational goals.

Objective

We developed a web-based platform to engage trainees in QI that better aligned with best practice methodology and matched identified institutional priorities.

Methods

A needs assessment survey was distributed to trainees to understand the obstacles to compliance with ACGME QI requirements. Based on the results, a web-based clearinghouse, called the QI Platform, was developed and launched in July 2016, and utilization was analyzed in February 2019.

Results

A total of 196 of 440 needs assessment surveys (45%) were completed. Themes extracted from surveys to identify barriers in QI participation included difficulties designing projects, lack of mentorship or expert support, and difficulty engaging an interprofessional team. Over 2.5 years, 151 projects were registered on the platform. Of these, 17 (11%) were collaborative entries. At the time of analysis, 166 of 437 trainees (38%) were listed as participants in active QI projects. A total of 22 projects were archived as complete, and 68 incomplete projects were reassigned to the “Ideas” section as works in progress after lead trainee graduation.

Conclusions

An institutional QI Platform clearinghouse for GME QI projects was feasible to develop and maintain, and it appeared acceptable to most GME programs and trainees for recording and tracking QI projects, and linking these to hospital QI priorities.

What was known and gap

Two reports from the CLER Pathways to Excellence have identified shortcomings in quality improvement (QI) education and a significant mismatch between educational and organizational goals.

What is new

A web-based platform to engage residents in QI projects, which better aligns with best practice methodology that matches identified institutional priorities.

Limitations

The needs assessment may not have captured the needs of the entire graduate medical education (GME) community and may have favored the needs of those most interested in QI.

Bottom line

An institutional platform for QI projects was feasible to develop and maintain, and appeared acceptable to GME programs and trainees for recording and tracking QI projects, and linking these to hospital QI priorities.

Introduction

Quality improvement (QI) can be defined as “systematic and continuous actions that lead to measurable improvement in healthcare services,” with the aim of achieving a safer, more efficient, cost-effective health system.^1,2 Education of the next generation of physicians is key to achieving a high-quality health care system.^3,4

To address the key role of educating physicians in QI methodology, the Accreditation Council for Graduate Medical Education (ACGME) mandated trainee (both resident and fellow) participation in QI projects in a 2012 update of the Common Program Requirements.⁵ In 2014, the ACGME Clinical Learning Environment Review (CLER) Pathways to Excellence outlined QI as 1 of 6 key pathways.⁶

Despite these new requirements, shortcomings have emerged in 2 comprehensive national reviews of the CLER Pathways. Identified weaknesses included (1) underinvestment in resources and education in patient safety and health care quality; (2) organizational strategic goals were often not coordinated with educational efforts in graduate medical education (GME); and (3) lack of interprofessional collaboration.^7,8

With this project, we aimed to identify local barriers to learner QI research participation, and to develop a tool that captured QI participation, improved interprofessional cooperation, and aligned projects with institutional priorities while also offering a methodologically appropriate template and resources to supplement expert oversight.

Methods

Starting with a formal needs assessment, we followed a systematic process to design, implement, and evaluate a web-based clearinghouse for QI projects in our institution.

Phase 1: Development Committee and Needs Assessment

The George Washington University (GWU) oversees approximately 450 trainees in 42 training programs at the GWU Hospital, a 431-bed, urban, quaternary care hospital. In 2014, a development committee was formed as a GME subcommittee to address implementation of the ACGME's Common Program Requirements and CLER Pathways. It included program directors, residents, QI leaders, a research fellow in informatics, and an institutional programmer.

The group developed a needs assessment survey (SurveyMonkey, Palo Alto, CA), which was distributed to residents and fellows within GWU. A mix of multiple-choice and free text questions (provided as online supplemental material) explored themes of exposure and attitudes to QI (10 questions), barriers to initiating and completing QI projects (5 questions), familiarity with institutional QI goals (3 questions), and knowledge of QI methodology (3 questions).

We gathered evidence for response process validity evidence by reviewing survey questions with resident focus groups, and for content validity by GME committee review. Several questions were removed after these reviews. Analysis of responses was conducted by 2 authors (J.S.B and G.H.), extracting themes from open-ended comments. An independent coder method was used. Themes were determined by consensus.

Phase 2: Platform Design

Based on the needs assessment, a medical informatics fellow and a computer programmer developed a web-based platform on Adobe ColdFusion (Adobe Systems Inc, San Jose, CA) and Microsoft SQL Server (Microsoft Corp, Redmond, WA). It used the open source projects jQuery (jQuery Foundation, Los Angeles, CA) and CKEditor (CKSource, Warsaw, Poland). It currently runs on a Microsoft IIS web server. Permissions included licensing fees and adherence to open source licenses. Programming from initial pilot to first stable version, including server and database setup, took approximately 400 hours. Afterward, technical maintenance was minimal.

Each trainee entering the program was assigned a unique log-in to access the database via a web-based portal, the QI Platform. Faculty, administrators, and other stakeholders were granted unique log-in access to serve as project mentors or site moderators.

A list of resources relevant to QI was developed and posted on the QI Platform, including site-specific instructional videos and guides and national curricula and templates.^9,10 These resources are reviewed at least annually and are updated by QI leaders on the development committee.

Trainees could create a project, modify a project for which they were assigned as a team member, and view the title, progress, lead trainee, faculty mentor, and department affiliation for all projects entered in the database (provided as online supplemental material). A faculty mentor had to be selected in order for a project to be created. Contact details for trainee and faculty leads could be readily obtained from the system to promote collaboration. Members of the team who were not in the database (ie, allied health professionals) could be added in a free text field to ensure transparency of team membership. There was a universal health care provider log-in that allowed allied health care professionals to log into the site and contact team leaders for potential collaborations.

Confidentiality was an important consideration. At the log-in webpage, a confidentiality notice stated that regular oversight was conducted by the associate dean for GME and designees, and all data entered were confidential under peer-review protections according to relevant statutes. Project details, beyond those listed above, were only available to users listed as team members, regardless of project status.

The interface invited new users to attach at least 3 short descriptor “tags (#)” to their project, which were populated from a generic list of potential QI topics. This list evolved over time, as each new project could include unique tags. In addition, hospital-designated “priority” tags were highlighted for selection and updated annually. Tags could be individually interrogated to review current projects aligned to hospital priority areas.

When setting up a project, trainees were guided through a 6-step process with text fields that required sequential completion (layout of project entry field provided as online supplemental material).

• ▪Step 1: Identify the problem
• ▪Step 2: Refine the problem
• ▪Step 3: Timeline
• ▪Step 4: Institutional Review Board (IRB) approval (or waiver, if applicable)
• ▪Step 5: Implementation—PDSA (plan, do, study, act); Lean Six Sigma (define, measure, analyze, improve, control); or other
• ▪Step 6: Final report

A project completion status bar turned from red (not started) to yellow (in progress), and then to green (completed) for each section, revealing project status updates to all users. As text fields were entered for each step, methodology and examples were provided to guide the trainee. All members of the research team, including faculty supervisors, could log into the detailed view of the project to review content entered and offer feedback, edits, and oversight. Successful completion was at the discretion of the faculty mentor and program director.

Once all steps were completed, the project moved to the “Archive” section, where it was available for retrieval, review, and aggregate analysis of outcomes by authorized parties. If the lead trainee left the institution prior to completion of the QI project, the project automatically moved to an “Ideas” tab. Trainees could view incomplete projects in the Ideas tab, along with their status of completion, and apply to reactivate the QI project as the new lead trainee.

Phase 3: Implementation

In 2015–2016, the platform was piloted with 8 residents who provided direct feedback on the user interface and project entry experience. The QI Platform was modified accordingly and made available to 2 residency programs with 132 trainees and 5 faculty members for further testing. Following successful launch of the QI Platform with these early adopters, it opened to the remaining training programs in July 2016. Information about it was broadly advertised to the GME committee for program directors and to the residents and fellows at institutional grand rounds. In year 2, program degree of participation was noted on annual program review scorecards, which were discussed with the associate dean for GME.

Phase 4: Evaluation

An analysis of QI Platform utilization was conducted in February 2019. The platform was interrogated for number and status of projects registered, number of tags in use, and trainee utilization patterns.

This study was determined exempt by the IRB at the GWU Office of Human Research.

Results

Needs Assessment (Phase 1)

Assessment surveys were distributed to all residents and fellows (440 trainees) at GWU. A total of 196 responses were obtained (45% response rate). Trainee demographics were not collected. A total of 132 trainees (67% of respondents) were currently involved in QI projects. Thirty-three of 196 respondents (17%) reported insufficient QI education. Reasons for dissatisfaction with QI training were collected from open-ended question responses (provided as online supplemental material).

Barriers to initiating QI projects included data collection issues (51%, 76 of 150) and “project too ambitious” (22%, 33 of 150). Other barriers included lengthy IRB approval times, lack of departmental cooperation, and lack of technical assistance (eg, statistician input; Figure 1).

Figure 1.

Needs Assessment: Barriers to Initiating and Completing Quality Improvement Projects (n = 150)

Of the 175 respondents who answered the section on familiarity with institutional QI goals, 161 (92%) had not read the hospital QI plan, and 146 (83%) did not know where to find it. PDSA methodology knowledge was more familiar to respondents than Lean Six Sigma methodology (44% [73 of 165] versus 14% [23 of 165]), although fewer than half felt comfortable with either.

Utilization Evaluation (Phase 4)

The QI Platform was launched in July 2016. As of February 2019, 61 projects were active, 68 were inactive, and 22 were complete. In total, 151 projects have been registered on the platform since its inception (Figure 2). Seventeen (11%) are registered as interprofessional collaborations. There are 166 residents (38% of total) currently involved in one or more active projects.

Figure 2.

Quality Improvement Projects Registered on Platform by Month (as of February 2019)

There are currently 853 “tags” registered on the system, 33 of which represent hospital priorities. A total of 140 hospital priority tags have been linked to 56 unique projects.

Discussion

Following an institutional needs assessment of residents and fellows to develop priorities, a web-based QI Platform characterized QI projects across all GME programs while allowing linkage to hospital priority areas. The platform promoted identification of team members, templating of common QI methods, centralized access to resources, and institutional memorialization of projects in an online archive. Usage over the past 2.5 years supports buy-in from GME programs, with 151 projects entered and 166 (38% of all residents) participating.

There are currently a variety of approaches GME programs take to incorporate QI education per ACGME requirements, including establishing bespoke curricula, quality committees, or elective time.^11–13 The QI Platform is not meant to replace these important elements of a successful QI program. Rather, it provides a central system from which to coordinate and nurture QI education, using an asynchronous, web-based platform to bridge the gap between didactic and experiential learning in the clinical environment.

Trainees identified inadequate strategic planning as a large deterrent to engaging in QI; therefore, the QI Platform emphasized several elements of project creation, including creating a title, listing team members, suggesting tags, identifying the problem (aims statement, measures, intervention), refining the problem (stakeholders, barriers, resources), creating a timeline, and addressing IRB considerations. Lack of expertise in QI methodologies was also identified during the needs assessment. Accordingly, resources and templates for implementing different steps in QI methodology with examples were embedded in the platform, such as PDSA and Lean Six Sigma cycles. Finally, trainees identified lack of time as another concern for QI project participation. In a prior study, we reported that when using information entered into the platform as a checklist, most QI projects qualify for automatic exemption from IRB review without the need for a separate IRB application.¹⁴

Ensuring data confidentiality was important in designing this platform, as QI often develops from a system failure, and the data protection clause had to be compliant with state (in this case district) regulations. As these laws differ across states, constructing a nationally shared platform could prove difficult. Further, the platform was intentionally constructed to limit visibility of project details (only title, project lead, and status of completion) for those with platform log-in credentials, but not affiliated with the project.

These findings are limited, as the needs assessment may not have captured the needs of the entire GME population. Respondents to the needs assessment may have been those who were more interested in QI. The institutional-specific nature of the IRB process may impart unique delays to outside institutions, and state-specific regulations surrounding peer review may impose additional restrictions on oversight and platform implementation. Nevertheless, we have constructed the QI Platform with open-source programming such that interested institutions can, upon request, obtain access to the source code and tailor the platform to institution- and state-specific requirements in building a local solution.

After several years of use, next steps for the QI Platform include conducting focus groups with faculty and trainees to determine barriers to access for programs and suggestions for site improvement. In addition, we hope to disseminate the platform broadly to other institutions and evaluate completed projects for trends and sustainability.

Conclusions

An institutional QI Platform clearinghouse for GME QI projects was feasible to develop and maintain, and it appeared to be acceptable to a substantial number of GME programs and trainees for recording and tracking QI projects, and linking these to hospital QI priorities.