Lessons Learned From a 5-Year Experience With a 4-Week Experiential Quality Improvement Curriculum in a Preventive Medicine Fellowship

Abstract

Background

Competency in practice-based learning and improvement (PBLI) and systems-based practice (SBP) empowers learners with the skills to plan, lead, and execute health care systems improvement efforts. Experiences from several graduate medical education programs describe the implementation of PBLI and SBP curricula as challenging because of lack of adequate curricular time and faculty resources, as well as a perception that PBLI and SBP are not relevant to future careers. A dedicated experiential rotation that requires fellow participation in a specialty-specific quality improvement project (QIP) may address some of these challenges.

Method

We describe a retrospective analysis of our 5-year experience with a dedicated 3-week PBLI-SBP experiential curriculum in a preventive medicine fellowship program at Mayo Clinic, Rochester, Minnesota.

Results

Between 2004 and 2008, 19 learners including 7 preventive medicine fellows participated in the rotation. Using just-in-time learning, fellows work together on a relatively complex QIP of community or institutional significance. Since 2004, all 19 learners (100%) participating in this rotation have consistently demonstrated statistically significant increase in their quality improvement knowledge application tool (QIKAT) scores at the end of the rotation. At the end of the rotation, all 19 learners stated that they were either confident or very confident of making a change to improve health care in a local setting. Most of the QIPs resulted in sustainable practice improvements, and resultant solutions have been disseminated beyond the location of the original QIP.

Conclusion

A dedicated experiential rotation that requires learner participation in a QIP is one of the effective methods to address the needs of the SBP and PBLI competencies.

Introduction

Training in practice-based learning and improvement (PBLI) aims to provide learners with the knowledge and skills to systematically evaluate patient care and implement changes aimed at providing higher-quality patient care. Training in systems-based practice (SBP) aims to develop resident skills in systems thinking, team collaboration, health care financing, safety, and patient advocacy.¹ A comprehensive knowledge and understanding of both these competencies empowers residents and fellows with the skills necessary to successfully plan, execute, and lead health care systems improvement efforts in the future.

These competencies are especially pertinent in the light of the Institute of Medicine (IOM) reports. In its reports, To Err is Human and Crossing the Quality Chasm, the IOM brought to attention the major impact of medical errors and systems flaws on morbidity and mortality in the United States.^2,3 In a subsequent report, Health Professions Education: A Bridge to Quality, the IOM recommended the incorporation of quality improvement (QI) training in medical education to empower health professionals with the skills necessary for improvement efforts towards providing safe, high-quality, patient-centered care.⁴ In the United States, the Accreditation Council for Graduate Medical Education (ACGME) requires all residents and fellows to be competent in PBLI and SBP, both of which are related to QI.⁴

Some of the strategies that have been described to teach PBLI include a 6-week curriculum in the setting of an ambulatory rotation during which surgical residents created suggestions for future QI efforts;⁵ didactics and clinical improvement projects for family medicine residents;⁶ the use of journal clubs in an ophthalmology residency;⁷ and didactics, self-reflection, and medical chart audits in an internal medicine residency.⁸ Strategies used for teaching SBP include the use of didactics, small group discussions, and grand round presentations in the setting of a plastic surgery residency;⁹ didactics and reflective analyses on resident-identified clinical cases for which systems issues contributed to inadequate delivery of patient care in an internal medicine residency;¹⁰ and small group discussions on systems as part of discussions in hematology journal clubs.¹¹

Experiences from graduate medical education programs describe the implementation and assessment of PBLI and SBP as challenging.^12,13 A lack of adequate time for the curriculum in the setting of a busy clinical schedule,^14,15,16 barriers to the completion of QI projects,⁵ and a perception that training in SBP and PBLI is not relevant to future careers are some of the common challenges faced by program directors in implementing SBP-PBLI curricula.

An experiential rotation dedicated to training in SBP and PBLI, through completion of a QI project (QIP) relevant to the training program, may be a possible solution to these challenges. On the basis of our 5-year experience with a preventive medicine fellowship with such a curriculum, we provide educators practical tips and methods to plan and execute such an experiential QI curriculum. We believe that a similar approach would provide other graduate medical education programs an effective method to teach QI and meet the needs of the SBP and PBLI competencies.

The QI Curriculum

Setting

A dedicated 3-week QI experiential rotation in the second (practicum) year of the preventive medicine fellowship provides the fellows experiential training in PBLI and SBP. Since the fellowship has a maximum of 3 fellows at any one time, this rotation has also been made available to other learners as an elective rotation to facilitate multidisciplinary team projects. Between 2004 and 2008, 19 learners participated in the QI rotation (table 1, column 3).

Table 1.

Quality Improvement Projects Conducted by Learners, 2004–2008

Faculty and Teaching Methodology

The preventive medicine program director, who also functions as the director of the QI rotation, partners with institutional and community leaders, as well as institutional experts in QI and health care systems, to facilitate learning. Didactics, small group discussions, case-based learning, brainstorming, problem solving, and participation in a QIP are the commonly used teaching methodologies. The specific curricular topics that address PBLI and SBP are as shown in table 2 and consist of subtopics related to the basics of QI, quality measurement, process and root cause analysis, medical errors and patient safety, evidence-based medicine, leadership, health care financing, and team collaboration.^17,18 Sessions on insurance systems, health care financing, strategic planning, and team collaboration, in addition to the QIP, contribute to fellow competence in SBP.

Table 2.

Curricular Content

Emphasis is placed on experiential learning that occurs through fellow participation in a relatively complex clinical practice or community systems issue that reinforces the practical application of knowledge and skills learned as part of the curriculum. The schedule is specifically designed to facilitate “just-in-time” learning.¹⁹ For example, if the theory and case exercises on process mapping are scheduled in the morning, they are reinforced in the afternoon when fellows perform the process mapping required for the QIP.

The QI Project

Before the start of the experiential QI curriculum, the director of the rotation identifies the QIP in conjunction with relevant division leadership. The factors that are considered in choosing a project include its relevance to preventive medicine, significance to patients and the learners, scope for improvement, and feasibility for completion within the duration of the rotation. The framework used by the fellows to conduct the QIP is as shown in the figure. The program director completes Steps 1 to 4 for the chosen project before the start of the QI rotation. This preliminary inquiry identifies potential challenges and possible solutions and helps in effectively guiding the fellows during the actual execution of the project, regardless of the issue at hand.

Figure.

Steps used to Conduct QI Projects in the Mayo Preventive Medicine Fellowship

Stakeholder identification, benchmarking, literature review, process mapping, root cause analysis, and Nolan model²⁰ are critical steps undertaken by the fellows in the first week of the rotation to develop improvement strategies. These improvement strategies are then implemented by using multiple plan-do-study-act (PDSA) cycles in the next 2 weeks. The PDSA method involves a “trial and learning” methodology during which a hypothesis or suggested solution for improvement is made, tested, carried out on a small scale (eg, with 5–10 patients), and evaluated before any changes are made to the whole system (figure). This logical 4-stage sequence is carried out over a course of repeated small cycles, which eventually leads to exponential improvements.^20,21 In the last week of the rotation, the learners evaluate the cost-effectiveness of their final recommended solution to ascertain if it should be disseminated to the rest of the project site.

The QIPs conducted by the learners during the last 5 years are summarized in table 1 and describe the impact that most of these project have had on patient care. Many of the solutions derived as a result of the QIPs have been disseminated beyond the site of the pilot project, as summarized in the table. Through the participation of fellows in a QIP, the curriculum provides just-in-time training, with practical insights into leadership, health care management, and systems improvement. In programs like ours, with 7 or fewer learners per year of training, group QIPs provide the additional advantages of team training and providing adequate number of personnel and resources to complete the project in a short time frame.

Learner Assessment

Current methods to assess PBLI consist predominantly of end-of-rotation global ratings, performance in the PBLI exercise, and participation in the QIP. In addition, knowledge, skills, and attitude in QI are assessed through the Quality Improvement Knowledge Application Tool (QIKAT).²² The QIKAT contains a total of 6 scenarios (3 each for the preintervention and postintervention tests) in which learners are asked to provide short answers to describe the aim, measures, and recommended changes for an improvement case provided to them. Current methods to assess SBP consist predominantly of quarterly rating of fellows via 360-degree evaluations from faculty, allied health staff, and patients,²³ as well as end-of-QI rotation global ratings that consider the fellow's skills in team collaboration and systems thinking in the implementation of the QIP. The 360-degree evaluations include narrative and quantitative evaluation of the fellows as it relates to their interpersonal relationships, timeliness, approachability, process efficiencies, patient advocacy, commendations, improvement opportunities, cost-effective practice, and concerns.

In 2007, in an effort to enhance assessment of the situational awareness and noncognitive skills needed in SBP and PBLI, we designed and conducted a pilot, 8-station objective structured clinical examination (OSCE) to assess competency in the major components of PBLI and SBP. Each station was designed to assess a component of the curriculum including prescription errors, negotiation, evidence-based medicine, team collaboration, root cause analysis, quality measurement, Nolan model, and insurance systems. Fellow performance in each station was assessed via checklists and global competency scales.²⁴

Curricular and Project Outcomes

Since 2004, all 19 learners (100%) participating in this rotation have consistently demonstrated a statistically significant increase in their QIKAT scores at the end of the rotation, with significant increase in knowledge by the end of the rotation.^17,18 As described earlier in this article, an 8-station OSCE was conducted to evaluate the 9 learners participating in the 2007 QI rotation. Summative results indicated that 1 fellow did not pass the insurance systems station and the Nolan station, and another fellow did not pass the insurance systems station and the quality measurement station. In addition, 2 other fellows did not pass the Nolan station.²⁴

The practical impact of the knowledge, skills, and attitudes of the rotation and learning on the learners may be evaluated through the outcomes of the QI project, which have all been consistently successful (table 1, column 6). Another marker of the success of these projects is that most have been disseminated beyond the site of the project (table 1, column 7). All 19 learners have expressed high or very high satisfaction with the rotation. In fact, the QI rotation is consistently one of the highest rated rotations in the preventive medicine fellowship. At the end of the rotation, all 19 learners stated that they were either confident or very confident of making a change to improve health care in a local setting.^17,18

Challenges

Regardless of the various teaching and assessment methodologies, significant challenges exist in the incorporation of the SBP and PBLI competencies in the setting of a training program. Some of the challenges that we faced included coordinating schedules of interdisciplinary learners, creating time in the curriculum, and meeting needs for faculty time and resources. Adequate assessment of competency in SBP and PBLI and its impact on outcomes are a challenge.^12,25 We have traditionally used the 360-degree assessments and formative assessment of performance in the QIP for the same. The 360-degree assessments provide a forum to receive comprehensive feedback from a variety of health care professionals, thus resulting in a more complete estimate of fellow performance in relation to the health care system. In addition, the performance of the fellow in the various stages of the QIP provides an opportunity for evaluation of the leadership and collaborative and change management skills of the fellow. Finally, the OSCE is an appealing assessment tool for QI because it tests situational awareness and problem solving and protects against subjective bias inherent in end-of-rotation evaluations. Although we have not conducted longitudinal assessments of learner success in enhancing systems of care, 3 of 6 fellows who graduated from the preventive medicine fellowship in the last 5 years use QI as a major focus of their career in health services administration and/or research.

We hope that the model of experiential QI training we describe will be useful for educators in other graduate medical education programs who are working to create curricula to meet the SBP and PBLI requirements. However, one must be careful in extrapolating our results to other institutions. The model we have proposed is a dedicated 3-week curriculum with an in-depth learning of the key principles of SBP and PBLI through the framework of a QI project. This may not be feasible in all specialties and programs. Many programs may find it more feasible to have a more longitudinal approach to conducting a QI project over a period of 4 to 6 months. For example, in our institution, several programs have their residents do group QI projects during their second or third years, with dedicated blocked time for group meetings and problem solving. Some of them have their chief residents lead the projects. Finally, programs with a large number of residents might face additional challenges in implementing curricula similar to what we describe because of the logistics issues and faculty resources necessary to mentor larger group of learners. In programs with more than 10 learners in each year, program directors may consider multiple projects, done by groups of learners, to achieve adequate experiential learning by each individual learner.

Conclusion

By increasing the awareness of the complex and interrelated structure of health care delivery systems and by training learners in designing better or improved processes, the ACGME competencies of SBP and PBLI can assist in preparing trainees for future careers in QI and management. An experiential curriculum during which learners work together to implement a QIP of community or institutional significance, in the setting of a curriculum that provides just-in-time learning, is an effective and efficient method to teach QI in graduate medical education programs. Further research is required to assess the sustained impact of such a curriculum on learner knowledge, skills, and attitudes in SBP and PBLI.