Teaching Internal Medicine Residents to Sustain Their Improvement Through the Quality Assessment and Improvement Curriculum

ABSTRACT

INTRODUCTION

Although sustainability is a key component in the evaluation of continuous quality improvement (CQI) projects, medicine resident CQI projects are often evaluated by immediate improvements in targeted areas without addressing sustainability.

AIM/SETTING

To assess the sustainability of resident CQI projects in an ambulatory university-based clinic.

PROGRAM DESCRIPTION

During their ambulatory rotation, all second year internal medicine residents use the American Board of Internal Medicine’s Clinical Preventive Services (CPS) Practice Improvement Modules (PIM) to complete chart reviews, patient surveys, and a system survey. The residents then develop a group CQI project and collect early post data. Third year residents return to evaluate their original CQI project during an ambulatory rotation two to six months later and complete four plan-do-study-act (PDSA) cycles on each CQI project.

PROGRAM EVALUATION

From July 2006 to June 2009, 64 (100%) medicine residents completed the CQI curriculum. Residents completed six group projects and examined their success using early (2 to 6 weeks) and late (2 to 6 months) post-intervention data. Three of the projects demonstrated sustainable improvement in the resident continuity clinic.

DISCUSSION

When residents are taught principles of sustainability and spread and asked to complete multiple PDSA cycles, they are able to identify common themes that may contribute to success of QI projects over time.

Electronic supplementary material

The online version of this article (doi:10.1007/s11606-010-1547-y) contains supplementary material, which is available to authorized users.

INTRODUCTION

In 2009, the Accreditation Council for Graduate Medical Education (ACGME) introduced new continuity clinic requirements for all internal medicine residents. These requirements included: 1) performance data review for each resident’s continuity clinic panel, and 2) involvement in a continuous quality improvement (CQI) process¹. Although practice performance evaluation has been a required element of practice-based learning and improvement (PBLI), many residency programs have had to develop unique curricula to fulfill these requirements^2–4. While there are many challenges to teaching CQI^4–12, the American Board of Internal Medicine’s (ABIM) Practice Improvement Modules (PIM) have been used as a tool to teach physicians important CQI principles^6,13–15. The previously published results of the University of Chicago Quality Assessment and Improvement Curriculum (QAIC) showed that residents were able to make significant changes in their continuity clinic using the Clinical Preventive Services (CPS) PIM^14,16.

Sustainability is a key component of the evaluation of CQI projects. However, many CQI projects are evaluated by immediate improvements in targeted areas and do not address long-term effects of improvement efforts. Sustainability is especially relevant in residency training due to the transient nature of resident education. QI projects which are embedded into institutional memory have greater potential for long term sustained improvements after the champions of that project graduate. Sustainability of CQI has been assessed in manufacturing¹⁷ but little is published in the medical literature^18–21. McDonald and colleagues developed the Quality Improvement Proposal Assessment Tool (QIPAT-7) to assess residents’ CQI proposals, including evaluation of sustainability²². However, despite involvement in CQI processes, residents are often not expected to evaluate the sustainability of these processes.

AIM

The aim of this educational innovation is to describe a novel sustainability curriculum for internal medicine residency CQI projects, to assess the sustainability of resident CQI projects in an ambulatory university-based clinic, and to determine qualitative factors that contributed to the sustainability of these projects.

PROGRAM DESCRIPTION

We previously described the first year of QAIC, in which post graduate year 2 (PGY2) residents use the ABIM’s CPS PIM to complete chart reviews, gather patient surveys, and complete a system survey. They use the data gathered by the PIM to reflect on the quality of care they provide in their continuity clinic and develop a plan for improvement. Residents work together in groups of 9-12 during their second PGY2 ambulatory month to develop an aim statement, map the targeted process, interview stakeholders, and perform a small test of change¹⁴.

The second year of the curriculum teaches PGY3 residents about principles of sustainability and spread (Table 1). PGY3 residents are assigned to complete their ambulatory rotation during one month in the summer/fall and one month in the winter/spring. Each month, there are 9-12 residents who had previously worked on a group project during their PGY2 year. For example, in January 2007 10 residents completed the first PDSA cycle of the BMI project, in March 2007 10 residents completed the first PDSA cycle of the Smoking Cessation Project, and in May 2007 10 residents completed the first PDSA cycle of the Medication Refill project. In August 2007, 3-5 residents from each of these projects (BMI, Smoking, and Refills) came together to participate in the Sustainability Block. During the Sustainability Block, the residents work in small groups to complete further PDSA cycles (Fig. 1). This structure enables representatives from each project to be present over the course of three sustainability blocks so that residents can complete PDSA cycles 2 thru 4, while also giving feedback on fellow residents’ QI projects.

Table 1.

Quality Assessment and Improvement Curriculum Outline

Residency year	Ambulatory block summer/Fall (1 month*)	Ambulatory block winter/spring (1 month*)
Post graduate year 2**	Quality assessment block	Quality improvement ( QI) project block
	Complete ABIM PIM	Develop and implement a QI project
Post graduate year 3	Sustainability and spread block	Quality summary block
	Measure QI project sustainability	Focus on pay for performance
	Week 1 : Review QI project results, assess pitfalls, re-work aim	Week 1: history, policy and theory of pay for performance
	Week 2: Sustainability lecture, measure long-term follow-up	Week 2: Pay for performance: reality check
	Week 3: Develop sustainability plan	Week 3: Complete ABIM PIM and present poster
	Week 4: Spread lecture, develop handoff for next group	Week 4: Review essential QI concepts

*Each block consists of four 90-minute sessions, ** Post Graduate year 2 curriculum was described in previously published article¹⁴ (Oyler JL, Vinci L, Arora V, Johnson, J. Teaching Internal Medicine Residents Quality Improvement Techniques Using the ABIM’s Practice Improvement Modules. Journal of General Internal Medicine 2008; 23(7):927-30.)

Figure 1.

Plan do study act cycles for the tobacco cessation project.

During Week 1 of the Sustainability Block, residents use class time to review their previous group project, assess the strengths and weaknesses of that project, and determine how their aim statement needs to be changed. The residents are given a lecture on sustainability during Week 2 and develop a plan to measure long term impact of their project. Faculty use Britain’s National Health Service sustainability model to discuss barriers and facilitators to projects, including project factors, staff factors, and organizational factors²³.

During this session, faculty recorded the successes and weaknesses brainstormed by the residents for each project. After reviewing their long term follow-up data during Week 3, the residents develop a sustainability plan for their project to address the resident turnover in the continuity clinic. Faculty supervise this sustainability plan and residents present the plan to peers to get feedback. During Week 4, the residents receive a lecture on spread of QI projects which includes social networking theory²⁴, diffusion of innovation²⁵, and adopter categorization²⁶. They are then asked to consider spread of their QI project and develop a short one page handoff document for their PGY3 peers from their QI project group to use as they reassess the project again in two months (Appendix 1). After all PGY3 residents have completed the Sustainability and Spread Block, a total of four plan-do-study-act (PDSA) cycles (Appendix 2) have been completed on each QI project (Fig. 1). The PGY3 residents return 6 months later to receive four ninety minute lectures about pay for performance and review essential QI concepts. These lectures are available in the online appendices (Appendix 3). During this time, they finalize poster presentations of their projects and complete the ABIM PIM.

PROGRAM EVALUATION

From July 2006 to June 2009, 64 (100%) internal medicine residents participated in the sustainability portion of QAIC. All residents completed written consent forms and the study was approved by the University of Chicago Institutional Review Board. Residents completed 6 group projects which they evaluated with both early (2 to 6 weeks) and late (2 to 6 months) post-intervention data. As previously described, residents brainstormed qualitative themes that contributed to the sustainability of each project. Sustained projects were identified as projects which were able to make a significant improvement after early post intervention data collection and were able to maintain these improvements during late post intervention follow-up. Stata 10 was used to perform statistical analysis. Two sample tests of proportions were used to assess changes observed in each project from baseline. Statistical significance was defined as p < 0.05. Three of the six groups showed a sustainable improvement in the resident continuity clinic. Each group developed resident and patient education components and targeted specific clinic interventions.

Projects with Sustained Improvements

Three groups were able to show sustained improvement over time (Fig. 2a). The medication bottle project added a reminder to the clinic hotline for patients to bring their medication bottle to clinic each visit and educated resident physicians about Joint Commission requirements for patients to receive an updated medication list at every visit. There was an initial improvement in the measure “doctor giving patient medications lists” from 56% to 76% 2 weeks post (p = 0.043), which was sustained at 74% 9 months post (p = 0.032). The number of patients who brought in their medication bottles initially did not change (17% to 18% 2 weeks post, p = 0.861), but then improved significantly to 40% 9 months post (p = 0.002).

Figure 2.

Results from Internal Medicine Resident Quality Improvement Projects (a) Sustained Improvements and (b) Non-Sustained Improvements. Blue Bar: Pre-Intervention (baseline). Red Bar: Early Post-Intervention (2-6 weeks). Grey Bar: Late Post Intervention (2-6 months).

The smoking cessation project worked on documentation of smoking status and then initiated a referral process to a smoking cessation clinic. This group was able to improve resident documentation of smoking status from 41% to 67% 4 weeks post (p < 0.001) and continued to increase to 82% 6 mosnth post (p < 0.001). The aspirin project added current aspirin guidelines to the residents’ clinic screening form and posted informational flyers for patients about the importance of aspirin use in exam rooms. The project resulted in an initial modest improvement in aspirin documentation for patients with any indication for aspirin use from 55% to 67% 4 weeks post (p = 0.070) that increased to 75% 4 months post (p = 0.002).

Projects with Non-sustained Improvements

Three of the groups were not able to sustain their projects over time (Fig. 2b). The body mass index (BMI) project incorporated height and weight into clinic triage and BMI was calculated and documented by residents using newly posted BMI charts. Although BMI documentation improved initially from 4% to 79% at 2 weeks post (p < 0.001), documentation of BMI had fallen to 41% at 6 months post (but still increased over baseline, p < 0.001). The medication refill center project attempted to improve the accuracy of clinic note medications lists that were used by the clinic refill center to refill medications. Inaccuracies in these medication lists led to residents getting paged by pharmacies, the refill center, and the patients about medication refills. The residents educated peers, updated business cards with the correct clinic refill center hotline, and integrated correct medication dictation instructions into intern clinic orientation. They then assessed the accuracy of dictated medication lists, counting medications without dose, route or directions as inaccurate. While they initially made an improvement in the accuracy of the medication lists from 75% to 91% 4 weeks post (p < 0.001), the accuracy dropped back down to 75% by 4 months post (p = 0.808). Finally, the social service resource project assessed resident confidence in providing social resources in clinic and then added accessible wall folders with updated information on transportation, food, mental health, and homemaker services. Although not significant, residents made an initial modest improvement in residents’ ability to provide social support at least once per clinic (10% pre to 25% post, p = 0.075). However, the group was not able to collect late post data and could not ascertain whether these initial improvements were sustained.

Common Themes for Quality Improvement Projects

The following were the common themes that residents noticed while evaluating their projects: 1) QI initiatives that included only educational interventions tended to wane over time, 2) QI initiatives which were integrated into the clinic system were more sustainable, 3) QI initiatives can integrate quickly into “clinic memory” because of resident turnover, 4) Residents were generally flexible and open to change and 5) QI initiatives which decreased burden in clinic were more likely to be sustained. For instance, developing a referral source for tobacco cessation or having patient bring in medication bottles simplified residents’ clinic visits, while requiring residents to calculate BMI on each patient was seen as an additional burden. Some interventions took a longer time to show significant change. For instance in the medication bottle project, changing the phone message which encouraged patient to bring their medication bottles to clinic did not make a difference until the late post follow-up 6 months later.

DISCUSSION

Sustainability is often overlooked when teaching and evaluating quality improvement projects. QI curricula teach residents about the principles of PDSA cycles, but rarely require residents to complete more than a few PDSA cycles on a given project. By assessing QI projects multiple times over a 2-year period, residents are able to evaluate barriers and facilitators to sustainability and spread.

There are a myriad of challenges to sustainability in QI projects and these challenges provide an opportunity to teach residents that ongoing efforts are needed to hold the gains from their improvement projects. Group projects may have varying levels of involvement between residents. Faculty addressed this issue in our curriculum by having resident groups fill out a PDSA form during each cycle listing each resident’s responsibilities (Appendix 2). By reviewing the PDSA form each week, residents and faculty were able to report on individual contributions to the larger group project. Although attending clinics were in the same physical location and have a common structure, we observed that the residents’ QI project initiatives were often hard to spread to attending clinics and met resistance to changing the “old ways” of doing things. Structural changes to the clinic, while positive for clinic workflow, often had unintended consequences that affected the sustainability of QI projects. For example, adding cubicles to resident workrooms and redecorating patient rooms led to loss of BMI charts, aspirin posters, and social services resource wall files. One resident team had difficulty completing long term follow-up, due to the timing of their project. There was another instance when the supporting guidelines changed in the middle of a project. The aspirin project sought to increase the number of diabetes patients who were prescribed aspirin. However, conflicting evidence about the use of aspirin to prevent heart disease in diabetic patients surfaced during their intervention.

Sustainability is a key learning objective when teaching internal medicine residents about quality improvement. It is important for residency programs to train residents to address the sustainability of a project during the initial design of a project as well as during subsequent PDSA cycles. Faculty teaching CQI should incorporate sustainability concepts early in QI curricula and plan for residents to evaluate sustainability of CQI projects. Lectures and worksheets for the University of Chicago QAIC are available to guide this process (Appendix 1-3)²⁷. When residents are taught principles of sustainability and spread and asked to complete multiple PDSA cycles they can learn to identify common themes that may contribute to success of their QI projects over time.