Quality Improvement Projects as Training Tools for Family Medicine Residents and Faculty

Abstract

Quality improvement (QI) is part of the future of medicine. However, QI concepts are often poorly understood by physicians. Although teaching QI is required in resident training, an effective QI curriculum is difficult to design due to competing demands from clinic schedules and required rotations. The objective of this project was to teach family medicine residents the basic concepts of QI and practical implementation skills based on use of a clinic population, electronic medical record (EMR) system, and Plan-Do-Study-Act (PDSA) cycles. To do this, the Family Medicine residents and faculty at the University of Hawai`i participated in a QI curriculum to improve diabetes care from October 2018 to February 2019 with 5 sessions consisting of lectures, videos, discussions about QI data for diabetes patients, and group activities. Residents and faculty used quality measures pulled from the EMR and PDSA cycles to discuss, select, and implement QI projects for diabetes patients. Pre- and post-tests measured participants' baseline and end QI knowledge and skills. All 18 residents and 12 faculty in the program participated in the curriculum. The pre- and post-test comparisons showed significant improvement in knowledge of QI concepts and the comfort level among residents showing a 59% average improvement in knowledge questions and a 57% average improvement in comfort level in implementing a QI project (Table 4). This study shows that a 5-session QI curriculum based on EMR and PDSA cycles successfully increased family medicine residents' and faculty's knowledge of QI concepts and skills.

Introduction

In the current healthcare climate, it is essential for physicians to be proficient in actively and continuously conducting quality improvement (QI) in their patient panel. Along with ensuring high quality patient care, knowledge of QI concepts and skills is necessary for meeting graduate medical education (GME) requirements,¹ certificate maintenance, and licensure. Additionally, Medicare, health plans, and other payers now incorporate quality scores into provider reimbursement and incentive schemes. Despite this, physicians often lack the training to measure quality or implement QI in real-life clinical practices.²

To address this QI knowledge gap, the Accreditation Council for Graduate Medical Education (ACGME) requires the integration of QI into clinical curriculum. In family medicine, residents in training must complete a QI project and “systematically analyze practice using QI methods and implement changes with the goal of practice improvement”.^1,3 Developing such a QI curriculum can be challenging given residents' clinical demands, training at off-site locations, and time constraints. The curriculum must teach not only QI concepts, but also the practical skills to effectively implement QI projects in busy patient settings.

QI curricula often emphasize how to conduct “top-down”, generic, disease-management interventions rather than teach residents the skills to develop targeted QI projects that address their own patient population.^4–6 However, if graduating residents are expected to competently incorporate QI into their practice populations, they must learn to develop achievable QI projects that target specific, small populations with quick turnaround times. Smith et al describe a resident-led hospital QI project in which third-year residents completed limited root cause analyses and proposed interventions to achieve system-wide change in their inpatient population.⁷ Evidence in the literature of similar resident-led QI efforts targeting outpatient community clinic populations is scarce.

In this study, a curriculum to teach QI concepts and skills to family medicine residents and faculty in a busy clinical practice was developed. The two goals were to:

• 1.Teach residents and faculty to use electronic medical records (EMR) as a QI tool for comprehensive, efficient gathering of quality data, ie, to identify patients, abstract clinical data, and track changes.
• 2.Teach residents and faculty to plan and complete QI projects using rapid Plan-Do-Study-Act (PDSA) cycles.⁸

Methods

Setting

The UH Family Medicine Residency Program (UHFMRP) is a community-based primary care practice with approximately 3600 patients served by 18 residents and 12 faculty (both fulland part-time). The residents and faculty operate in a busy, challenging clinic environment. The clinic provides a full range of care to children, adults and geriatric patients including general primary care, obstetrics/prenatal care and office-based procedures.

Curriculum Development and Delivery

Using concepts from the Institute of Healthcare Improvement (IHI) modules, the QI curriculum was developed and delivered by the first and second authors. The biostatistics lecture during the fourth QI session was led by the third and fourth authors. The number of QI sessions were determined by looking at the UHFMRP didactic schedule. The residents only have 1 halfday of didactics per week, and because the didactic schedule had already been finalized prior to the inception of this project, the QI sessions had to be scheduled whenever there was an opening in a half-day didactic. A total of 5 didactic half-days were open to deliver the curriculum. From there, the major objectives were mapped out and distributed across 5 QI sessions. A pre-test, which was adapted from the IHI and aligned to the major objectives, was subsequently developed to assess participants' baseline knowledge and comfort levels, with the intention of administering an identical post-test to assess their knowledge acquisition and impact on comfort levels. Finally, each QI session was developed using a variety of modalities (see below), ensuring complete alignment to the objectives and engagement from participants. The final curriculum was delivered at the Physician Center at Mililani, the UHFMRP headquarters (at the time).

UH Office of Research Compliance deemed this project exempt under the category of Quality Improvement.

Participants

Curriculum participants included all 12 faculty members and 18 residents of the UHFMRP. The faculty included physicians, a PharmD, and a behavioral therapist. All faculty and residents completed the pre-test and post-test. In terms of participation in QI sessions, however, 100% attendance was not achieved for various reasons. Not all faculty were available during didactic sessions due to competing obligations, particularly part-time faculty. Additionally, residents who were on away electives (rotations occurring at a site outside of a sponsoring institution or associated hospital) or on night float, did not attend didactics. Regardless, all faculty and residents were responsible for acquiring the material and assigned tasks from their teams (see QI Projects below).

Curriculum – QI Sessions

The QI curriculum was constructed as 5 sessions, between 1–2 hours each, occurring between October 2018 and February 2019 to teach residents and faculty concepts and skills of QI as it would apply to their clinic population. The 5 sessions focused on basic QI concepts, the PDSA cycle, QI's role in modern healthcare, presentation of extracted clinic data, and choosing a QI project (Table 1). Each session involved videos (acquired from the IHI website or YouTube) and short lectures (adapted from IHI modules) to teach QI concepts, followed by group discussions. Group discussions reinforced the content of the QI session and allowed participants to apply the content to the development of their own QI projects. Specific discussion questions for each QI session are outlined in Table 1. Diabetes was selected as the focus because it is very prevalent in the clinic's patient population, requires integration across healthcare teams, and has several ambulatory quality metrics under payment transformation. A diagram of the integration of curriculum components is displayed in Figure 1.

Table 1.

Summary of Quality Improvement Teaching Sessions

Session	Total Duration	Content	Description
1	1 hour	Pre-Assessment Duration: 20 min	Administered electronic 10-question pre-assessment (see Table 2) to all faculty and residents.
		Lecture: Introduction to Quality Improvement Duration: 15 min	One video and 6 slides on basic QI concepts: 1.10 min. video by Dr. Mike Evans explaining QI and the PDSA cycle10 2.Two slides on the Swiss Cheese Model 3.One slide on 6 dimensions of quality in health care pre IHI 4.One slide on the process/steps of QI 5.One slide on the essential elements of a QI team 6.One slide on root cause analysis
		Activity: Whole Group Brainstorming Duration: 25 min	Whole group discussion around why patients with diabetes in their clinic might have difficulty with compliance, particularly regarding HbA1c measures (see Table 3).
2	1 hour	Lecture: PDSA Cycle and the Clinic Duration: 25 min	Two slides demonstrating the practical application of a PDSA cycle to a patient who has diabetes — taken from IHI education materials.8
		Activity: Clinic Care Team (Resident and Faculty) Brainstorming Duration: 35 min	Clinic care team group discussions around applying the PDSA technique to our own clinic population – focusing on potential opportunities and barriers.
3	3 hours	Lecture: Why Teach Quality Improvement? Duration: 1.5 hours	Guest lecturer (population health practice liaison) from local insurance company given 30 minutes to explain payment transformation and performance measurements. Word cloud exercise to discuss difference of quality of care based on perspective – ie, patients vs providers vs ancillary staff. Nine slides explaining why QI is being taught in the residency program. 1.Two slides on how healthcare is changing and how it is incorporating QI 2.Two slides on forming and leading a QI team 3.One slide on the goal of QI (moving patients to active participants in their care and the IHI triple aim of Population Health, Experience of Care and Per Capita cost 4.Three review slides of basic QI concepts (repeated from previous session) 5.One slide of PDSA cycle example (repeated from previous session)
		Activity: Clinic Care Team (Resident, Faculty and clinic staff) Brainstorming Duration: 1.5 hours	Clinic care team group discussions around potential interventions to address barriers to compli- ance that were discussed in Session one. Discussions during this session involved clinic staff members (registrars and medical assistants).
4	1 hour	Lecture: QI Review and Data Presentation Duration: 45 min	7 slides reviewing QI concepts & basic biostatistics (repeated from previous sessions): 1.Two slides defining QI 2.One slide reviewing the QI team 3.One slide reviewing the QI process 4.One slide reviewing PDSA cycle 5.Two slides on basic biostatistics 15 slides of clinic data analysis
		Activity: Whole Group Reflection Duration: 15 min	Residents and faculty reflected on the data that was presented, discussed as a whole group whether the data made sense, asked any questions about the data, and thought about how this data (or lack thereof) would inform their approach to QI.
5	30 minutes	Lecture: Selecting ABFM (American Board of Family Medicine) Performance Improvement Activity Duration: 15 min	Residents and faculty were shown how to select their required ABFM Performance Improve- ment Activity via a short (15 slides) step-by-step lecture, as residents and faculty followed along with their accounts.
		Activity: Clinic Care Team (Resident & Faculty) Selection of Intervention Duration: 15 min	Clinic care team group discussions about which specific intervention to develop. They also selected 10 patients to which they would apply their interventions.
		Post-Assessment	Administered electronic 10-question post-assessment (see Table 4).

Figure 1.

Relationship and development of the QI sessions, use of EMR data, QI projects, and evaluation

Use of EMR Data and Limited Patient Surveys

Concurrent with the teaching and discussion sessions, the clinic's information technology (IT) team used the EMR to abstract data on clinic patients with diabetes, including demographics (eg, age and sex), care received (eg, visit date) and quality of care measures (eg, HbA1C and blood pressure). The Biostatistics Core team within UH JABSOM's Department of Quantitative Health analyzed the de-identified patient data. Additionally, a second-year resident created and distributed a 4-question survey to 25 patients asking for thoughts and suggestions about delivery of care (ie, what the clinic was doing well and how the clinic could improve). Both the EMR and survey-based data were presented to the residents and faculty during the fourth QI session. The participants discussed the data in the context of the clinic's population with diabetes, which informed the selection of their QI projects.

QI Projects

The UHFMRP clinic had 3 teams – each team was comprised of 5–6 residents, 4 faculty members, 2 medical assistants, and 1 patient service representative. The development of QI projects occurred within this team structure. At each session, the teams worked on developing their unique QI projects. The teams submitted their final QI project proposals for approval by the first and second authors within 1 month of the last session. The residents chose home glucose monitoring, diet, exercise management, and medication compliance and understanding as top priorities based on the clinic data gathered (see Use of EMR data and limited patient surveys).

Evaluation

Teaching outcomes were measured with residents and faculty pre- and post-testing of QI concepts (Table 2). The test questions were adapted from the IHI website.⁸ The post-test was administered 3 weeks after the last QI content lecture. Assessments were on paper, and data were entered manually into an Excel spreadsheet. Pre- and post-test results were summarized and compared using descriptive statistics such as mean, median, standard deviation (SD), and inter quartile range (IQR). Wilcoxon Signed Rank test was used for comparing pre- and post-scores, and R statistical software version 3.5.1 (R Foundation for Statistical Computing, Vienna, Austria) was used for data analysis.

Results

The curriculum was developed in September 2018. The curriculum was then delivered from October 2018 to February 2019 with 5 sessions consisting of lectures, videos, discussions of QI data for diabetes patients, and group activities. Teams developed their QI projects throughout the curriculum, using knowledge acquired from the QI sessions to inform their discussions and plans.

In order to develop the QI projects, the clinic's patient population was analyzed using EMR data. The clinic's patient population is racially and ethnically diverse (23% white, 32% Asian American, 25% Native Hawaiian/Other Pacific Islander). Most clinic patients have Medicaid (51%) or Medicare (13%), with about a third on private or commercial insurance. Seventeen percent of the patients have been diagnosed with diabetes, with 22% of those with poor control as measured by HbA1C levels greater than 9%. EMR data was used to identify 672 (out of a total of 4,037) patients in the clinic with an ICD-10 billing diagnosis of diabetes between July 2016 and June 2018. Data obtained from these records included zip code, insurance payer, age, sex, medications, body mass index (BMI), blood pressure, lowdensity lipoproteins, comorbidities, HbA1c, and visit counts.

Concurrent to the aforementioned EMR analysis, factors identified as contributing to the patients' diabetic compliance was determined during the first QI session team discussion. Such factors included challenging social situations, proximity to fast food, limited access to healthy food, and language barriers. The complete list of factors is presented in Table 3.

Table 3.

Group Discussion Session 1: Population Factors Believed to Contribute to Poor Diabetic Compliance

Demographics	Socioeconomics	Clinical	Other
Age Sex Ethnicity/Race BMI Family history Household size Cultural emphasis on food Language barriers	Zip code Insurance Education level Low wages Proximity to fast food Access to healthy food Living situation	Co-morbidities Number of medications Number of missed visits Number of total visits Aversion to needles/injectables	No time to monitor glucose Denial Behavioral Health Challenging social situations

In the fifth and final QI session, each team applied everything they had learned, discussed, and planned from the QI sessions to inform the development of a 12-month intervention.

Knowledge acquisition and change in comfort level were assessed using pre- and post-testing. Results showed a 59% average improvement in knowledge questions and a 57% average improvement in confidence level in implementing a QI project (Table 4) for residents. While there was a statistically significant improvement in both knowledge acquisition and change in comfort level for the residents, the improvement in faculty scores was not statistically significant.

Table 4.

A Summary of Pre and Post Test Scores

Comfort levels (question 1) a
	Pre-Test		Post-Test		Δ score		P-value
	Median (IQR)	Avg. (SD)	Median (IQR)	Avg. (SD)	Median (IQR)	Avg. (SD)
Combined (n=30)	3.00 (2.00)	2.80 (1.27)	4.00 (1.00)	3.80 (.81)	1.00 (2.00)	1.00 (1.44)	.002
Residents (n=18)	2.00 (2.00)	2.33 (1.19)	4.00 (1.00)	3.67 (.84)	1.00 (2.00)	1.33 (1.44)	.003
Faculty (n=12)	4.00 (1.00)	3.50 (1.09)	4.00 (.50)	4.00 (.74)	.00 (1.00)	.50 (1.38)	.280
Knowledge questions (questions 2–10) b
	Pre-Test		Post-Test		Δ score		P-value
	Median (IQR)	Avg. (SD)	Median (IQR)	Avg. (SD)	Median (IQR)	Avg. (SD)
Combined (n=30)	4.50 (2.00)	4.20 (2.00)	5.50 (3.75)	5.60 (2.06)	1.50 (3.00)	1.40 (2.36)	.005
Residents (n=18)	4.00 (2.00)	3.56 (1.50)	6.00 (2.75)	5.67 (2.00)	2.00 (2.00)	2.11 (2.14)	.003
Faculty (n=12)	5.00 (3.00)	5.17 (2.29)	5.00 (4.25)	5.50 (2.24)	.00 (2.25)	.33 (2.35)	.675

^aQuestion 1 was on a scale of 1–5.

^bQuestions 2-10 were given 1 point for each correct answer.

Discussion

The first goal of this curriculum was to teach residents and faculty to use EMR as a QI tool for comprehensive, efficient gathering of quality data. The initial discussion in session 1 was used to formulate a list of variables to pull from the EMR. The results of this discussion are presented in Table 3. Unfortunately, a lot of these variables were not easily captured in our EMR. This knowledge was used to improve the EMR system by creating new fields where some of this information can be stored, such as household size, living situation, and education level. Going forward, this will improve the EMR's capabilities of designing better informed QI projects.

The second goal of this curriculum was to teach residents and faculty to plan and complete QI projects using rapid PDSA cycles. The residents and faculty have completed the curriculum and successfully identified and planned out their QI projects. However, there are no clinical outcome data to date because the QI projects are still ongoing. Regardless, the residents and faculty understand the necessary steps in a PDSA cycle as evidenced by their knowledge acquisition and comfort levels. While the residents showed a statistically significant improvement in both knowledge acquisition as well as comfort level, the faculty who participated did not reach statistical significance in their improvement. This is likely because of the smaller number of faculty involved. We still believe it was important to have the faculty involved in the learning as they are key players in the training of the residents and also key players in the delivery of care to our patients.

Implementing a QI curriculum in a residency training site is challenging but feasible. The biggest challenge was the paucity of time available outside of resident rotations, which led to difficulties tracking down pre and post tests and team assignments. With increased charting requirements to cover all the quality measures already required by insurance companies and health partners, the residents already have little time to spare. In a given 80-hour work week, residents are in clinic between 1 and 4 half-days a week and, with the time that is not spent on external rotations, they need to work on finishing the charting for those clinic hours. The residents only have a half-day of didactics scheduled per week. QI sessions were scheduled during this time. However, the breadth of topics that must be covered in Family Medicine training made it difficult to carve out time for QI. In total, QI training took 5 and a half hours over 6 months. The time in didactics was used to learn concepts and develop interventions. Based on lessons learned from this project, a future directive would be to dedicate 30 minutes to 1 hour every month for teams to follow up on QI projects.

An ideal QI curriculum would address knowledge of what quality care is, who are the essential elements of an improvement team and how to work with them, how to utilize data (and an EMR to collect it) as well as how to develop an aim, and how to work through a PDSA cycle. While the residents have demonstrated understanding of how to work through a PDSA cycle, there were challenges with some of the other curriculum components.

Understanding what quality care is for patients proved to be a challenge because of residents' proximity to issues of resident wellness and lack of experience focusing on QI in patient care. This was addressed by incorporating a few exercises to get residents and faculty discussing quality care as seen by patients. One technique was to discuss “What is good care from the perspective of a physician?” and “What is good care from the perspective of a patient?” A second technique was to present the patient survey results to the residents and faculty to read and discuss. These 2 strategies proved to be useful, and the subsequent discussions were more patient-centered.

One of the goals of QI, according to IHI, is to move the patient from a passive to an active recipient of care.⁸ A good way to do this is to include them in the early stages of planning a QI project. Having patient representatives involved in the QI curriculum would be1 way to accomplish this. While this project was able to include the ancillary staff and high-level decision makers in the QI teams, it was not able to include patient representatives. This is still a future consideration because the team believes that an important step for any residency program developing a QI curriculum is to help the residents and faculty see that the challenges they personally face in healthcare delivery do not always translate to challenges in patient healthcare. Likewise, it is important to realize that the challenges patients see in receiving healthcare are not always factors that jeopardize the delivery of quality care.

Curriculum success factors included choosing core concepts of QI and repeating them at every session. Multiple studies have shown that spaced repetition improves retention.⁹ Incorporating team-based brainstorming proved an engaging way to reinforce this knowledge. Having the ancillary staff involved in the group discussions proved beneficial. The residents expressed they did not realize the challenges to workflow experienced by the staff or the different perspectives they afforded. Including patient surveys in group discussions was helpful to allow the groups to see the patient point of view. Finally, resident and faculty “buy-in” was accomplished by having them register for an activity required for their boards or recertification, thus saving time in their busy workday. Despite all these challenges, the rewards of creating a QI curriculum for the residents were enough to justify continued work on sustainable implementation to programs, possibly through the use of improved EMR systems and incorporation of online training. A limitation of this study was that it was conducted at only 1 training program and clinic site. While the curriculum can be replicated by other programs, it would likely require modifications tailored to address individual program constraints.

Conclusion

In summary, residents must receive training and experience in QI. Yet too often this training and experience is not robust or effective. The research team developed a way to teach residents QI by having them complete a QI project on clinic patients and involving them in every step, thereby helping them learn the process of QI. This 5-session QI curriculum based on EMR and PDSA cycles successfully increased family medicine residents' knowledge of QI concepts and skills.

Table 2.

Assessment of Knowledge Acquisition and Comfort Levels Before and After Curriculum Implementation

Item No.	Question	Correct Answer
1	How comfortable do you feel in implementing a Quality Improvement project by yourself on a scale of 1-5, with 1 being not comfortable and 5 being very comfortable	1-5
2	What are the six dimensions of quality in healthcare? a)Performance, features, reliability, durability, conformance, and serviceability b)Safety, effectiveness, patient-centeredness, timeliness, efficiency, and equity c)Timeliness, completeness, consistency, conformity, accuracy, and integrity d)Patient-centeredness, timeliness, accuracy, completeness, safety, and integrity	b
3	What is a type of communication tool that “closes the loop” between sender and receiver? a)Advocacy inquiry b)Briefings and debriefings c)Verbal repeat back d) SBAR	c
4	What does data collection look like for quality improvement? a)Gather just enough data to inform improvement and only collect data on 1-2 confounders as needed b)Gather enough data to authoritatively study for effect and control for all known confounders c)Gather just enough data to inform improvement and control for all known confounders d)Gather enough data to authoritatively study for effect and only collect data for 1-2 confounders	a
5	Which of the following does NOT characterize a System Authority? a)Has authority in all areas affected by the change, who can overcome barriers that may arise b)Able to help the team determine what to measure c)Able to allocate time and resources to achieve the team's aim d)Understands the implications of the proposed change for various parts of the system	b
6	What is the scientific method used for action-oriented learning and tests a change in the real work setting? a)Vision-Plan-Execute (VPE) cycle b)Plan-Do-Study-Act (PDSA) cycle c)Problem-Plan-Solution (PPS) cycle d)Aim-Measure-Action (AMA) cycle	b
7	What is a written, measurable, and time-sensitive statement of the expected results of an improvement project? a)A vision b)An aim c)A hypothesis d)A measure	b
8	What are the essential elements of an improvement team? a)Clinical-technical expertise, healthcare providers, and day-to-day leadership b)Residents, faculty, and other healthcare employees c)System authority, clinical-technical expertise, and day-to-day leadership d)Faculty, clinical-technical expertise, and residents	c
9	What are the general steps (in the correct order) of a Quality Improvement project? a)Form a team → Set an aim → establish measures → identify changes → test changes → implement changes b)Set an aim → establish measures → collect data → identify changes → test changes → implement changes c)Form a team → set an aim → establish measures → collect data → identify changes → implement changes d)Set an aim → form a team → establish measures → identify changes → test changes → implement changes	d
10	What is a difference between quality research and quality improvement? a)The purpose of quality research is sustained improvement, whereas the purpose of quality improvement is proof of effectiveness b)The methods behind quality research involve a large test with a fixed hypothesis, whereas the methods behind quality improvement involve rapid sequential tests with a hypothesis that changes as learning takes place c)The data collecting process in quality research entails gathering just enough data to inform improvement, whereas the data collecting process in quality improvement entails gathering enough data to authoritatively study for effect d)Quality research requires no effort in controlling bias, whereas quality improvement requires controlling bias as much as possible	b

Acronyms

ACGME

Accreditation Council for Graduate Medical Education

BMI

Body Mass Index

EMR

Electronic Medical Record

GME

Graduate Medical Education

HbA1c

Hemoglobin A1c

ICD-10

International Classification of Diseases Tenth Revision

IHI

Institute of Healthcare Improvement

IQR

interquartile range

IT

information technology

JABSOM

John A. Burns School of Medicine

PDSA

plan-do-study-act

QI

Quality Improvement

SD

standard deviation

UH

University of Hawai`i

UHFMRP

University of Hawai`i Family Medicine Residency Program

Conflict of Interest

None of the authors identify a conflict of interest.