Abstract
Introduction
Quality improvement (QI) training during residency may not be adequately preparing physicians for achieving Accreditation Council for Graduate Medical Education goals and the Institute for Healthcare Improvement (IHI) Triple Aim. The purpose of this evaluation was to identify residents’ perceptions and impact of their QI curriculum.
Methods
We conducted a mixed-methods evaluation of an active-learning QI curriculum for internal medicine residents at one academic medical center. Data from 2017 to 2018 included a focus group, pre-post survey, project data, and curricular materials. Results were categorized using Kirkpatrick’s model of evaluation.
Results
All second-year internal medicine residents completed the curriculum (N = 14). Residents were satisfied with the structure and perceived accomplishment with the curriculum, however were dissatisfied by the impact of inconsistent attendance due to clinical conflicts. Their confidence in QI increased; however, they reported difficulty retaining knowledge and skills. Survey scores related to usefulness and anticipated application of QI were unchanged from baseline.
Conclusions
This applied QI curriculum appeared to improve short-term learning. However, the curriculum did not promote long-term understanding of QI. Finding ways to promote skills and retention beyond the curriculum requires further study.
Keywords: Curriculum, Education, Medical, Graduate/standards, Internship and Residency, Quality Improvement, Quality of Health Care
Introduction
The Institute for Healthcare Improvement (IHI) Triple Aim includes improving patient experience (quality and satisfaction), improving population health, and reducing healthcare costs [1]. Patient experience incorporates the six domains of healthcare quality outlined in the groundbreaking report, Crossing the Quality Chasm, and includes care that is safe, effective, patient-centered, timely, efficient, and equitable [2]. The Accreditation Council for Graduate Medical Education (ACGME) identifies multiple quality improvement (QI) competencies for internal medicine resident physicians [3]; however, the ideal way to achieve these competencies and prepare future physicians for achieving the IHI Triple Aim is still evolving [4].
Several systematic reviews have summarized the current state of QI curricula [4–6]. Most evaluations of QI curricula involve pre-post surveys, the Quality Improvement Knowledge Assessment Tool [7], chart audits, or other observations and feedback. There is a paucity of published literature using qualitative approaches to evaluating QI curricula in residency programs, with only one recently published focus group evaluation [8]. Qualitative approaches may complement other evaluation strategies for improving QI curricula.
The purpose of this evaluation was to understand the impact of a resident QI curriculum within the context of Kirkpatrick’s model of evaluation [9], defined as reaction, learning, behavior, and results. This evaluation model was selected to allow for structured organization of our qualitative data and to identify areas for improvement. A secondary goal was to identify residents’ perceptions of future revisions to the curriculum to include co-learning with faculty physicians, based on recently published literature [10].
Methods
The curriculum was based on the ACGME Internal Medicine Milestones [3] and the Model for Improvement promoted by the IHI, as this model is nationally recognized and accepted [11]. The active-learning curriculum incorporated recommendations from the literature [4], including a longitudinal and applied experience, an emphasis on systems-thinking, selection of projects meaningful to residents, incorporating faculty development, and engagement with a scholarly outcome (i.e., poster). Residents were expected to identify an opportunity for a QI project that was within their control, implement changes and collect data, analyze their results, and prepare a poster following the SQUIRE guidelines [12] that was presented at our local health network Quality Forum.
The curriculum was delivered in ten 45-min sessions across a 10-month period, with 4–5 weeks in between sessions (Table 1). Each session had a curriculum guide with instructions, pertinent definitions, and a timeline. There were no lectures or didactic modules. The residents were divided into two small groups with faculty mentors and used QI tools, including affinity diagrams, process maps, SMART goals, 5 Why analyses, and Plan-Do-Study-Act (PDSA) cycles. Each session ended with action steps, including any assignments to be completed before the next session and a communication plan for residents that missed the session.
Table 1.
Curriculum overview
| Session | Topic |
|---|---|
| 1 | Introduction to quality in healthcare and the Institute for Healthcare Improvement (IHI) Model for Improvement: What does “quality” healthcare look like to you? |
| 2 | Create an affinity diagram to select a focus area: What are we trying to accomplish? |
| 3 | Develop a process map: What are the steps in this process? |
| 4 |
Introduction to SMART (Specific, Measurable, Achievable, Relevant and Time-bound) Goals • Plan to collect baseline data: How do we know we have a problem? |
| 5 |
Reflect on baseline data and conduct a “Five Whys” Analysis to select changes and establish measures • What change can we make that will result in an improvement? How will we know a change is an improvement? • Finalize SMART Goal • Plan for PDSA Cycle 1† |
| 6 |
Test changes • PDSA Cycle 1: Study, Act • Plan for PDSA Cycle 2 |
| 7 |
Test changes • PDSA Cycle 2: Study, Act • Plan for PDSA Cycle 3 |
| 8 | Finish PDSA Cycle 3 and start the poster to be presented at local Quality Forum |
| 9 | Finish the poster and discuss logistics for presenting at local Quality Forum |
| 10 | Wrap-up discussion |
This table reflects the revised curriculum based on the evaluation. Session 1 was added to connect the curriculum to the larger context of quality in healthcare. This required removing a session dedicated to discussing overall project progress. Additionally, the time between sessions was shortened to every 2–3 weeks to sustain project momentum
†The “Do” step in the Plan, Do, Study, Act (PDSA) cycle is completed by the residents between sessions
Formative evaluation occurred throughout the curriculum, primarily in the form of pre- and post-session mentor debriefs. The goals of these iterative evaluations were to make the content easy to understand, decide on revisions to the curriculum guides, and to consider opportunities for general curriculum improvement.
Summative evaluation included analyzing data from the focus group, pre-post survey, QI project data and outcomes, and the curriculum materials. The evaluation was reviewed by the University of Vermont Committees on Human Research under a determination of not research (e.g., quality assurance, educational evaluation). Residents were informed that their participation in the focus group and surveys was voluntary and that their responses may be used in a manuscript.
The focus group of the residents was conducted in May 2018 using a standard approach [13]. The focus group was facilitated by an independent faculty member with extensive experience in educational assessment, using a semi-structured interview guide. The interview questions focused on the evaluation of the completed curriculum. All second-year medicine residents were invited to participate in the focus group. There were no exclusions. The focus group occurred in the classroom at the University of Vermont Medical Center where the residents routinely attended educational sessions. Faculty from the curriculum were not present during the evaluation, and residents were encouraged to be honest with their opinions. The focus group was audio recorded and transcribed without identifiers. Data were maintained in a secure folder at the University of Vermont Medical Center.
We conducted an iterative, qualitative content analysis of the focus group’s transcript, pulling out the most informative and richest text to synthesize and make sense of emerging themes [14]. Consensus among the co-investigators was used to resolve discrepancies. The qualitative analysis was conducted in Word 2016 (Microsoft Corporation, Redmond, WA), as this software was widely accessible by all coinvestigators.
The pre-post surveys occurred approximately 10 months apart and focused on confidence with QI skills. Questions centered on confidence in setting aims for a QI project, establishing measures, testing changes, evaluating the impact of a QI project, and presenting the results. The survey also asked whether participating in a QI project is useful and if residents anticipate applying QI techniques after completing residency training. Survey responses were on a 1–5 scale, ranging from strongly disagree to strongly agree. Data were summarized descriptively in Excel 2016 (Microsoft Corporation, Redmond, WA), as sample sizes were small.
Results, including focus group data, pre-post surveys, resident QI project data, outcomes, and curriculum materials were placed into context using Kirkpatrick’s model of evaluation [9]. In this curriculum, we considered level 3 (behavior) and level 4 (outcomes) to include reaching the ACGME aspirational milestones (PBL12). We did not have evidence of level 3 or level 4 data as part of this evaluation.
Results
Fourteen second-year internal medicine residents completed the curriculum, representing 100% of available second-year residents. Level 1 and level 2 results are presented in Table 2. Nine residents (64%) participated in the focus group. The lack of participation by five residents was felt to represent conflicts with clinical schedules (e.g., night coverage duties) rather than discomfort with providing opinions, although we did not formally assess this. The transcript revealed good participation by all residents, with no single dominant voice and a diversity of voices in the audio recording. Ten residents (71%) completed pre-post surveys (Table 3).
Table 2.
Results organized by Kirkpatrick’s model of evaluation [9]
| Data source | Level 1: Reaction (e.g., learner satisfaction or dissatisfaction) | Level 2: Learning (e.g., learner knowledge, skills, attitude, confidence) |
|---|---|---|
| Focus group |
(+) Sense of accomplishment in completing the QI curriculum (+) QI Curriculum structure (direction, organization) (−) Impact of inconsistent attendance by residents (−) Completing priorities (−) Time between sessions too long to sustain momentum (−) Technology barriers (−) Physical work environment (−) Insufficient mentoring early in the curriculum |
(−) Lack of skills (−) Lack of long-term understanding of QI |
| Pre-post survey | (N) QI is useful |
(+) Confidence with individual QI skills (N) Using QI after completing residency training |
| QI projects | (+) Posters completed and presented locally | |
| Curriculum Materials | (+) Ability to complete specific QI tasks |
There were no results supporting level 3 or level 4 evaluation
+ positive aspect of the curriculum, −negative aspect of the curriculum, N neutral aspect of the curriculum
Table 3.
Survey results (N = 10)
| Kirkpatrick level [9] | Survey question | Pre (mean) | Post (mean) | Change |
|---|---|---|---|---|
| 1 | Participating in a quality improvement project is useful to me | 3.9 | 4.0 | 0.1 |
| I feel confident: | ||||
| 2 | Setting aims for a quality improvement project | 3.4 | 3.7 | 0.3 |
| 2 | Establishing measures for a quality improvement project | 3.1 | 3.5 | 0.4 |
| 2 | Testing changes for a quality improvement project | 2.9 | 3.6 | 0.7 |
| 2 | Evaluating the impact of a quality improvement project | 3.2 | 3.5 | 0.3 |
| 2 | Presenting the results of a quality improvement project | 3.3 | 3.6 | 0.3 |
| 2 | I will use QI techniques after completing my residency training | 3.8 | 3.8 | 0 |
Scores ranged from 1 = strongly disagree to 5 = strongly agree
Level 1: Reaction (e.g., Learner Satisfaction or Dissatisfaction)
The residents’ perceived the benefits of the QI curriculum to be the curriculum structure and the sense of accomplishment in completing the curriculum. With respect to curriculum structure, one resident noted, “I think without direction most of us never would have completed a quality improvement project during residency.” Another commented, “These are the clear tasks and everybody has a specific role. It made it very easy.” A sense of accomplishment was characterized by several residents. “It was sort of amazing that we went from nothing and in just a few sessions to actually having a poster presented.”
However there were several themes related to dissatisfaction with the curriculum. One noted challenge of the curriculum was the impact of inconsistent attendance by residents. “[You] would come to some sessions and then you would be on nights. .. and so sometimes things got a little fuzzy in those situations but that is just the inherent nature of our schedules.” Another resident noted, “It seemed like half the time I felt like either I missed a session because of service or it felt so long that I didn’t even remember what we were doing. I felt like it was a lot of back-tracking to get back on task.”
Other noted challenges specific to the curriculum included between-session intervals too long to sustain project momentum, difficulty in accessing data through the electronic medical record, lack of sufficient direction from mentors early in the project, and challenges of the physical environment, specifically not having a dedicated office or place to work. “I think if we were people who had office jobs here at the hospital who had an office or a desk that we sat at all of the time. .. it would be pretty easy but given that we are just running around all over the place and don’t really have a home base, like I often don’t even sit down at a computer to do non-clinical work until I go home at night.”
Survey scores related to the usefulness of participating in a QI project were essentially unchanged from baseline.
Level 2: Learning (e.g., Learner Knowledge, Skills, Attitude, Confidence)
The survey data suggest improved confidence with various aspects of the scholarly QI process, with changes in pre-post mean scores towards improvement in all confidence questions, in the range of 0.3–0.7 points (see Table 3). Scores related to using QI skills after completing residency training were unchanged from baseline.
The two groups of residents selected QI projects about communication. One group focused on streamlining care team communication about patients who were directly admitted to the internal medicine service in transfer from other hospitals. The second group focused on improving care team communication around patient handoffs. The residents were able to successfully participate in this active learning curriculum using their curriculum materials, as evidenced by completion of specific tasks each session. Both groups completed and presented a poster of their work.
However, residents expressed concerns that they did not retain QI knowledge and skills outside of the individual curricular sessions. One resident noted, “To be honest I think I was losing the forest for the trees. .. I think it was maybe the time in between or just how quickly things were done I don’t think I really truly understood it.” Another commented, “I feel like the tools that they taught in the moment doing it and having the instruction at one session was doable. .. I don’t feel like I actually have retention of the overall skills that I learned. It was transient.”
Perceptions of Co-learning Between Residents and Faculty
As part of the focus group, residents were asked to comment on possible future revision of the curriculum to include co-learning with faculty physicians in the Division of Hospital Medicine. Residents’ perceptions of co-learning centered on the themes of perceiving value in resident-driven project ownership and fear of losing project control. “In theory it sounds great, but I worry that it will just be more of a Hospitalist-driven quality project as opposed to one coming from us. It would be nice to have senior people on the team to drive things forward, but I don’t know if they are going to care as much about handoffs.” Other residents’ commented, “It seems like it would most likely be someone else telling us which project to work on” and “I feel like I would just have a very superficial role.” “I feel it might be harder for us to voice our opinions and say what we think and drive it ourselves if the people who are used to being our superiors at work are in the group. I feel like we would end up deferring to what they want done.”
Discussion
The main finding from our evaluation suggests that we designed a QI curriculum so focused on task-driven, active-learning activities that the larger context of quality in healthcare was lost. Resident physicians described a high degree of engagement and understanding during individual QI training sessions. They also successfully completed focused QI projects and scholarly posters of their work. However, they endorsed low retention of the information and skills beyond the curricular sessions.
We did not find any evidence that our curriculum supported residents in achieving Kirkpatrick level 3 (behavior) or level 4 (outcomes) learning. The challenges we found in applying this evaluation model have recently been described [15]; however, we found the model useful for understanding our current curriculum and for planning future evaluations.
Butler et al. recently identified five major themes related to residents’ perceptions of QI, including challenges understanding the vision of QI, confusion, residents’ perceptions of feeling valued, competing priorities, and situations when QI experiences are positive [8]. The impact of inconsistent attendance to curriculum sessions due to competing priorities with other aspects of residency training was a major source of dissatisfaction to residents in our curriculum. Successfully completing applied QI projects requires an investment of time, including protected time to work as a team and time to collect meaningful data. It is unclear how to best align applied QI projects within the current clinical responsibilities that are required as part of residency training. Unlike Butler et al., we did not find evidence of confusion. Residents commented on the structure of the curriculum as being a helpful component, which possibly minimized the potential for confusion.
Successful QI curricula for residents require prepared faculty mentors and role models. Many clinicians do not have formal training in QI and are inadequately prepared to mentor residents in systematically improving processes and outcomes for their patients [4]. Structured approaches for co-learning between faculty physicians and resident physicians have recently been published [10] and offer a potential method for developing faculty while educating residents. Similar to Butler et al., we learned that perceptions of project ownership and feeling valued as part of the process impact learning. We found residents are hesitant about the idea of partnering with faculty in a co-learning model. Our findings suggest incorporating faculty as co-learners into our QI curriculum would require additional efforts to ensure residents are equal members of the project team and reassurance that residents bring a unique and important value to the QI efforts of the organization. This finding has resulted in our decision not to pursue co-learning with faculty physicians until we understand more about how to partner resident and faculty physicians successfully.
Our study is limited by the small sample size at one institution and the absence of long-term follow-up. Additionally, we did not collect measures of resident behavior change, including self-reported behavior change. However, we believe the insights gained from the study are valuable for developing and improving QI curricula for residents and raise both strengths and weaknesses about active-learning and co-learning models.
Conclusions
Our findings suggest that residents appreciate a QI curriculum that incorporates structured, active-learning activities and opportunities for presenting results. We suggest adding content that connects the immediate learning to the larger context of quality in healthcare, and adding formal mixed methods evaluations to QI curricula. Careful consideration of residents’ perceptions should be explored before incorporating co-learning models with faculty physicians into QI curricula. Finding ways to promote long-term understanding and integration of QI methods into practice requires further study.
Acknowledgments
The authors wish to thank Kathryn N. Huggett, Ph.D., Director of The Teaching Academy at the Larner College of Medicine at the University of Vermont for her assistance in facilitating the focus group session with the residents.
Compliance with Ethical Standards
Conflict of Interest
The authors declare that they have no conflict of interest.
Research Involving Human Participants
The evaluation was reviewed by the University of Vermont Committees on Human Research under a determination of not research (e.g. quality assurance, educational evaluation).
Informed Consent
Physician residents were informed that their participation in the focus group and surveys was voluntary and that their responses may be used in a manuscript. There was no formal informed consent document, as this evaluation was reviewed by the University of Vermont Committees on Human Research under a determination of not research (e.g., quality assurance, educational evaluation).
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Project Hope) 2008;27(3):759–769. doi: 10.1377/hlthaff.27.3.759. [DOI] [PubMed] [Google Scholar]
- 2.Institute of Medicine (U.S.). Committee on Quality of Health Care in America . Crossing the quality chasm : a new health system for the 21st century. Washington, D.C.: National Academy Press; 2001. [Google Scholar]
- 3.ACGME. The Internal Medicine Milestone Project 2015. https://www.acgme.org/Portals/0/PDFs/Milestones/InternalMedicineMilestones.pdf. Accessed 13 Feb 2019.
- 4.Massagli TL, Zumsteg JM, Osorio MB. Quality improvement education in residency training: a review. Am J Phys Med Rehabil. 2018;97(9):673–678. doi: 10.1097/phm.0000000000000947. [DOI] [PubMed] [Google Scholar]
- 5.Patow CA, Karpovich K, Riesenberg LA, Jaeger J, Rosenfeld JC, Wittenbreer M, Padmore JS. Residents’ engagement in quality improvement: a systematic review of the literature. Acad Med. 2009;84(12):1757–1764. doi: 10.1097/ACM.0b013e3181bf53ab. [DOI] [PubMed] [Google Scholar]
- 6.Wong BM, Levinson W, Shojania KG. Quality improvement in medical education: current state and future directions. Med Educ. 2012;46(1):107–119. doi: 10.1111/j.1365-2923.2011.04154.x. [DOI] [PubMed] [Google Scholar]
- 7.Singh MK, Ogrinc G, Cox KR, Dolansky M, Brandt J, Morrison LJ, et al. The Quality Improvement Knowledge Application Tool Revised (QIKAT-R) Acad Med. 2014;89(10):1386–1391. doi: 10.1097/acm.0000000000000456. [DOI] [PubMed] [Google Scholar]
- 8.Butler JM, Anderson KA, Supiano MA, Weir CR. “It feels like a lot of extra work”: resident attitudes about quality improvement and implications for an effective learning health care system. Acad Med. 2017;92(7):984–990. doi: 10.1097/acm.0000000000001474. [DOI] [PubMed] [Google Scholar]
- 9.Kirkpatrick JD, Kirkpatrick WK. Kirkpatrick’s four levels of training evaluation. Alexandria: ATD Press; 2016. [Google Scholar]
- 10.Wong BM, Goldman J, Goguen JM, Base C, Rotteau L, Van Melle E, et al. Faculty-resident “co-learning”: a longitudinal exploration of an innovative model for faculty development in quality improvement. Acad Med. 2017;92(8):1151–1159. doi: 10.1097/acm.0000000000001505. [DOI] [PubMed] [Google Scholar]
- 11.Langley GJ. The improvement guide: a practical approach to enhancing organizational performance. 2. San Francisco: Jossey-Bass; 2009. [Google Scholar]
- 12.Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf. 2016;25(12):986–992. doi: 10.1136/bmjqs-2015-004411. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Krueger RA, Casey MA. Focus groups : a practical guide for applied research. 5th edition. Ed. SAGE: Thousand Oaks, California; 2015. [Google Scholar]
- 14.Crabtree BF, Miller WL. Doing qualitative research. 2. Thousand Oaks: Sage Publications; 1999. [Google Scholar]
- 15.Moreau KA. Has the new Kirkpatrick generation built a better hammer for our evaluation toolbox? Med Teach. 2017;39(9):999–1001. doi: 10.1080/0142159x.2017.1337874. [DOI] [PubMed] [Google Scholar]