Abstract
Background
The American Board of Internal Medicine approved the use of Practice Improvement Modules (PIMs) to help training programs teach and assess practice-based learning and improvement (PBLI) and systems-based practice (SBP).
Methods
We surveyed individuals who ordered a PIM in a residency or fellowship training program between June 2006 and August 2009. The 43 programs that volunteered to participate completed a 30-minute anonymous online survey.
Results
Program directors or associate program directors led the PIM process in 30 programs (70%). Trainees' degrees of involvement in PIMs were highly variable between programs, and several respondents felt that trainees were either not sufficiently engaged or not engaged with enough consistency. The most common activity for trainee involvement was data collection through patient surveys or chart review, although only 17 programs (40%) provided protected time for this activity. Few trainees participated in higher level activities such as data analysis or identification for areas of improvement or were given leadership roles; yet most respondents reported that completing the PIM helped trainees learn basic principles of QI and develop competence in PBLI and SBP and that PIM completion improved the program's ability to develop QI initiatives and resulted in program or institutional improvements, including sustainable improvement in patient care. Most respondents reported that the outcome warranted the effort to complete PIMs.
Conclusions
PIMs may be a valuable but underused educational experience for trainees as well as training programs. Focusing on particular themes and facets of PIMs may facilitate implementation.
Editor's note: The online version of this article contains the survey (82.4KB, pdf) used in this study.
What was known
Residents need to become familiar with and be assessed on competency in practice-based learning and improvement (PBLI) and systems-based practice (SBP).
What is new
Internal medicine programs that reported use of Practice Improvement Modules (PIMS) found the modules helped residents learn quality improvement (QI) principles and develop competence in PBLI and SBP. PIM completion frequently resulted in program or institutional improvements, including sustainable improvements in patient care.
Limitations
Nonvalidated survey instrument and the potential for response bias with respondent programs “ahead of the curve” in QI.
Bottom Line
PIMs may be a valuable but currently underused educational experience and approach to QI for trainees and residency programs.
Introduction
Internal medicine training programs are expected to teach residents quality improvement (QI) skills and behaviors and assess residents' on practice-based learning and improvement (PBLI) and systems-based practice (SBP), yet few validated approaches exist to facilitate the development of these competencies. The American Board of Internal Medicine (ABIM) offers Practice Improvement Modules (PIMs) to internal medicine residency and fellowship programs to use as a part of their QI curricula.
While PIMs were originally designed as a tool for self-evaluation of practice performance for use by individual practicing physicians completing Maintenance of Certification (MOC),1,2 they are also used by physicians-in-training to facilitate learning about QI. Trainees collect baseline clinical performance data, survey patients, and examine local practice systems to identify areas of improvement. After choosing an area for improvement, trainees complete a Plan-Do-Study-Act QI cycle3 and report on that improvement. Academic faculty can request MOC credit for participating in a PIM in training.
Prior research on the completion of PIMs in training reveals a positive learning experience for trainees, including demonstrated benefits in areas of teamwork4 and QI awareness, ownership, and skills.4–7 Use of PIMs has furthermore been linked to improvement in clinical outcomes in ambulatory settings4 and was reported to be a catalyst for change and for increased engagement in the QI process.6 However, most of the studies with positive outcomes were carried out in the context of supported research conducted at a single institution. Less is known about how well these tools facilitate residents' educational development and local change without external support. Also, the impact of the PIM on programs and institutions has been largely unexplored. Finally, a deeper understanding of factors that systematically facilitate and/or hinder implementation across a broad range of programs is needed.
We examined these questions from the perspectives of educators who have led and implemented a PIM in training, as there is increasing recognition of the need for physician leadership in quality and patient safety8 and emerging evidence that physician leadership contributes to successful QI project implementation.9
Methods
Individuals who ordered a PIM in a residency or fellowship training program between June 2006 and August 2009 (N = 58) were e-mailed and asked to identify the person who was most responsible for implementing and completing the PIM at their program. Those who volunteered to participate completed a 30-minute anonymous online survey (provided as online supplemental material). The Internet-based survey was developed by ABIM staff and was not piloted; thus, no validity evidence is available. Follow-up e-mails were sent at 1 and 3 months. Participants were notified that data would not be used in accreditation or certification processes or decisions. The study was approved by the Henry Ford Hospital Institutional Review Board and completion of the survey implied informed consent. Respondents received a small financial incentive.
Descriptive statistical analyses were used to analyze quantitative data. Conventional content analysis was used to analyze the open-ended qualitative data. In conventional content analysis, coding categories are derived directly from text data. Researchers then quantify and analyze the presence, meanings, and relationships of codes and concepts, and make inferences about the messages within the texts.10
Results
In May 2011, there were 380 internal medicine training programs and 1637 fellowship programs with a total of 33 297 trainees in the United States.11 Of the 58 programs that had ordered one or more PIMs, 43 responded, yielding a response rate of 74%. table 1 shows the distribution and type of program, including the number and level of trainees. The Diabetes PIM was the module most frequently chosen, used by 14 programs (33%). All PIMs available at the time of the study were chosen at least once (box).
table 1.
Characteristics of Programs Who Completed a PIM in Training June 2006 to August 2009 (N = 43)
box Available PIMs for Training Programs
Asthma
Cancer Screening
Care of Vulnerable Elderly (COVE)
Colonoscopy
Communication
- ○ Primary Care
- ○ Subspecialists
• Diabetes
• Hepatitis C
• HIV
• Hypertension
• Osteoporosis
• Preventive Cardiology
• Clinical Supervision (Faculty Only)
PIM Implementation
PIMs were used primarily longitudinally over the course of a year (N = 30, 71%). Seven programs (17%) used PIMs as part of a block rotation. Most trainees were not given protected time to complete the activities required for the PIM (26 of 43; 60%).
Educators reported that a majority of programs (37; 86%) used trainees to collect baseline and postintervention clinical data via chart audits. Patient surveys were completed by patients in the waiting room (N = 13; 30%) or during the clinic checkout process (N = 10; 23%). Both trainees and clinic staff were described as helpful in the distribution and collection of the surveys. When asked to recall how they collected information to complete the systems portion of the PIM, educators described a number of processes including interviews and/or discussions with staff, personal observation, and document reviews. This section was challenging for many educators, and several could not recall this portion of the PIM (N = 7; 16%). Reasons given for choosing measures for improvement included a suboptimal baseline rate in a particular aspect of care and/or an area targeted for improvement (N = 22; 51%), trainee interest (N = 9; 21%), or ease or feasibility of implementation (N = 7; 16%).
Impact of the PIM
A majority of respondents (N = 31, 72%) felt that the overall experience and impact of the PIM in training was positive, that the outcome of the PIM was worth the effort (N = 36; 84%), and that the PIM was easy to use (N = 31; 71%).
Impact on Trainees
PIMs reportedly facilitated trainees' learning of basic principles of QI (N = 36; 88%) and development of competence in PBLI (N = 34; 83%) and prepared trainees for MOC (N = 30, 73%). In addition, programs reported that the PIM process benefitted trainees in a variety of ways (table 2). A number of respondents felt that the tool was too limited in its breadth and depth and that residents may not have been interested or invested enough to develop sustainable skills in QI.
table 2.
Examples of Educators' Perspectives on the Impact of the PIM on Trainees
Impact on Programs and/or Institutions
More than half of the respondents stated completion of a PIM resulted in changes or enhancements in a process or system in their program (N = 25; 61%) and/or institution (N = 26; 63%). The improvements most commonly reported included clinic changes, such as orienting residents to the clinic; increased documentation or screening procedures; and increased communication (table 3). Four respondents (9%) reported that some of these improvements might have been isolated or not sustainable, particularly due to graduation of residents or the structure of the PIM itself, which was frequently reported as too isolated and not part of a larger program or institutional context (N = 7, 16%). Notably, a majority of respondents reported that the PIM resulted in sustainable improvements in patient care outcomes or delivery of patient care (N = 30; 73%).
table 3.
Changes or Enhancements to Programs and Institutions after Completing a PIM in Training
Facilitators for Implementing PIMs
We surveyed respondents on the facilitators for implementation of PIMs. Findings suggest that no single factor can be considered responsible for the successful implementation of a PIM (table 4). Several themes emerged that may influence the ease and impact of implementing a PIM in training, which are discussed below.
table 4.
Reported Critical Factors for the Successful Implementation of a PIM in Training
Engaged Leadership
table 5 describes the leaders of the PIM process. Consistent with prior studies, educators highlighted leadership as an essential component of implementation (N = 16; 37%) and articulated a broad range of leadership roles. Five respondents (12%) reported they played a central role in PIM implementation, facilitating all aspects of the module. Another 5 described a minimal role, indicating that they, for example, played “no role other than support” or “encouraging it [the PIM] but not being directly involved.” The majority (N = 33; 77%) reported they led specific administrative duties (eg, ordering and funding the PIM, assigning faculty, making sure trainees completed the chart audits). These respondents also reported delegating and assigning tasks to trainees and/or dedicated faculty leaders (N = 18; 42%).
table 5.
Leaders of the PIM Process in Training
Support and Buy-In
A common theme in facilitating PIM implementation was trainee-level and program-level support and buy-in. One educator summarized: “It is a good tool and requires significant leadership, but also institutional buy-in and resources (personnel and financial), otherwise it is person-specific and set up for failure.”
Understanding of roles and expectations emerged as a subtheme within support and buy-in. Trainees' responsibilities were described within the context of basic QI activities, including enrolling patients and collecting data via medical record review and patient surveys. Only a few educators described trainee involvement within the context of complex aspects of the PIM process, including data analysis, QI plan development and implementation, and data recollection after intervention (N = 8; 19%). Several educators (N = 6; 14%) lamented they did not engage trainees enough, and some felt programs should identify opportunities to involve larger numbers of trainees in the design and implementation of the QI intervention.
Faculty Involvement and Professional Development
Other than program directors, clinical faculty involvement in the PIM process was reported to be limited. Most educators discussed the faculty role as one aspect of the PIM (N = 28, 65%), for example, “being kept notified of the results of the PIM,” “supervising trainees,” or “attending meetings.” Nine programs (21%) reported that faculty had “no role.” Many educators stated they would encourage additional faculty participation if their program completed another PIM (N = 13, 30%). More than half (57%) of educators believed that faculty earning credit for MOC was a valuable incentive for faculty buy-in, but it is unclear from our analysis whether faculty participation in PIMs in the responding programs resulted in earned MOC credits.
Teaching QI Principles
The majority (N = 32, 78%) of QI interventions were developed as part of a stand-alone project (designed specifically for the PIM), while a small number (N = 5, 12%) of interventions dovetailed with ongoing QI projects.
More than half (N = 24; 59%) of programs used regular meetings, such as a clinical conference, to discuss the PIM and ongoing QI intervention. One fourth (N = 11; 26%) used a formal QI curriculum to teach trainees and/or faculty about the principles of QI prior to or during the implementation of the PIM. Others reported that the structure of the PIM served as their curriculum (N = 6, 14%).
At the program level, many educators desired more hands-on mentorship in developing a basic QI curriculum (N = 11; 26%) and more assistance in designing a curriculum that could be implemented within the time constraints of a residency program.
Discussion
Many of our findings corroborate earlier results showing the facilitators and benefits of the use of PIMs in teaching settings4–7 and affirming that many educators in leadership roles find the PIM to be a feasible, useful tool to teach and assess the PBLI and SBP competencies. This work also extends findings from earlier studies12 showing that use of PIMs led to a positive programmatic or institutional change in a diverse group of internal medicine training programs in the absence of external research support or funding.
Despite these benefits, our results raise several concerns. First, PIMs are designed to guide an individual physician through a critical review and redesign of a patient care delivery system. When used in training, this process is adapted to allow a group to complete a similar task, yet the structure of a PIM used in this setting may not allow all participants to be actively involved in the QI process. This may explain the lower number and limited involvement of faculty and trainees in this study. Roles, levels of involvement, and expectations for those participating in a PIM in training should be clearly defined.
From a trainee perspective, this lack of adequate involvement in “top-down” QI processes may lead to passive involvement in QI projects,13 potentially reducing impact on learning. Engaging trainees in every aspect of the PIM, rather than relegating their role to data collection or data entry, could translate to trainees developing their own improvement plans and lead to greater understanding of QI in practice. Our findings also suggest that trainees would benefit from receiving program and institutional support for curricular and improvement activities and protected time for completing PIM-related activities.
PIMs can provide valuable work-based experience addressing multiple general competencies (SBP, PBLI, and patient care). However, they cannot serve as a stand-alone QI curriculum. The role a PIM plays in a larger competency-based curriculum, therefore, should also be clearly defined to maximize the learning potential of the exercise. Because a PIM is a complex intervention,17 opportunities exist to embed it as part of a curriculum on complex adaptive systems and how best to use multiple sources of performance data to stimulate change in the training environment. Future work should explore the specific mechanisms about how the PIM process and data may lead to change in these environments. Identifying key mechanisms must be interpreted in the contexts where the PIM is implemented, anticipating that certain contexts may be more supportive than others.
Our study has several limitations. We report on the perspectives of educators who have led and implemented a PIM in training, and we acknowledge that these perspectives may not fully capture those of others, including trainees or faculty involved in the PIM process. Furthermore, as our survey lacks validity evidence, questions may have been interpreted differently by responders. While our response rate was adequate, it is also important to recognize that we focused on a small number of training programs that completed a PIM and that of these programs, not all responded to our survey. Future work around barriers to PIM completion and nuanced differences between small and large programs is warranted. The programs that responded may not be representative of all residency and fellowship programs in the United States. We furthermore recognize that a period of time elapsed between the survey and completion of the PIM, potentially limiting respondents' ability to recall details effectively or accurately.
Conclusions
QI is a complex “team sport,” and trainees need effective role modeling and physician “champions” to implement and sustain successful QI interventions. Future studies addressing PIMs or other QI interventions should focus on the faculty role and consider faculty development to motivate and train faculty in the essentials of QI. A team approach, as well as initial and continued faculty engagement in the QI process, may contribute to success.
These concerns aside, this naturalistic descriptive study of PIMs in multiple programs provides additional evidence for their effectiveness in clinical settings. Results are particularly salient for smaller training programs that may not have sufficient local resources and need to use PIMs or other tools to facilitate learning in PBLI and SBP. We hope that educators outside of internal medicine may be able to apply our findings to their own training programs, but future work will address this more systematically.
Footnotes
Elizabeth Bernabeo, MPH, is Research Associate at the American Board of Internal Medicine and a PhD candidate at Bryn Mawr College Graduate School of Social Work and Social Research; Sarah Hood, MS, is Director of Academic Affairs, American Board of Internal Medicine; William Iobst, MD, is Vice President of Academic Affairs, American Board of Internal Medicine; Eric Holmboe, MD, is Senior Vice President and Chief Medical Officer, American Board of Internal Medicine; and Kelly Caverzagie, MD, is Academic Hospitalist and Associate Vice Chair for Quality and Physician Competence, Division of General Internal Medicine, Department of Internal Medicine, University of Nebraska Medical Center.
Funding: This study was funded by the American Board of Internal Medicine Foundation.
The authors thank all the educators who responded to the survey for providing such valuable information. We also extend many thanks to Siddharta Reddy, MPH, for thoughtful review of the manuscript. Kelly J. Caverzagie, MD, is an Associate in Clinical Performance Assessment at the Department of PIM Development, American Board of Internal Medicine.
References
- 1.Duffy FD, Lynn LA, Didura H, Hess B, Caverzagie K, Grosso L, et al. Self-assessment of practice performance: development of the ABIM Practice Improvement Module (PIM) J Contin Educ Health Prof. 2008;28(1):38–46. doi: 10.1002/chp.154. [DOI] [PubMed] [Google Scholar]
- 2.Holmboe E, Lynn L, Duffy D. Improving the quality of care via maintenance of certification and the Web: an early status report. Perspect Biol Med. 2008;51(1):71–83. doi: 10.1353/pbm.2008.0002. [DOI] [PubMed] [Google Scholar]
- 3.Nelson EC, Batalden PB, Godfrey MM. Quality by Design: A Clinical Microsystems Approach. San Francisco, CA: Jossey-bass; 2007. [Google Scholar]
- 4.Oyler J, Vinci L, Arora V, Johnson J. Teaching internal medicine residents quality improvement techniques using the ABIM's practice improvement modules. J Gen Intern Med. 2008;23(7):927–930. doi: 10.1007/s11606-008-0549-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Oyler J, Vinci L, Johnson JK, Arora VM. Teaching internal medicine residents to sustain their improvement through the quality assessment and improvement curriculum. J Gen Intern Med. 2011;26(2):221–225. doi: 10.1007/s11606-010-1547-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Bernabeo E, Conforti L, Holmboe E. The impact of a preventive cardiology quality improvement intervention on residents and clinics: a qualitative exploration. Am J Med Qual. 2009;24:99–107. doi: 10.1177/1062860608330826. [DOI] [PubMed] [Google Scholar]
- 7.Vinci LM, Oyler J, Johnson JK, Arora VM. Effect of quality improvement curriculum on resident knowledge and skills in improvement. Qual Saf Health Care. 2010;19:351–354. doi: 10.1136/qshc.2009.033829. [DOI] [PubMed] [Google Scholar]
- 8.Taitz JM, Lee TH, Sequist TD. A framework for engaging physicians in quality and safety. BMJ Qual Saf. 2012;21(9):722–728. doi: 10.1136/bmjqs-2011-000167. [DOI] [PubMed] [Google Scholar]
- 9.Rask KJ, Gitomer RS, Spell NO, Culler SD, Blake SC, Kohler SS, et al. A two-pronged quality improvement training program for leaders and frontline staff. Jt Comm J Qual Patient Saf. 2011;37(4):147–153. doi: 10.1016/s1553-7250(11)37018-3. [DOI] [PubMed] [Google Scholar]
- 10.Hsieh H, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15:1277–1288. doi: 10.1177/1049732305276687. [DOI] [PubMed] [Google Scholar]
- 11.Accreditation Council for Graduate Medical Education. Number of Accredited Programs for the current academic year (2011–2012) http://www.acgme.org/adspublic. Accessed December 11, 2012. [Google Scholar]
- 12.Mladenovic J, Shea J, Duffy D, Lynn L, Holmboe E, Lipner R. Variation in internal medicine residency clinic practices: assessing practice environments and quality of care. J Gen Intern Med. 2008;23:914–920. doi: 10.1007/s11606-008-0511-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Philibert I. August 2008. Accreditation Council for Graduate Medical Education and Institute for Healthcare Improvement 90-day project: Involving residents in quality improvement: contrasting “top-down” and “bottom-up” approaches. http://www.residentcouncil.net/wp-content/uploads/downloads/2011/06/2008-Involving-Residents-Philibert.pdf. Accessed December 28, 2012. [Google Scholar]
- 14.Holmboe ES, Bradley ES, Mattera JA, Roumanis SA, Radford MJ, Krumholz HM. Characteristics of physician leaders working to improve the quality of care in acute myocardial infarction: a qualitative study. Jt Comm J Qual Saf. 2003;29:289–296. doi: 10.1016/s1549-3741(03)29033-x. [DOI] [PubMed] [Google Scholar]
- 15.Bradley EH, Holmboe ES, Mattera JA, Roumanis SA, Radford MA, Krumholz HM. A qualitative study of increasing Beta-blocker use after myocardial infarction: why do some hospitals succeed. JAMA. 2001;285:2604–2611. doi: 10.1001/jama.285.20.2604. [DOI] [PubMed] [Google Scholar]
- 16.Caverzagie K, Bernabeo E, Reddy S, Holmboe E. The role of physician engagement on the impact of the hospital-based practice improvement module. J Hosp Med. 2009;4:466–470. doi: 10.1002/jhm.495. [DOI] [PubMed] [Google Scholar]
- 17.Pawson R, Tilley N. Realist Evaluation. London: Sage; 1997. [Google Scholar]