Abstract
Background
Training in quality improvement (QI) is a standard component of family medicine residency education. Graduating family medicine residents' ability to lead QI initiatives is unknown.
Objective
We assessed the preparedness of graduating family medicine residents to lead QI projects and to identify factors that may increase such readiness.
Methods
Milestone data for all graduating family medicine residents were linked to a practice demographic questionnaire completed by the same residents who registered for the American Board of Family Medicine certification examination between 2014 and 2017. The change in self-assessed QI preparedness over time and its association with faculty-assigned milestone ratings were examined using descriptive and regression analyses.
Results
The questionnaire had a 100% response rate (12 208 responded). Between 2014 and 2017, the percentage of residents who self-reported being “extremely” or “moderately” prepared to lead QI projects increased from 72.7% (2208 of 3038) to 75.8% (2434 of 3210, P = .009). Self-reported QI team leadership was associated with 93% higher odds of feeling extremely prepared compared to moderately prepared (odds ratio 1.93, 95% CI 1.58–2.35). The average midyear faculty-assigned milestone rating for QI among residents who felt “extremely” prepared was 3.28 compared to 3.14 among those who felt “not at all” prepared.
Conclusions
Over the past 4 years, family medicine residents' self-assessed preparedness to lead QI projects has barely increased. There was no correlation between self-assessed preparation and faculty-assigned milestone rating. However, we found a small association between self-reported QI leadership and self-assessed QI preparedness.
What was known and gap
Family medicine residents are required to complete quality improvement (QI) projects and gain QI skills, but it is not known what factors are important for them to be able to lead future QI projects.
What is new
An analysis of milestone data on residents' QI project involvement and a questionnaire that included questions on self-reported preparation for QI.
Limitations
The study focused on family medicine residents, which may not be generalizable to other specialties, and was cross sectional so causal inferences cannot be made based on the results.
Bottom line
No correlation was found between self-reported ability to lead QI and faculty-assigned milestone ratings, but did find that leading a QI project in residency was associated with self-assessed QI leadership preparedness later.
Introduction
Quality improvement (QI) training is an essential component of residency education. Both the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties define being able to implement practice changes as a core competency.1,2 The Affordable Care Act mandated that physicians be trained to participate in “continuous quality improvement projects to improve health outcomes of the populations they serve.”3 Training physicians in these QI techniques is essential to achieving the Triple Aim of health care—improved patient care experience, decreased cost, and better population health.4
Residency programs throughout the United States have responded to this national mandate with a variety of curricula. Previous research has evaluated the effect of these interventions, noting that program structure is variable, consisting of didactics, experiential opportunities, or a combination of both, with divergent results.5,6 In recognition of the important role institutional factors play in residency training, such as faculty QI knowledge and organizational capacity for change,7 the ACGME implemented the Clinical Learning Environment Review (CLER).8 In 2017, after 297 site visits, CLER found that QI training continues to be highly variable across sites.9 A median of 79% of postgraduate year 2 (PGY-2) and higher residents reported participation in a QI project, but many had a limited knowledge of specific concepts, such as plan-do-study-act (PDSA) cycles. An article detailing a radiology residency program's improved QI curriculum highlighted the importance of QI project completion. The new curriculum required residents to complete a QI project, which resulted in an increased self-reported ability to lead QI projects and a higher average systems-based practice milestone score over 3 years.10
The 2016 CLER National Report of Findings only included institutions with 3 or more residency programs; most family medicine residency programs are part of single residency institutions and were not included in the initial surveys.9,11 Since 2012, family medicine residents have been required to complete a QI project as one of their 2 scholarly activities during residency.12 In addition, as part of the initial board certification process, residents must complete a performance improvement (PI) activity.13 Prior research has shown that programs have been able to adjust their curriculum to meet the scholarly activity requirement,14 but the impact of these efforts has not been well-studied. While previous studies have identified conditions crucial to enhancing QI skill acquisition,15 the important factors influencing residents' ability to lead QI initiatives remain uncertain.
The purpose of this study was to investigate self-assessed preparedness to implement QI projects among a national sample of graduating family medicine residents. We also aimed to identify predictive factors for higher perceived QI ability and the relationship between self-assessed and faculty-assigned QI preparedness.
Methods
Study Sample
We used data from the American Board of Family Medicine (ABFM) certification examination practice demographic questionnaire completed by graduating residents from 2014 to 2017.16 Data from the questionnaire for certification and continuing certification candidates provided comparable state-level estimates on adoption of electronic health records17 and future enrollment in fellowships.18 In 2017, the questionnaire was redesigned by the same survey developers to include many of the questions in the ABFM National Family Medicine Graduate Survey, whose content was extensively pretested with validity evidence.19 This questionnaire is a mandatory component of the examination registration process and is completed approximately 3 months before the examination date. The resident questionnaire contains items on level of educational debt, self-reported preparation for QI, and future practice characteristics, among others. We also used faculty-assigned milestone data on each resident from their midyear ratings in PGY-3, which is concurrent with examination registration. Personal characteristics were obtained from ABFM administrative data.
The main outcome for this analysis was derived from the question: “How prepared are you to lead quality improvement projects?” Candidates could respond: not at all prepared, somewhat prepared, moderately prepared, or extremely prepared. We created a dichotomous outcome combining the “moderately prepared” and “extremely prepared” answers compared to the combination of “somewhat prepared” and “not at all prepared” for analysis. Candidates are also asked about their degree of participation in the QI activity completed to meet their ABFM PI activity requirement. All residents are required to complete a PI activity during residency—either an ABFM-produced activity or their own project. The questionnaire asks what tasks the residents completed as part of their PI activity, such as data entry, data abstraction, and QI team leadership. Lastly, we matched resident responses from the certification examination questionnaire to their milestone ratings, specifically practice-based learning and improvement–3 (PBLI-3) milestone: “Improves systems in which the physician provides care.”
This study was approved by the American Academy of Family Physicians Institutional Review Board.
Data Analysis
We used descriptive statistics to analyze the characteristics of residents in each year. We conducted t tests to analyze any differences between continuous variables and chi-square tests to compare categorical variables. We specifically analyzed residents' responses each year to assess for a change in level of self-assessed preparation over the study period. We also examined the association between level of preparation and PI activity participation and its correlation with PBLI-3 milestone ratings. All analyses were completed in SAS 9.4 (SAS Institute Inc, Cary, NC).
Results
The questionnaire had a 100% response rate (12 208 of 12 208) as it was a required component of examination registration. Between 2014 and 2017, there was no significant difference between each class cohort based on age, sex, prevalence of Hispanic or Latino residents, or self-identified race (of note, the category “Other” was added in 2016, which represents a statistically meaningful change, but the distribution was still similar between years). Between 2014 and 2017, there was a significant increase in doctor of osteopathic medicine candidates (17.0% [516 of 3040] to 20.4% [640 of 3210], P = .007), with the most prominent change occurring between 2014 and 2015, while the subsequent years were more similar (Table 1). Figure 1 shows the change in self-reported preparedness to lead QI projects between 2014 and 2017. In 2014, 72.7% (2208 of 3038) of residents responded feeling either “extremely prepared” or “moderately prepared.” In 2017, this increased to 75.8% (2434 of 3210, P = .009).
Table 1.
Personal Characteristics of Graduating US Family Medicine Residents From 2014–2017 (N = 12 208)
| Characteristic | 2014 | 2015 | 2016 | 2017 | P Value | ||||
| Mean age | 3040 | 33.5 | 2806 | 33.4 | 3152 | 33.3 | 3210 | 33.2 | .68 |
| Medical degree | |||||||||
| MD | 2524 | 83.0% | 2233 | 79.6% | 2500 | 79.3% | 2570 | 80.1% | .001 |
| DO | 516 | 17.0% | 573 | 20.4% | 652 | 20.7% | 640 | 19.9% | |
| Sex | |||||||||
| Male | 1403 | 46.2% | 1247 | 44.4% | 1429 | 45.3% | 1473 | 45.9% | .57 |
| Female | 1637 | 53.9% | 1559 | 55.6% | 1723 | 54.7% | 1737 | 54.1% | |
| Race | |||||||||
| American Indian or Alaska Native | 34 | 1.1% | 31 | 1.1% | 35 | 1.1% | 31 | 1.0% | .38 |
| Asian | 754 | 24.8% | 688 | 24.5% | 797 | 25.3% | 685 | 21.3% | |
| Black or African American | 234 | 7.7% | 209 | 7.5% | 243 | 7.7% | 237 | 7.4% | |
| Native Hawaiian or Other Pacific Islander | 18 | 0.6% | 21 | 0.8% | 27 | 0.9% | 10 | 0.3% | |
| Othera | N/A | N/A | N/A | N/A | 15 | 0.48% | 253 | 7.88% | |
| White | 1998 | 65.8% | 1857 | 66.2% | 2035 | 64.6% | 1994 | 62.2% | |
| Ethnicity | |||||||||
| Hispanic or Latino | 254 | 8.4% | 231 | 8.2% | 249 | 7.9% | 275 | 8.6% | .81 |
| Non-Hispanic | 2784 | 91.6% | 2575 | 91.8% | 2903 | 92.1% | 2935 | 91.4% | |
The “Other” response was added partway through the 2016 examination registration process.
Abbreviation: N/A, not available.
Figure 1.
Change in Preparedness to Lead Quality Improvement Among Graduating Family Medicine Residents by Year (N = 12 208)
In 2017, 3078 residents registered to take the certification examination. Table 2 shows the relationship between their level of self-reported preparedness to lead QI projects and the degree of participation in their PI activity. Residents reporting QI team leadership had 1.93 times higher odds of reporting feeling extremely prepared compared to moderately prepared (P ≤ .0001; OR 1.93; 95% CI 1.58–2.35). Participation with each PI activity task was significantly associated with being extremely prepared. The size of the associations was larger for tasks related to leadership rather than lower-level skills such as data entry.
Table 2.
Correlation Between Performance Improvement Activity Tasks and Self-Reported Quality Improvement Preparedness Among Graduating Family Medicine Residents in 2017 (N = 3152)
| Performance Improvement Activity Task | Moderately Prepared (N = 1756) | Extremely Prepared (N = 582) | P Value | Unadjusted Odds Ratio | 95% CI |
| Quality improvement team leadership | 440 (25.1%) | 228 (39.2%) | < .001 | 1.93 | 1.58–2.35 |
| Completion of written summary of project for academic presentation or paper | 279 (15.9%) | 142 (24.4%) | < .001 | 1.71 | 1.36–2.15 |
| Data abstraction | 796 (45.3%) | 330 (56.7%) | < .001 | 1.58 | 1.31–1.91 |
| Identification of quality measure/topic for improvement | 1276 (72.7%) | 461 (79.2%) | .002 | 1.43 | 1.14–1.80 |
| Identification of quality improvement intervention | 1092 (62.2%) | 408 (70.1%) | .001 | 1.43 | 1.17–1.74 |
| Quality improvement team participation (shared decision-making of project) | 710 (40.4%) | 281 (48.3%) | < .001 | 1.38 | 1.14–1.66 |
| Data entry | 1080 (61.5%) | 397 (68.2%) | .004 | 1.34 | 1.10–1.64 |
| Implementing chosen quality improvement intervention | 1261 (71.8%) | 444 (76.3%) | .035 | 1.26 | 1.02–1.57 |
Finally, we correlated faculty-assigned milestones with resident self-reported assessment of QI ability. The average midyear PBLI-3 faculty-assigned milestone rating in 2017 for residents who reported being “extremely prepared” was 3.28 compared to 3.17 for “moderately prepared” (Figure 2). This 0.11 difference on a 5-point scale, while statistically significant, is not educationally or clinically meaningful. Of note, all residents who received an evaluation of 5 for PBLI-3, the highest rating, reported feeling either “moderately” or “extremely prepared” (Table 3).
Figure 2.
Average Midyear 2016 PBLI-3 Milestone Score for Graduating Residents by Self-Reported Preparedness to Lead Quality Improvement (N = 3078)
Table 3.
Average Midyear 2017 PBLI-3 Milestone Score for Graduating Residents by Self-Reported Preparedness to Lead Quality Improvement (n = 3078)
| Resident Response | n | Average | SD | Minimum | Maximum |
| Not at all | 52 | 3.14 | 0.59 | 2 | 4.5 |
| Somewhat | 688 | 3.11 | 0.58 | 1 | 4.5 |
| Moderately | 1756 | 3.17 | 0.58 | 1 | 5 |
| Extremely | 582 | 3.28 | 0.58 | 1 | 5 |
Abbreviation: PBLI, practice-based learning and improvement.
Discussion
In this large national study of graduating family medicine residents over 4 years, we found that self-assessed preparation to lead QI efforts increased minimally. In addition, there was no correlation between faculty-assigned readiness to lead QI and resident self-assessment. To our knowledge, this is the first study to examine the yearly change in self-assessed QI preparedness of graduating family medicine residents nationally.
Between 2014 and 2017, more residents reported being either moderately or extremely prepared to lead QI projects, but the 3.1% increase suggests that the combined efforts to promote QI—from ACGME and ABFM requiring and facilitating QI project completion during residency8,20 to new payment models with specific quality measures21—are barely increasing residents' ability to do this type of work. Being able to effectively implement local system changes will be necessary for the next generation of family physicians as we shift from fee-for-service to value-based payments and continue to develop how we determine the quality of care we provide. Accrediting organizations and medical associations may need to consider whether the current approach is yielding the desired results.
The effect of PI activity participation on residents' self-reported ability was more prominent. Residents who reported feeling extremely prepared were significantly more likely to have completed each improvement activity component, which closely mirrors all aspects of a QI project. QI team leadership and completion of a written summary for academic presentation had the strongest association with feeling extremely prepared, whereas data entry and implementing the chosen QI intervention were the weakest. Among extremely prepared residents, less than half (48.3%, 281 of 582) reported shared decision-making of the project, and among those who reported being somewhat or not at all prepared, it was less than one-third (30.9%, 229 of 740). These results demonstrate the importance of ensuring residents participate in the full cycle of a QI project. Rather than simply “checking the box” to achieve compliance, programs should design opportunities that require resident leadership. Residents should have protected time to lead these projects and mentorship from qualified faculty to provide support and feedback throughout the process. Making these experiences substantive and effective could result in better learning and may also increase residents' confidence to lead future initiatives.
We also examined the relationship between self-assessed QI preparedness and faculty-assigned milestone rating for PBLI-3. While the average midyear score was statistically greater for “extremely prepared” compared to the other categories, the difference was not educationally meaningful (0.17 on a 5-point scale). These data suggest that residents' self-perceived ability in QI is not correlated to external review. Indeed, previous research has shown that people tend to be poor at self-assessment.22 The high threshold for a 4 or greater on the PBLI-3 milestone might also contribute to this inability to distinguish residents' QI ability. A rating of 3 includes “using a systematic improvement method (PDSA) to address an identified area of improvement” in addition to “using an organized method, such as registry, to assess population health,” while a 4 entails “establishing protocols for continuous review and comparison of practice . . . outcomes,” and 5 necessitates, “role models continuous QI of personal practice . . . using advanced methodologies.”23 Attaining a 3 is possible with only partial participation in a QI project whereas achieving a 4 requires successful completion and implementation of a significant project—a much higher standard. In addition, residents may self-report high levels of preparation because their selected QI project may have been simple to implement or they had a superficial experience and did not have a chance to appreciate the complexity of QI. True preparedness is likely to occur only after working on multiple projects over time and learning from failures.
This study focused exclusively on family medicine residents, thus generalizability to other specialties may be limited. In addition, our study was cross-sectional; therefore, we cannot make causal inferences based on our results. Most residents complete the questionnaire midway through their third year of training and, depending on curriculum design, may not have had their QI rotation or experience yet (thereby potentially lowering their milestone ratings) or may not have completed their PI activity. While there is evidence of validity for some questions on the ABFM examination questionnaire, respondents may have interpreted these questions differently, potentially introducing bias. Lastly, milestone ratings are dependent on local faculty's interpretation and assessment, which is variable across institutions.
Next research steps may include identifying new techniques for teaching QI. In particular, assessing whether requiring residents to lead QI initiatives result in higher self-assessed QI preparedness would be helpful for curriculum design. The ABFM has multiple pathways for achieving credit toward the PI activity; the impact of completing an ABFM-designed activity compared to one specific to the resident's local residency or health center should be examined as well. Finally, the minimal difference in faculty-assigned milestone ratings suggests that this milestone may need to be modified to better identify resident differences in QI ability.
Conclusions
Over the past 4 years, family medicine residents' self-assessed preparedness to lead QI projects has barely increased. We found no correlation between self-reported ability to lead QI and faculty-assigned milestone rating. Leading a QI project in residency is likely to be particularly important to feeling more prepared.
References
- 1.Accreditation Council for Graduate Medical Education. ACGME Common Program Requirements. 2019 https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRs_2017-07-01.pdf Accessed August 20.
- 2.American Board of Medical Specialties. Based on Core Competencies. 2019 https://www.abms.org/board-certification/a-trusted-credential/based-on-core-competencies Accessed August 20.
- 3.111th Congress (2009–2010) HR3200—America's Affordable Health Choices Act of 2009. 2019 https://www.congress.gov/bill/111th-congress/house-bill/3200/text#toc-H34FE0036F4104A87A321C48E7316FD3B Accessed August 20.
- 4.Institute for Healthcare Improvement. The IHI Triple Aim. 2019 http://www.ihi.org/Engage/Initiatives/TripleAim/Pages/default.aspx Accessed August 20.
- 5.Boonyasai RT, Windish DM, Chakraborti C, Feldman LS, Rubin HR, Bass EB. Effectiveness of teaching quality improvement to clinicians. JAMA. 2007;298(9):1023–1037. doi: 10.1001/jama.298.9.1023. [DOI] [PubMed] [Google Scholar]
- 6.Wong BM, Etchells EE, Kuper A, Levinson W, Shojania KG. Teaching quality improvement and patient safety to trainees: a systematic review. Acad Med. 2010;85(9):1425–1439. doi: 10.1097/ACM.0b013e3181e2d0c6. [DOI] [PubMed] [Google Scholar]
- 7.Philibert I, Gonzalez del Rey JA, Lannon C, Lieh-Lai M, Weiss KB. Quality improvement skills for pediatric residents: from lecture to implementation and sustainability. Acad Pediatr. 2014;14(1):40–46. doi: 10.1016/j.acap.2013.03.015. [DOI] [PubMed] [Google Scholar]
- 8.Accreditation Council for Graduate Medical Education. CLER Pathways to Excellence: expectations for an optimal clinical learning environment to achieve safe and high quality patient care. Executive Summary. 2019 https://www.acgme.org/Portals/0/PDFs/CLERExecutiveSum.pdf Accessed August 20.
- 9.Accreditation Council for Graduate Medical Education. Issue Brief No. 3 Health Care Quality. CLER Clinical Learning Environment Review. National Report of Findings 2016. 2019 https://www.acgme.org/Portals/0/PDFs/CLER/ACGME_CLER_Health_Care_Quality.pdf Accessed August 20.
- 10.Leddy R, Lewis M, Ackerman S, Hill J, Thacker P, Matheus M, et al. Practical implications for an effective radiology residency quality improvement program for milestone assessment. Acad Radiol. 2017;24(1):95–104. doi: 10.1016/J.ACRA.2016.08.018. [DOI] [PubMed] [Google Scholar]
- 11.American Academy of Family Physicians. Residency Directory. 2019 https://nf.aafp.org/Directories/Residency/Results Accessed August 20.
- 12.American Board of Family Medicine. ABFM News for Family Medicine Residency Directors. 20122019 July. https://www.theabfm.org/sites/default/files/2019-02/ProgramDirectorNewsJuly2012.pdf Accessed August 20.
- 13.American Board of Family Medicine. Family Medicine Certification Examination. 2019 https://www.theabfm.org/cert/index.aspx Accessed August 20.
- 14.Simasek M, Ballard SL, Phelps P, Pingul-Ravano R, Kolb NR, Finkelstein A, et al. Meeting resident scholarly activity requirements through a longitudinal quality improvement curriculum. J Grad Med Educ. 2015;7(1):86–90. doi: 10.4300/JGME-D-14-00360.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Mathias E, Sethuraman U. ABCs of safety and quality for the pediatric resident and fellow. Pediatr Clin North Am. 2016;63(2):303–315. doi: 10.1016/j.pcl.2015.11.003. [DOI] [PubMed] [Google Scholar]
- 16.Peterson LE, Fang B, Phillips RL, Avant R, Puffer JC. A certification board's tracking of their specialty: the American Board of Family Medicine's data collection strategy. J Am Board Fam Med. 2019;32(1):89–95. doi: 10.3122/jabfm.2019.01.180138. [DOI] [PubMed] [Google Scholar]
- 17.Xierali IM, Phillips RL, Green LA, Bazemore AW, Puffer JC. Factors influencing family physician adoption of electronic health records (EHRs) J Am Board Fam Med. 2013;26(4):388–393. doi: 10.3122/jabfm.2013.04.120351. [DOI] [PubMed] [Google Scholar]
- 18.Sairenji T, Dai M, Eden AR, Peterson LE, Mainous AG., 3rd Fellowship or further training for family medicine residents. Fam Med. 2017;49(8):618–621. [PubMed] [Google Scholar]
- 19.Weidner AKH, Chen FM, Peterson LE. Developing the national family medicine graduate survey. J Grad Med Educ. 2017;9(5):570–573. doi: 10.4300/JGME-D-17-00007.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Accreditation Council for Graduate Medical Education. Scholarly Activity Guidelines Review Committee for Family Medicine. 2019 https://www.acgme.org/Portals/0/PFAssets/ProgramResources/120_scholarly-activity-guidelines.pdf Accessed August 20.
- 21.Centers for Medicare & Medicaid Innovation. BPCI Advanced. 2019 https://innovation.cms.gov/initiatives/bpci-advanced Accessed August 20.
- 22.Kruger J, Dunning D. Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. J Pers Soc Psychol. 1999;77(6):1121–1134. doi: 10.1037/0022-3514.77.6.1121. [DOI] [PubMed] [Google Scholar]
- 23.Accreditation Council for Graduate Medical Education; American Board of Family Medicine. The Family Medicine Milestone Project. 2019 https://www.acgme.org/Portals/0/PDFs/Milestones/FamilyMedicineMilestones.pdf?ver=2017-01-20-103353-463 Accessed August 20.