Quality and Safety Training in Primary Care: Making an Impact

Abstract

Purpose

Preparing residents for future practice, knowledge, and skills in quality improvement and safety (QI/S) is a requisite element of graduate medical education. Despite many challenges, residency programs must consider new curricular innovations to meet the requirements. We report the effectiveness of a primary care QI/S curriculum and the role of the chief resident in quality and patient safety in facilitating it.

Method

Through the Veterans Administration Graduate Medical Education Enhancement Program, we added a position for a chief resident in quality and patient safety, and 4 full-time equivalent internal medicine residents, to develop the Primary Care Interprofessional Patient-Centered Quality Care Training Curriculum. The curriculum includes a first-or second-year, 1-month block rotation that serves as a foundational experience in QI/S and interprofessional care. The responsibilities of the chief resident in quality and patient safety included organizing and teaching the QI/S curriculum and mentoring resident projects. Evaluation included prerotation and postrotation surveys of self-assessed QI/S knowledge, abilities, skills, beliefs, and commitment (KASBC); an end-of-the-year KASBC; prerotation and postrotation knowledge test; and postrotation and faculty surveys.

Results

Comparisons of prerotation and postrotation KASBC indicated significant self-assessed improvements in 4 of 5 KASBC domains: knowledge (P < .001), ability (P < .001), skills (P < .001), and belief (P < .03), which were sustained on the end-of-the-year survey. The knowledge test demonstrated increased QI/S knowledge (P  =  .002). Results of the postrotation survey indicate strong satisfaction with the curriculum, with 76% (25 of 33) and 70% (23 of 33) of the residents rating the quality and safety curricula as always or usually educational. Most faculty members acknowledged that the chief resident in quality and patient safety enhanced both faculty and resident QI/S interest and participation in projects.

Conclusions

Our primary care QI/S curriculum was associated with improved and persistent resident self-perceived knowledge, abilities, and skills and increased knowledge-based scores of QI/S. The chief resident in quality and patient safety played an important role in overseeing the curriculum, teaching, and providing leadership.

Editor's Note: The online version of this article contains an appendix of knowledge test scores by resident ^{(20.2KB, docx)} .

What was known

Residents need skills to improve the quality and safety of care, calling for new curricula and teaching approaches.

What is new

A VA GME enhancement program added staffing to develop an interprofessional patient-centered curriculum that offers foundational experiences in quality and safety for residents and supports a chief resident in quality and safety.

Limitations

Small sample, single-site study limit generalizability; outcomes are limited to self-assessed improvements in knowledge, ability, and skills.

Bottom line

The interprofessional quality curriculum resulted in a gain in self-reported knowledge, ability, skills and attitude, an increase in residents′ QI knowledge, and learner satisfaction.

Introduction

The Accreditation Council for Graduate Medical Education (ACGME) requirements state that residents must learn to advocate for quality and optimal patient care systems and work to enhance patient safety and care quality.¹ However, barriers to engaging residents in quality and safety include lack of curricular time, limited faculty knowledge and interest, and the absence of resident leadership.^2–4 Furthermore, studies on quality improvement and safety (QI/S) curricula have not shown that improvement in knowledge, skills, or attitudes are sustained over time.^5–7

At a Veterans Health Administration medical center, we developed a pilot QI/S curriculum that includes a chief resident in quality and safety. We administered before and after assessments to determine if a QI/S curriculum improved resident knowledge, skills, and attitudes, and if so, whether such improvements were sustainable over time.

Methods

Setting and Participants

The Loma Linda University Medical Center Internal Medicine Residency includes categorical, preliminary, and primary care tracks and trains approximately 100 residents annually who spend one-third of their time at the Veterans Affairs Loma Linda Healthcare System (VALLHCS). Through the Veterans Affairs (VA) Graduate Medical Education Enhancement Educational Innovations,^8,9 the internal medicine (IM) residency and VALLHCS received funding for 1 chief resident in quality and safety¹⁰ and 4 full-time equivalent IM resident positions for academic year 2010–2011. The 4 positions were filled with postgraduate year (PGY)–1 through PGY-3 residents who were assigned to a required 1-month block rotation where they spent 3 half days per week in QI/S didactics, self-study, and projects in addition to primary care, interprofessional, and outpatient clinical experiences through the Primary Care Interprofessional Patient-Centered Quality Care Training (PC-ImPaCQT) Program, an educational innovation encompassing the block rotation and IM continuity clinic. The study was reviewed and approved by the VALLHCS Research and Development Committee.

Quality Improvement and Safety Curriculum

Based on the ACGME competencies and IM milestones,¹¹ the curriculum was developed by an interprofessional planning committee made up of residency program and VA leadership, physicians with QI/S expertise, the VA patient safety officer, a performance improvement consultant, and the chief resident in quality and safety.

The chief resident in quality and safety (hereafter “chief resident”) position is for residents who have completed residency training and seek expertise in QI/S during a non–ACGME-accredited year. The chief resident's duties included organizing and overseeing the QI/S curriculum, teaching quality improvement and patient safety through interactive didactic sessions, mentoring required resident QI/S projects, as well as teaching introduction to patient safety, and root cause analysis, common causes of malpractice lawsuits, medication reconciliation, and formal chart auditing. Human factors engineering was taught in a scenario-based, interactive workshop. Quality topics taught by the chief resident and other faculty included an introduction to QI, plan-do-study-act cycles, and process modeling including flow charts and fishbone diagrams.

To prepare for the role, the chief resident attended the VA National Center for Patient Safety (NCPS) fellows course, and completed Institute for Healthcare Improvement Open School self-study modules on QI, measuring improvement and safety. Mentoring was provided by several faculty members with QI/S experience and the patient safety officer. The chief resident spent 4 half days in clinical care managing a primary care patient panel and leading and teaching trainees in group medical appointments. The chief resident also presented morbidity and mortality conferences, and served on medical center committees.

Outcome Measures

Quality improvement and safety education outcomes were determined by using the resident-reported knowledge, abilities, skills, beliefs, and commitment (KASBC) survey⁵ and a knowledge test. The KASBC survey contains 5 domains with 3 to 6 questions on knowledge, attitudes, skills, beliefs, and commitment, and was completed voluntarily by residents online at the start of the block rotation, post rotation, and at the end of the academic year. Though not validated, it has been used in a setting similar to ours.⁵

The knowledge test is a modification of the validated Quality Improvement Knowledge Assessment Test,^6,7 which was used to assess a QI/S curriculum in another VA medical center.⁵ The knowledge test consists of 2 pretest and 2 posttest scenarios with 5 questions each on system errors, strong interventions, aims, metrics, and sustainability. Responses to each question were scored by 2 independent raters who were blinded to both resident identity and pretest or posttest status. Combined scores ranged from 0 to 15, with higher scores indicating more knowledge.

The authors developed the postrotation resident and faculty surveys and pilot tested them on a small group of residents and faculty. On the monthly postrotation survey, residents reported on their educational experiences. Faculty mentors and planning committee members completed an evaluation of the chief resident at the end of the academic year.

Data Analysis

The resident and faculty surveys were analyzed with descriptive statistics. We compared pretest KASBC with postrotation scores, and prerotation with end-of-year scores for each domain and element. Differences were computed by using generalized linear models with identity links, normal distributions, and Wald χ² test statistics. Estimates were adjusted for respondent training level, percentage of time spent in VA settings, and number of months in primary care during the current academic year. Sensitivity analysis was performed for missing end-of-year data by replacing the end-of-year score with the respondent's prerotation score.

Prerotation and postrotation differences in resident knowledge test scores were computed across reviewers by using a generalized estimating equation with linear linking function, which are necessary because scores will be correlated whenever both reviewers score the same resident and time period. Interrater reliability was evaluated by using intraclass correlations based on 2-way mixed effects models.

Results

On prerotation surveys, 43% (17 of 40) of residents assigned to the rotation were PGY-1; 13% (5 of 40), PGY-2; and 33% (13 of 40), PGY-3; and 63% (25 of 40) indicated they had spent no more than 1 month at the facility before the rotation, with 2 residents providing no response.

Twenty-one of 42 residents (50%) completed both prerotation and postrotation knowledge tests. Cronbach α assessing consistency across reviewers was 0.589, indicating each reviewer was assessing different aspects of the resident responses to knowledge test questions. The pre-post knowledge test difference was 0.964 (95% confidence interval, 0.34–1.59; P  =  .002), indicating that residents had increased knowledge after completing the program (appendix provided as online supplemental material).

The postrotation survey was completed by 33 residents, with 76% (25) and 70% (23) rating the quality and the safety curriculum as always educational or usually educational, respectively (scale: always educational, usually educational, educational, usually not educational, never educational). Answers to open-ended questions from the postrotation survey reflect a general positive impression of the curriculum and the chief resident (table 1).

TABLE 1.

Selected Quality and Safety Curriculum Postrotation Narrative Responses

The faculty survey was completed by 10 of 15 mentors (66%) and planning committee members, all of whom had contact with the chief resident: 20% (2) daily, 50% (5) weekly, and 30% (3) monthly. Participants agreed or strongly agreed with the following statements of the chief resident role: increased faculty commitment and involvement with QI/S (70%, 7of 10); important to QI/S curriculum success and facilitated resident involvement (90%, 9 of 10); enhanced the culture of QI/S among faculty and residents (80%, 8 of 10); and served as a systems-based practice role model for residents (70%, 7 of 10).

Rate of KASBC survey completions varied, with 30 of 42 residents (71%) completing the prerotation survey, 26 of 42 (62%) completing the postrotation survey, and 26 of 42 (62%) completing the end-of-year survey. table 2 presents presurvey-postsurvey comparisons showing large, statistically significant increases in the knowledge, skills, and ability domains (P < .001) (and in their component elements), but not in the belief or commitment domains where overall scores tended to be higher than other domains. Self-assessed knowledge, abilities, and skills, and all corresponding elements except one, are statistically significantly higher than pretest scores, but not lower than posttest means. The sensitivity analysis showed no change when “no difference” scores were imputed to missing values.

TABLE 2.

Knowledge, Abilities, Skills, Beliefs, Commitment (KASBC) of Quality and Safety: Comparison of Prerotation, Postrotation, and End-of-Year Surveys^a

TABLE 2.

Continued

Residents originated and developed a number of QI/S projects.

Costs including salary and benefits were $62 153.83 to $66 336.23, per resident (PGY-1 to PGY-3) for the 4 additional full-time equivalent IM residents and $90 775.88 for the chief resident. The NCPS conference was tuition free with travel costs of about $1000. Faculty contributed to QI/S teaching 1 to 2 half days per month and provided ongoing chief resident mentoring throughout the year.

Discussion

Our institution has long acknowledged the need to involve residents in QI/S. However, like others, we had difficulty dedicating resident full-time equivalent to the residency rotation schedule and identifying faculty with the expertise and willingness to teach it. The VA Graduate Medical Education Enhancement Program provided the opportunity for a foundational QI/S curriculum in an IM residency 1-month block rotation. Like others, we demonstrated the educational effectiveness of a didactic curriculum by using a variety of assessment strategies including self-assessed knowledge and skills, a knowledge test, and a rotation evaluation.^5–7 In contrast to a previous study,⁵ we did not find increases in the self-assessed belief and commitment to QI/S. This may be the result of residents having completed the pretest after an introductory didactic session about the QI/S curriculum. Our data do show that the curriculum was effective in significantly increasing residents' self-perceived knowledge, skills, and abilities, and improvements that persisted after the rotation.

These self-reported findings were confirmed in part by a modest increase in knowledge measured at the end of the block rotation. Compared to prior studies for which scores ranged from 2.2 to 4.6 on the 15-point scale,^6,12,13 the adjusted difference of just less than 1 in this study, while statistically significant, may not represent an educationally meaningful difference. However, given the dramatic increase in self-perceived knowledge, the lack of difference on the knowledge test may be attributable to not having the raters score a sample of knowledge tests together to establish interobserver reliability.⁷

The chief resident provided leadership and dedicated faculty support for the program. Chief residents have an important role in administration, teaching, and clinical care in residency programs.^14,15 They act as a liaison between residents, institutional leadership, and faculty.¹⁶ They can also be effective in teaching and contributing to the professional development of residents.¹⁷ Given the barriers to resident involvement in QI/S, chief residents whose responsibilities are directed to teaching these skills may be ideally suited to communicate with residents, role model desirable behaviors, and facilitate resident involvement.³ While other faculty members could have perhaps filled the role, the chief resident quickly implemented our QI/S curriculum by providing expertise, dedicated teaching time, and leadership in a much needed educational program.

Our study has several major limitations. First, while self-assessed knowledge and skills are often used in studies of QI/S curricula, the validity of such an approach has been questioned.¹⁸ Second, our curriculum has only been in place for 1 year and has not demonstrated long-term sustainability. Third, we did not link pre-KASBC, post-KASBC, and end-of-year KASBC responses to individual residents in order to preserve anonymity and encourage candid responses. Potential biases were reduced by adjusting statistically for differences in respondent characteristics. Fourth, missing data were managed by using sensitivity analyses, and our essential findings remain even when missing values were replaced with scores that bias estimates towards no difference.

Conclusion

A Veterans Affairs primary care QI/S curriculum, enhanced by the teaching and leadership of a chief resident in quality and safety, led to increased and sustained self-assessed resident knowledge, skills, and abilities in quality and safety.