Development and Assessment of a Web-Based Clinical Quality Improvement Curriculum

Abstract

Background

Understanding quality improvement (QI) is an important skill for physicians, yet educational interventions focused on teaching QI to residents are relatively rare. Web-based training may be an effective teaching tool in time-limited and expertise-limited settings.

Intervention

We developed a web-based curriculum in QI and evaluated its effectiveness.

Methods

During the 2011–2012 academic year, we enrolled 53 first-year internal medicine residents to complete the online training. Residents were provided an average of 6 hours of protected time during a 1-month geriatrics rotation to sequentially complete 8 online modules on QI. A pre-post design was used to measure changes in knowledge of the QI principles and self-assessed competence in the objectives of the course.

Results

Of the residents, 72% percent (37 of 51) completed all of the modules and pretests and posttests. Immediate pre-post knowledge improved from 6 to 8.5 for a total score of 15 (P < .001) and pre-post self-assessed competence in QI principles on paired t test analysis improved from 1.7 to 2.7 on a scale of 5 for residents who completed all of the components of the course.

Conclusions

Web-based training of QI in this study was comparable to other existing non–web-based curricula in improving learner confidence and knowledge in QI principles. Web-based training can be an efficient and effective mode of content delivery.

Editor's Note: The online version of this article contains the scoring rubric ^{(21.4KB, docx)} and validation evidence for the tool, the self-efficacy survey used in this study, and a table ^{(20.8KB, docx)} describing titles, objectives, and length of the developed online modules and mean scores of the 37 learners on the accompanying multiple-choice quizzes of the modules.

Introduction

Practice-based learning and improvement is a core competency,^1–3 yet limited evidence exists for how best to train physicians to lead quality improvement (QI) initiatives.⁴ The literature indicates that QI curricula are generally well accepted, improve knowledge, and can lead to improvements in processes of care.⁵ Challenges to the widespread introduction of QI curricula include a shortage of faculty expertise, limited faculty and trainee time, and a lack of stakeholder buy-in.^6–8 Web-based training may offer a solution by addressing learners' time constraints and expanding access to expertise.⁹ Although web-based training in medical education is promising,¹⁰ the effectiveness of such curricula in QI has not been widely studied.

We designed and assessed the feasibility and effectiveness of a web-based curriculum to teach QI principles to interns.

Methods

Curriculum Description

We used Ogrinc and Headrick's Fundamentals of Health Care Improvement: A Guide to Improving Your Patients' Care,¹¹ with permission from the publisher, to develop 8 web-based interactive modules. Interns in internal medicine, internal medicine-pediatrics, and internal medicine-psychiatry programs at Duke University Medical Center were enrolled in the course during the 2011–2012 academic year. Learners were provided an average of 6 hours of protected time during their 1-month geriatrics rotation to complete the modules and the accompanying pretests and posttests. Completion of the modules was tracked. E-mails were sent at the midpoint and end of the rotation to remind learners to complete the curriculum.

The study protocol was reviewed by Duke University's Institutional Review Board, which granted exemption.

Evaluation

A pre-post design was used to measure changes in QI knowledge and skills. We created a survey with a 5-point Likert scale (0  =  no confidence at all to 5  =  completely confident) based on the objectives of the modules to determine self-efficacy. Before viewing modules, learners completed the self-efficacy survey and the Quality Improvement Knowledge Application Tool (QIKAT) pretest. The QIKAT has previously been validated in trainees and has been shown to discriminate between learners with and without practice-based learning and improvement training.¹² This tool offers 3 scenarios, and learners must state an aim, measures, and an intervention associated with QI initiatives. The responses are scored from 0 to 5 for each scenario, giving a maximum score of 15 (provided as online supplemental material).¹³ The open-ended format forces the learners to spontaneously generate answers, avoiding cueing and enabling assessment of problem-solving ability. The learners then completed the modules in sequential order without bypassing any parts. Each module was immediately followed by a multiple-choice quiz. At the end of all 8 modules learners completed the same self-efficacy survey and the QIKAT posttest. Using the first 30 QIKAT scenarios, 2 authors (M.Y. and G.T.B.) refined their scoring techniques to achieve a weighted Kappa statistic of 0.90, which is considered excellent agreement.¹⁴ The same 2 authors then divided and scored the remaining QIKAT scenarios independently. Scorers were blinded to the identity of the learners and to the prestatus and poststatus of each QIKAT. Residents who completed the study were encouraged to complete open-text fields offering their feedback about the curriculum to the investigators.

The primary outcome measures were the change in the QIKAT scores. Secondary outcome measures were the self-efficacy results.

Statistical Analysis

Paired t tests were conducted for the change scores (post-pre) for QIKAT and self-efficacy surveys to determine whether there was a significant change in knowledge (QIKAT) and confidence (self-efficacy). The analysis was conducted using SPSS Version 20 (IBM Corp, Armonk, NY).

Results

Of the interns assigned to the course, 96% (51 of 53) completed all of the modules and 72% (37 of 51) completed the accompanying pretests and posttests. Participants were mostly categorical interns (n  =  27) and women (n  =  20), and had a mean age of 28; few interns had prior QI experience (n  =  2).

A statistically significant difference in knowledge was seen between the preintervention and postintervention QIKAT scores (maximum score  =  15; preintervention mean  =  6.1, SD  =  1.8; postintervention mean  =  8.5, SD  =  3.2; P < .001; table). Statistically significant differences in confidence were also noted (scale of 0–4; preintervention mean  =  1.7, SD  =  0.65; postintervention mean  =  2.8, SD  =  0.68; P < .001; table), and there was improvement for all 16 items (figure). The mean scores on multiple-choice quizzes that followed the 8 modules are shown in the table provided as online supplemental material. The average score on all the tests was 82% correct.

TABLE.

Knowledge and Self-Efficacy Scores of Interns Who Completed All Elements of the Course

FIGURE.

Interns' Self-Efficacy in Quality Improvement Concepts Measured Before and After Completion of the Modules (n  =  37)

Discussion

Our online QI curriculum demonstrated small gains in knowledge. Residents also reported improved confidence in proficiency of QI concepts. Our results are comparable to those from several other, non–web-based QI curricula that showed similar knowledge gains on QIKAT, despite involving different types of learners (a table presenting validity evidence of QIKAT in other settings is provided as online supplemental material).^12,15–17 Our results were significant, but improvement on QIKAT scores were modest and had a high standard deviation (3.2 for a test with a total score of 15). The latter may have resulted from the fact that some learners spent insufficient time on the free text responses. Interns scored an average of 2.5 points higher from baseline; on average, learners scored 2 points for each scenario before the course and 3 points after the course. According to the scoring criteria, a score of 2 indicated that the learner's response needed substantial modification, whereas a score of 3 was good, but the response needed modification. We believe this is a meaningful change in practice.

We used self-efficacy as 1 of the evaluation tools, as confidence and personal drive are major components in initiating and completing a QI project. Self-efficacy is a learner's belief that he or she can successfully complete a task. It predicts success because it influences choices, effort, and persistence in the face of challenges.^18,19 The results on the self-efficacy metric reflected positive change in the learner's confidence, but the results may be affected by self-report bias.

Learner feedback identified improvements for future implementation of the curriculum, including time for group discussions to facilitate active learning and reflection and shortened modules with the option to self-advance slides (after scoring the minimum passing score on the posttest to avoid advancing slides without focusing on the learning content).

Our study has several limitations. First, we did not include a hands-on QI project experience as part of the curriculum. This would have been desirable as health care is best improved through implementing skill-based activities in combination with theory instruction.^20–24 Additionally, the evaluation focused only on immediate knowledge and self-efficacy gain but not long-term retention.

Our web-based QI curriculum was generally acceptable and feasible for learners. The course promoted self-directed learning that can occur at the learner's convenience and required less faculty time than other methods of teaching QI. Our residency program did not have a preexisting QI curricula, and the web-based approach was a good starting point and foundation for future project work. The curriculum could be easily transferrable to other specialties with no need for additional costs or faculty expertise.

Conclusion

Our QI curriculum was effective, as evidenced by the statistically significant knowledge and confidence gain in the learners who completed the course. Positive learner feedback accompanied by improvements in QIKAT scores and self-efficacy in these specific content areas support this finding.