A Quality Improvement Approach to Modification of a Point-of-Care Ultrasound Curriculum

Abstract

Background

There is increasing emphasis on resident involvement in quality improvement (QI) efforts, yet resident engagement in QI has remained low for many reasons. Although QI methods are classically applied to clinical processes, there are many opportunities to incorporate QI principles into curricular design and implementation.

Objective

Demonstrate the utility of QI methods when applied to curricular design and the implementation of a novel point-of-care ultrasound portfolio development and quality assurance program at a large internal medicine residency program.

Methods

We applied foundational QI methods, including process mapping, plan–do–study–act (PDSA) cycles, time-trap identification, run-chart analysis, and qualitative interviews throughout the curricular design and implementation phases to rapidly identify areas for improvement and perform timely tests of change.

Results

Fifty-one interns participated in the curriculum, submitting 731 images in the first trimester. Process mapping and submission review revealed that 29% of images were saved to the incorrect digital archive. Resident–reviewer interpretation concordance was present in 80.7% of submissions. In 95.2% of completed quality assurance cards, the same information was provided in the commentary feedback and the evaluator’s checklists, representing a time trap. Interventions included restricting access to image archives and removing redundant fields from quality assurance cards. The time to feedback fell from 69.5 to 6.5 days, demonstrating nonrandom variation via run-chart analysis.

Conclusion

This pilot study demonstrates the successful application of QI methods to a novel point-of-care ultrasound curriculum. The systematic use of these methodologies in curricular design and implementation allows expeditious curricular improvement. Emphasizing the relevance of QI methods to subject matter beyond clinical processes may increase resident engagement in QI efforts.

Background

Point-of-care ultrasound (POCUS) is a rapidly growing area of education within internal medicine (IM) and critical care, with support from several professional organizations to incorporate POCUS in IM training (1–3). The potential to improve patient safety and increase the rapidity of diagnosis by incorporating POCUS into clinical practice is substantial (4–6). However, POCUS competency requires supervised practice to ensure skill development and prevent misapplication, which can contribute to poor patient outcomes (5, 7). POCUS training should involve not only image acquisition and interpretation but also portfolio development and expert feedback (1, 2, 8). Although many IM residency programs have implemented POCUS curricula, few include portfolio development and quality assurance (QA) (2).

The real-time improvement of existing curricula is often limited by assessment and modification occurring exclusively at the end of a curricular cycle rather than throughout the academic year. In addition, a common barrier to educational interventions’ success is a propensity toward “solutionism,” described as “the tendency to jump to a solution without incorporating an understanding of the deeper theory and evidence to design a robust and thoughtful implementation approach” (9, 10). The deliberate and systematic application of quality improvement (QI) methodology to curricular development can mitigate these effects. Although QI methods are classically applied to clinical processes, there are many opportunities to incorporate QI principles into curricular design and implementation, which requires frequent assessment to ensure ongoing effectiveness.

We describe the application of QI methodology to improve POCUS education through the initiation of a novel, longitudinal POCUS program that emphasizes portfolio development and QA.

Aim

We aimed to apply QI methodologies to a novel POCUS portfolio QA program at a large IM residency to improve the timeliness of learner feedback to less than one week within one academic year.

Methods

We followed the Standards for Quality Improvement Reporting Excellence 2.0 guidelines for reporting this intervention (11). We used foundational QI methods, including process mapping, iterative plan–do–study–act (PDSA) cycles, time-trap identification, run-chart analysis, and “voice-of-the-customer” interviews during the curricular design and implementation phases to rapidly and accurately identify areas for improvement and perform timely tests of change.

Baseline State

Before implementing our curriculum, resident POCUS education consisted of intermittent lectures, electives with limited availability, and self-directed learning. Our needs assessment included Likert-style questions assessing resident desire for expanded POCUS training in the end-of-year program evaluation preceding curricular development. In addition, residents did not have the ability to develop image portfolios. No mechanism existed for residents to receive formal feedback on acquired images.

Curricular Development

Key stakeholders including local POCUS experts, residents, and program leadership convened to outline curricular objectives and scope. This included collecting voice-of-the-customer data to identify residents’ goals for the curriculum.

Curriculum Overview

We developed a yearlong multifaceted POCUS curriculum for first-year IM residents. The curriculum includes small-group didactics, mentored scanning with standardized and hospitalized patients, portfolio development via independent scanning, and QA by expert POCUS faculty. Residents received access to Butterfly iQ handheld ultrasound devices at each clinical site and individual Butterfly Cloud accounts to build and store their portfolios of at least 25 images. Didactics were temporally paired with mentored scanning to reinforce content and provide an opportunity for POCUS experts to provide real-time feedback. The 25-image portfolio included standard views covered in the didactics. Using the Butterfly iQ Cloud QA program, a customized digital “worksheet” was designed for each required portfolio image. The digital worksheets directed the learner to complete a self-QA process that included the identification of structures and image interpretation to reinforce a systematic approach to image optimization. The worksheets contained the components of an optimized POCUS image, modeled after the requirements detailed by the American College of Chest Physicians POCUS Certificate of Completion and were available for review by residents before scanning (12). The worksheets were digitally linked to the images Butterfly cloud–based portfolio and submitted for review by a small group of faculty experts, all certified by a professional society or local equivalent. Reviewers attached and completed a digital “QA card” for each submitted worksheet to provide feedback on the image quality and image interpretation. Once the QA card was submitted, the resident was notified automatically via e-mail and the Butterfly iQ app. Time to feedback, resident–reviewer interpretation concordance, and the percentage of passing images were the outcome metrics selected for the data-driven assessment of achieved curricular objectives and efficacy of interventions. These data were collected monthly via audits of the Butterfly iQ Cloud.

Qualitative feedback from residents and POCUS faculty was obtained via informal focus groups and organized into themes.

Three iterative PDSA cycles were performed, which included restricted access to image archives (PDSA 1), modified QA cards (PDSA 2), and feedback standardization (PDSA 3). We used run-chart analysis to assess the impact of interventions on time to feedback and assess for nonrandom variation.

Results

Needs Assessment

Program leadership identified the need for a structured POCUS curriculum to meet updated IM training recommendations (1, 3, 8). Data from the program evaluation survey revealed that 133 (94.7%) of residents desired additional POCUS training.

Curricular Design

Stakeholder interviews identified covering basic knobology, image acquisition, image interpretation, and clinical integration as major curricular objectives. A faculty-identified curricular priority was the delivery of specific, actionable, and timely feedback on acquired images.

Timeliness of Feedback

All 51 categorical interns participated in the curriculum, submitting 731 images in the first four months. Process mapping (Figure 1) and submission review revealed that 29% of images were saved to the incorrect digital archive, requiring manual relocation before QA. Informal qualitative feedback from residents and faculty revealed a desire for more expedient feedback delivery, confusion regarding how to appropriately save images, and the perception that QA cards were unnecessarily long. POCUS faculty believed that QA card length made it difficult to complete reviews in a timely fashion. There were two components to each QA card, including a checklist and a section for comments. We found that 95.2% of completed QA cards provided the same information in the commentary feedback and the evaluators’ checklists. Process mapping identified redundant fields on QA cards and image relocation to appropriate archives as “time traps,” defined as “any process step that inserts delay time into a process” (13). After PDSA cycle 1 (restricting access to archives) and PDSA cycle 2 (removing redundant fields from QA cards), the average number of QA card fields fell from 8.2 to 4 over the 4 months, and the average time to feedback fell from 69.5 to 6.5 days, demonstrating nonrandom variation via run-chart analysis (Figure 2).

Figure 1.

POCUS portfolio development process map. POCUS = point-of-care ultrasound; QA = quality assurance.

Figure 2.

Run chart: time to feedback for consecutive worksheet submissions. PDSA = plan–do–study–act.

Quality of Feedback

All 64 worksheets with discordant interpretations between the resident and faculty reviewer received commentary feedback specifically addressing the difference in interpretation. Twenty-eight (8.5%) worksheets received failing grades. Four of the 28 failed worksheets (14.3%) did not receive commentary feedback as to why the submission did not pass. Careful review of passing worksheet submissions by two reviewers demonstrated that 32 (9.7%) had easily identifiable opportunities for improvement but did not receive commentary feedback addressing the opportunity (e.g., excess depth of >10 cm, flipped image). In addition, residents frequently reported that the feedback from the checklist was formatted in a way that was difficult to understand.

Curricular Efficacy

Of the 331 submitted worksheets, 267 (80.7%) had image interpretation concordance between the resident and faculty reviewer. The majority of studies with discordant interpretations were assessing pathologic fluid collections.

Rationale for Interventions

PDSA 1: Restricting Access to Archives

Twenty-nine percent of images were saved to the incorrect archive, requiring manual relocation to be accessible to the reviewer. The Butterfly iQ cloud automatically saves images to a “default archive,” which is not monitored by reviewers unless manually changed. This resulted in additional monitoring needs, time for relocation, and time spent educating residents on the existing protocol. After defining the magnitude of the problem, it was evident that the educational intervention was insufficient, and a system-level change was necessary. Access to image archives was modified so that each resident could exclusively save images to the correct location, eliminating the need for monitoring and relocation.

PDSA 2: QA Card Modification

Qualitative interviews with POCUS faculty revealed a perception of unnecessarily long QA cards, making it difficult to complete reviews in a timely fashion. Although the comments on the QA cards provided an opportunity to stress the most relevant and nuanced feedback, we found that they repeated all the data from the checklist in approximately 95% of the assessments. In addition, the residents found that the feedback from the checklist was formatted in a way that was difficult to understand. Thus, the inclusion of checklists on the QA cards was determined to be a no-value-added step that contributed to delays in feedback delivery. Of note, the checklists were retained on the residents’ worksheets to reinforce a structured approach to self-review of acquired images.

PDSA 3 (Ongoing): Feedback Standardization

Our initial assumption was that creating a checklist, modeled after professional societies’ portfolio requirements, would be adequate to ensure a standardized review process; however, data analysis revealed that >90% of images received passing grades, approximately 10% of passing images had easily identifiable opportunities for improvement without corresponding comments, and several failed images did not have comments addressing the reason for failure, representing an opportunity to improve quality and standardize the content of feedback. Subsequently, qualitative feedback from reviewers revealed that 1) reviewers believed that strict adherence to the professional standards was unrealistic for novice scanners, 2) there was a hesitance/sense of guilt when failing images, and 3) there was no clear process on how to ensure that residents had opportunities to fulfill the portfolio requirements after failed submissions. Interventions included formalized expectations that all studies receive, at a minimum, commentary feedback on knobology, depth, axis, incorporating image review in regularly scheduled meetings, and the identification of key requirements that, if missing, constituted a failed study. Data collection is ongoing to assess the efficacy of this intervention.

Discussion

Lessons Learned

Data from a national survey of IM program leadership revealed barriers to implementing POCUS curricula, including the cost of equipment and a paucity of local expertise, among others, resulting in few, if any, curricula incorporating portfolio development with QA (2). Our experience demonstrates the feasibility of implementing a QA program at a large residency program with few faculty experts and serves as a roadmap to adapting curricula/processes to institution-specific barriers with the application of QI methodologies.

It was evident early in the implementation process that images were frequently saved to the incorrect location. Our initial response was to redirect residents to the protocol for appropriately saving images. In essence, we implemented an educational intervention without fully understanding the problem, also known as solutionism. After defining the magnitude of the problem, we realized that our educational intervention was insufficient and that a system-level change was necessary, prompting the design of PDSA 1. This reemphasized the need for a systematic, data-driven approach to interventions and that educational interventions that functionally tell people to “do better” are not as effective as system-level changes.

In addition, evaluating the resident–reviewer interpretation concordance allowed us to understand that most images with erroneous interpretations were made when assessing pathologic fluid collections. We can use this information when planning curricular revisions, redistributing the amount of time spent on various curricular objectives, and modifying the delivery/composition of existing content.

This project emphasized that we could effect change more efficiently by performing iterative PDSA cycles than with discussion and planning alone. It is unlikely that we could anticipate the limitations of our initial design without implementation, highlighting the need for a clear data collection and analysis plan.

Despite increasing emphasis on QI training by the Accreditation Council for Graduate Medical Education, resident engagement in QI has remained low for a multitude of reasons (14, 15). In contrast, resident involvement in clinician educator pathways is high and growing in popularity (16). Emphasizing the broader relevance of QI methods and potential application to curricular design and high-interest medical education topics may increase resident engagement in QI efforts and training.

Conclusions

This pilot study demonstrates the successful application of QI methods to a novel POCUS curriculum. The routine, deliberate, and systematic use of these methodologies in curricular design and implementation allows expeditious curricular improvement. Emphasizing the relevance of QI methods to subject matter beyond clinical processes may increase resident engagement in QI efforts and training.