Supplemental Digital Content is Available in the Text.
Keywords: quality measurement, program evaluation, measurement for improvement, data skills, continuing professional development
Introduction:
The ability of health care professionals to measure change is critical for successful quality improvement (QI) efforts. Currently, there are no systematic reviews focusing on continuing education for health care professionals in data skills for QI. The purpose of this systematic review is to define effectiveness and sustainability of QI programs for health care professionals containing a measurement skills component and to identify barriers and facilitators to effectiveness and sustainability.
Methods:
The systematic review involved study identification, screening, full text review, and data extraction. Four electronics databases and grey literature sources were searched to identify studies published between 2009 and 2019 (11 years). A customized data extraction form was developed. Mixed methods appraisal tool was used for quality assessment and a thematic analysis was conducted for narrative synthesis.
Results:
Fifty-three studies from 11 countries were included. Most study designs were quantitative descriptive (17/53) and used a blended learning approach (25/53) combining face-to face and distance learning modes. The programs included basic, intermediate, and advanced data skills concepts. Overall, studies reported positive outcomes for participant reaction, learning, and behavior, but reported variable success in sustainability and spread of QI.
Discussion:
Studies discussed measurement as a key competency for clinical QI. Effectiveness definitions focused on the short-term impact of the programs, whereas sustainability definitions emphasized maintenance of outcomes and skills in the long-term. Factors that influenced effectiveness and sustainability of the included studies were strategic approach to QI, organizational support, intervention design, communication, accountability, leadership support, and learning networks.
Health care organizations worldwide continue to test new systems and ways to enhance health care quality and patient safety.1 Organizations are using continuing education programs in quality improvement (QI) methodologies to transform care and improve patient safety, reduce variations in care outcomes, and deliver sustainable changes in the health care system.2 The use of such programs to improve health care has also gained considerable popularity in the health care system.3 However, the health care system is complex and professional knowledge alone is not enough to engage in QI work to bring about change.4 Numerous QI training programs have been developed to train health care staff in QI methodology and application.
QI training can improve processes, staff knowledge, and health outcomes.5 Measurement is an important construct in all QI efforts because unless we measure, it is impossible to demonstrate whether the change has resulted in an improvement or not.6 For health care staff today, collecting, processing, and understanding data is a part of routine practice.7 This makes a strong case to train health care staff in quality measurement and to develop their expertise in the use of data.8
Although there are several systematic reviews evaluating QI training and curricula,9–11 none have focused on the evaluation of measurement for improvement training components. This systematic review differs from previous reviews by focusing on QI curricula and training programs containing a significant data skills component. The concepts of effectiveness and sustainability are critical to assessing the impact of teaching measurement skills to health care staff, but these concepts are underexplored in the QI literature. Effectiveness is a micro concept and refers to the assessment of the usefulness of an output at a certain point, with little reference to context. On the other hand, sustainability is a macro concept which extends over a longer period as the new ways of working or improved outcomes become the norm, with context being an essential element.12 The purpose of this systematic review is to address this gap in literature and define effectiveness and sustainability of QI programs for health care professionals that have a data for improvement component and to identify the associated barriers and enablers.
METHODS
Protocol and Registration
Review protocol for this systematic review is registered on PROSPERO (ID: CRD42019122997). This study was approved by the IRB of our institution.
Eligibility Criteria
Studies were included if:
Conducted in health care setting
Intervention was QI-based training and included a measurement component.
Study was about development, evaluation, or implementation of the program
Population was health care staff or postgraduate students
Based on primary research
Studies were excluded if:
There was no measurement for improvement component in intervention
Conference proceedings
Population was undergraduate students
Information Sources
Systematic review protocols were scanned in Prospero and Cochrane library to ensure novelty of the review question. A scoping search of databases was conducted to inform the development of the search strategy. Databases were purposively selected to include health care and education sources. The databases were: PubMed, CINAHL Plus, ERIC (via Pro-Quest), and Web of Science. Grey literature sources included two databases: OAIster and OpenGrey along with websites of leading organizations (see Supplemental File 1, Supplemental Digital Content 1, http://links.lww.com/JCEHP/A103). The reference lists of eligible studies were scanned to identify additional papers.
Search
The search strategy (see Supplemental File 2, Supplemental Digital Content 2, http://links.lww.com/JCEHP/A104) was optimized toward sensitivity rather than specificity because the scoping search revealed that measurement for improvement was integrated into QI studies rather than being delivered as a standalone training.13 The authors finalized the search strategy and databases iteratively. The systematic search of the literature was conducted in January 2019 and updated in June 2020. The search was restricted to papers published in last 11 years (Search date: January 1, 2009–December 31, 2019). Foreign language papers with English abstracts were considered at the initial stage but only included in full text review if a complete translation was available.
Study Selection
The systematic review consisted of four stages: study identification, title and abstract screening, full text review, and data extraction. Study screening was completed using Covidence tool.14 Two reviewers independently conducted title and abstract screening. The reviewers met regularly to resolve disputes. The full text review was also conducted independently by two reviewers and discrepancies resolved via discussion. The two reviewers consulted a third reviewer to assist in making the decision on one paper at the full text review stage. Because the studies were heterogenous, a narrative synthesis was performed.
The database search returned 6184 articles, which were imported into Covidence. The 2499 duplicates were removed, leaving 3685 studies eligible for screening. After screening, 110 studies were shortlisted for full text review. A total of 53 studies were included in the review. The PRISMA flow diagram is presented in Figure 1 and the checklist is attached in Supplemental Digital Content 3 (see Supplemental File 3, http://links.lww.com/JCEHP/A105).15
FIGURE 1.
PRISMA Statement. An overview of the study selection process. The 6184 records identified through database searching were screened by 2 reviewers. Nineteen records were identified and included from grey literature sources. Exclusion reasons at each stage are shown.
Assessment of Methodological Quality
The Mixed Methods Appraisal Tool (MMAT) was used to evaluate methodological quality.16 The validity and reliability of the MMAT has been established and is suitable for appraising mixed method studies.17 Two reviewers assessed quality independently and results were compared. Studies meeting the screening questions of the MMAT on clarity of research questions and appropriateness of collected data were considered appropriate quality for inclusion. All 53 studies met these criteria and were included in the review. The quality assessment is presented in Table 1.
TABLE 1.
Assessment of Methodological Quality Using MMAT*
| Study | Screening Questions | Qualitative | Quantitative | Mixed Methods | |||||||||||||||||||||||
| Randomized Control Trial | Nonrandomized CT | Descriptive | |||||||||||||||||||||||||
| S1 | S2 | 1.1 | 1.2 | 1.3 | 1.4 | 1.5 | 2.1 | 2.2 | 2.3 | 2.4 | 2.5 | 3.1 | 3.2 | 3.3 | 3.4 | 3.5 | 4.1 | 4.2 | 4.3 | 4.4 | 4.5 | 5.1 | 5.2 | 5.3 | 5.4 | 5.5 | |
| Adams et al18 | Y | Y | Y | C | Y | Y | Y | Y | Y | Y | C | C | Y | Y | Y | Y | Y | ||||||||||
| Bardfield et al19 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Barker et al20 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Berry21 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Bidassie et al22 | Y | Y | Y | Y | Y | C | Y | ||||||||||||||||||||
| Brandrud et al4 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Bundy et al23 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Butler et al3 | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | ||||||||||
| Chinthammit et al24 | Y | Y | Y | Y | Y | C | Y | ||||||||||||||||||||
| Cosimi et al25 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Cranley et al26 | Y | Y | Y | Y | C | C | C | Y | Y | Y | Y | Y | Y | C | C | C | C | ||||||||||
| Davis et al27 | Y | Y | Y | Y | C | C | C | Y | Y | Y | C | C | Y | Y | Y | Y | Y | ||||||||||
| Davis et al28 | Y | Y | Y | C | C | C | C | Y | Y | Y | Y | Y | Y | C | C | C | C | ||||||||||
| Devers et al29 | Y | Y | Y | Y | C | C | C | ||||||||||||||||||||
| Dolins et al2 | Y | Y | Y | Y | Y | C | Y | ||||||||||||||||||||
| Doyle et al30 | Y | Y | Y | Y | Y | C | C | ||||||||||||||||||||
| Dückers et al31 | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | C | Y | ||||||||||
| Dunbar et al32 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Fernald et al33 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Fieldston et al34 | Y | Y | Y | Y | Y | C | C | ||||||||||||||||||||
| Fok et al35 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Gaetke-Udager et al36 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Glasgow et al37 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Godfrey et al38 | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | |||||||||||||||
| Godfrey et al39 | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | |||||||||||||||
| Gustafson et al40 | Y | Y | Y | Y | Y | N | Y | ||||||||||||||||||||
| Hajjar-Nejad et al41 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Joly et al42 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Kamal et al43 | Y | Y | Y | Y | C | C | C | Y | Y | Y | C | Y | Y | Y | C | Y | C | ||||||||||
| Kaminski et al44 | Y | Y | Y | C | C | C | C | Y | Y | Y | Y | Y | Y | Y | C | C | C | ||||||||||
| Laing et al45 | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | C | Y | ||||||||||
| McLinden et al46 | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | C | C | Y | Y | Y | C | Y | ||||||||||
| McNamara et al47 | Y | Y | Y | C | C | C | C | Y | Y | Y | Y | Y | Y | C | C | C | C | ||||||||||
| Mold et al48 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Morganti et al49 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| New et al50 | Y | Y | Y | Y | Y | N | Y | ||||||||||||||||||||
| O'Connor et al51 | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | ||||||||||
| O'Leary52 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Peden et al53 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Rask et al54 | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | C | C | Y | Y | Y | C | Y | ||||||||||
| Riley et al5 | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | C | Y | ||||||||||
| Rinke et al55 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Robert et al56 | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | |||||||||||||||
| Rogers et al57 | Y | Y | Y | Y | C | C | C | Y | Y | Y | C | C | Y | Y | Y | C | Y | ||||||||||
| Sarin et al58 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Scott et al59 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Sellers et al60 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Sepulveda et al61 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Shaw et al62 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Tudiver et al63 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Vaughn et al64 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| White et al1 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
| Wong et al65 | Y | Y | Y | Y | Y | Y | Y | ||||||||||||||||||||
Y, yes; N, No; C, Cannot tell.
Data Extraction
Two reviewers completed the data extraction independently. A customized data extraction form was developed (see Supplemental File 4, Supplemental Digital Content 4, http://links.lww.com/JCEHP/A106). One reviewer compared the data extraction forms and discrepancies were resolved through discussion between reviewers.
RESULTS
The 53 included studies were published between 2009 and 2019 and set in 11 countries. Most studies (35/53) were based in the United States. Most Study designs were quantitative descriptive (17/53) followed by mixed methods studies (16/53). The population varied widely, ranging from frontline staff, clinical and nonclinical staff, and leaders. A summary of studies is presented in Supplemental File 5, Supplemental Digital Content 5, http://links.lww.com/JCEHP/A131.
Training Description
Less than half (14/53) of the studies were based on a collaborative approach. Duration of the collaboratives was variable, ranging from 2 months to 72 months. Half of the studies used a blended learning approach (25/53) combining face-to face and distance learning modes, whereas 21 studies relied solely on face-to-face learning modes. Four trainings were delivered online, whereas three studies did not state training modality used. Interventions included multiple training methods; the most common (39/53) one being face-to-face learning sessions. Other methods included teleconferencing (12/53), online modules (10/53), workshops (9/53), webinars (6/53), and emails (6/53).
Curriculum Description
The curricula were summarized into categories of basic, intermediate, and advanced data skills based on complexity of data concepts taught. Figure 2 summarizes the three categories and highlights the data concepts part of the training and curricula in the included studies. Basic data skills include concepts of measurement and QI knowledge, which are important for all health care staff. Intermediate data skills concepts are useful for staff working in improvement teams, whereas advanced data skills concepts are useful for improvement team leads and advisors.
FIGURE 2.
Measurement for Improvement Concepts. Summary of basic, intermediate, and advanced measurement and data skills taught in the QI programs.
Study Outcomes
Study outcomes are categorized as participant reaction, participant learning, participant behavior, sustainability, spread, and course design elements (see Supplemental File 6, Supplemental Digital Content 6, http://links.lww.com/JCEHP/A107). All studies measuring participation reaction to training and improved learning reported positive outcomes regardless of the study design. Studies measuring participant behavior also reported positive results except two quantitative descriptive studies.63,65 Two randomized control trial studies24,53 and a controlled interrupted time series study50 reported not achieving the clinical outcomes being measured. A cohort study also reported not achieving the outcome of developing a culture of QI.42 In spread, one quantitative descriptive study49 reported no spread of QI methods. For sustainability outcomes, Glasgow et al37 (Interrupted time series), Doyle et al30 (Quantitative descriptive), and Cranley et al26 (Mixed methods) reported a lack of sustainability of QI.
The Role of Measurement
The included studies reported on the role of measurement in QI for tracking progress toward goals and offering a systematic way to test changes to close performance gaps.20,61 Measurement was identified as a key competency for clinical QI21 to understand variation and improve the design health care.4 Measurement was used to view data over time and draw conclusions regarding variations.54 Measurement also played a role in implementation of QI methods19 and contributed to the success of QI.49
Knowledge about statistics and statistical process control4 and additional support for measurement skills was reported by studies as critical.2,33,39 Defining clear aims and measuring progress toward them was described as essential for QI.62 Measurement was used to demonstrate patient outcomes to the host organizations30 and provide guidance to decision makers.49 Continuous measurement followed the processes through the project period into daily operations.4 One study labelled measurement as one of the crucial elements of strategy for QI spread and sustainability.31 Timely data and measurement are important for assessing progress and evaluation.30
Challenges in identifying, collecting, and displaying appropriate measures of care impact QI program success.4,18,63 Studies cited measurement challenges such as difficulty in obtaining measurable data34,64 and presenting data in run/control chart formats.35 Many participating hospitals were not equipped for systematic data collection.31 Data collection and measurement was valued,29 but perceived to be time consuming by participants.52,53
Defining Effectiveness and Sustainability
The purpose of the review is to define effectiveness and sustainability and identify the barriers and enablers to success, in the context of QI programs with a focus on data and measurement for improvement. There is variability in how effectiveness and sustainability is defined in the studies. Another related concept that emerged was that of spread. It is therefore important to distinguish between effectiveness, spread, and sustainability. Effectiveness and sustainability definitions were extracted as part of the data extraction tool and summarized in Supplemental Digital Content 7 (see Supplemental file 7, http://links.lww.com/JCEHP/A108). The aspects addressed by these extracted definitions were then used to synthesize definitions. This was completed via consultation between three reviewers.
Effectiveness definitions focused on the short-term impact of the QI programs and were measured using participant reaction to the program, improved knowledge and skill application of participants, program participation and completion of QI projects by participants, and improvement in clinical outcomes at the end of the intervention period. Sustainability on the other hand, is defined not only as long-term outcomes beyond the intervention period, but also as a continuous process. Spread definitions focused on the diffusion of QI methods, processes, and skills from the intervention setting to nonintervention settings. We synthesized the following definitions of effectiveness, sustainability, and spread for measurement for improvement programs:
Effectiveness
Demonstrating improvement in key process, outcome, or quality measures being tracked, accompanied with an improvement in measurement knowledge, skills, and behaviors of learners during the intervention period.
Sustainability
Ongoing measurement, and development of processes and policies to maintain and improve the achieved gains in outcomes and participant skills and integration of measurement practices into routine after the intervention period, without further support from the trainers.
Spread
Active and passive diffusion of measurement skills and practices to areas and staff within and outside the organization that were not exposed to the training intervention.
Barriers and Enablers
A six-phased thematic analysis methodology (familiarization with data, initial coding, identifying themes, reviewing themes, naming themes, and reporting) was used to identify the barriers and enablers of sustainability66 (see Supplemental file 8, Supplemental Digital Content 8, http://links.lww.com/JCEHP/A109). The definitions of effectiveness and sustainability synthesized were used as reference. The coding process was done manually by one reviewer and final themes were discussed and agreed with two other reviewers.
Effectiveness
The four themes that emerged in enablers to effectiveness were intervention design, staff engagement, supportive leadership, and organizational support. Intervention design was the most important factor in the effectiveness of the program. Customizing training allows teaching of skills relevant to participant's role.3,31 Considering the implementation context3,39,52,53 and the challenges and opportunities of the setting33,43,62 leads to targeted skill building.19 A good starting point is to assess the prior knowledge and experience of participants21 to determine training needs31 and design a suitable range of resources57,65 corresponding to diversity of experiences and knowledge levels.64 Offering online modules44 and online resources53 also helps bridge this gap.
Intervention effectiveness can be enhanced using multiple learning strategies21 and evidence-based curricula.28 An effective intervention is responsive to participant learning styles65 and improves the training based on feedback.28,64 The best way to learn is by doing21 and incorporating experiential learning principles23 through demonstration projects29 and case studies65 builds capability. Another aspect of customized content is developing an interdisciplinary and team-based course46,60,65 as working in teams prevents participants from becoming overburdened with measurement.64 Having a participatory, data-driven approach contributes to effectiveness.19,53 Focusing on real-time data increases20 the program's value as participants can identify gaps in current processes.23,55 Teaching practical data gathering,21 statistical control charts,4 data analysis, and comparison contribute to effectiveness. Feedback from fellow participants allows them to learn from each other and adds to effectiveness.36,64 Similarly, informing participants about other team's progress creates healthy competition and prevents redundancy of efforts.64
Effective coaching also plays an important role. Customized coaching experience through just-in-time coaching28 and direct onsite, in-person support33 improves effectiveness. Coaching is more effective when trainers can respond directly to participant concerns.40 The ability of coaches to provide measurement support22 in creating data collection processes27 and data quality troubleshooting33 adds to effectiveness. Practice facilitation33 is also an enabler because providing private coaching between learning sessions,44 ongoing mentorship21 and tools and resources39 are valued by participants. Coaches can provide customized feedback and assistance.29 When participants perceive the training organization to be credible and have a sense of affiliation with it, they consider the training to be more effective.23
Guiding participants in indicator selection by focusing on establishing clear, realistic, mutually agreed,18,22 and clinically meaningful goals43,47,50 is a successful strategy. Encouraging participants to focus on simple solutions18,25 and making small changes26,62 leads to effectiveness. In-person workshops21 are an effective mode of training as face-to-face contact33 is preferred by participants. Using technology for designing easy to access, self-paced and self-initiated interventions57 improves effectiveness.
Successfully engaging health care staff is another important theme in effectiveness. Clinical staff feel empowered when they can identify and address gaps25 and select relevant QI topics.41,52 Providing dedicated time to participants to attend training sessions34,41,42,45,53 also adds to effectiveness. Demonstrating the value of competency in QI skills64 and offering maintenance of certification credit59 also helps in creating enthusiasm among staff. Supportive policies of the organization such as assuring time release recognizes the training as a valued activity.21 Leadership support is an important factor in the success of such programs.21
There are four themes in barriers to effectiveness: incompatible intervention design, lack of staff engagement, lack of organizational support, and lack of strategic approach. Fast pace of collaboratives18 and didactic instruction30 which did not correspond to learning needs of all participants, especially those in support roles3 were perceived as barriers. The number of concepts covered in the training made it difficult for participants to keep up and the terminology used was sometimes difficult to understand.29 A single day of classroom training was an insufficient dose50 and scheduling a full day training workshop is tiring for participants.63 When training programs that did not incorporate advice on implementation53 and leading change,36 it proved to be a barrier to effectiveness.
Lack of organizational support was visible in cases where participants were not provided protected time and struggled to attend the sessions.3,64 Poor data infrastructure impeded data collection31 and obtaining baseline measures.34,44 Lack of staff engagement and a negative perception about QI work and training because of previous negative experiences18,45 dampened effectiveness. Some programs failed to incorporate appropriate reward systems to motivate behavior36 and the lack of interest among participants resulted in low attendance35,50 and in some cases, staff disliked new tools and processes that required learning new methods.39 Staff struggled with learning measurement skills such as presenting data as run charts/control charts35,36 which decreased collaborative effectiveness. Some did not see any value in investing time in such collaboratives63 and believed the burdens outweighed the benefits.29,36 Another barrier was the lack of a strategic approach and the participants selected projects that were incompatible with the goals of their institutions.36,51
Sustainability
The themes observed in enablers to sustainability were taking a strategic approach, accountability, communication, learning networks, staff engagement, organizational support, intervention design, and supportive leadership. Taking a strategic approach requires connecting the program to organizational and national priorities,21 strategic goals,2,27 and teaming up with other departments64 and organizations53 with similar agendas.29 As organizations prioritize and implement QI,57 they move from sporadic efforts toward performance management systems,5 which sustains learning. Incorporating strategies to address psychology20 of change improves sustainability. Using a standard approach to QI ensures a common and clear improvement language.44
Another aspect of sustainability is to recruit the right people in the project team.4,22 A purposeful participant selection strategy46 ensures inclusion of individuals who are interested in improvement work. Scale-up plans18 with a goal of institution-wide diffusion31 add to sustainability. It is important to integrate QI into programs and services42,53 through updated job descriptions,33 building QI responsibility into operational responsibilities47 and continually reinforcing skills.49 Engaging all stakeholders from an early stage2,18,19,21,53 is also an enabler. In addition, while planning evaluations, it is important to assess learner involvement and QI project outcomes beyond completion of the programme.52 A strategic approach requires taking a system-level view20,21 of improvements with a blameless culture focusing on systems rather than individuals,24,53 which considers challenges as system issues rather than staff issues.28
Supportive organizational practices encourage QI by removing barriers,26 investing in workforce capacity and culture change28 and providing a conducive environment for teamwork.38,45 It also commits resources3,33,46,56 and provides opportunities to practice the skills learned.19,22,27 Accountability is an important enabler for sustainability. A clear definition of responsibilities,34 tasks39 and individual roles62 is key. Establishing time-bound targets20 and regular meetings to follow through on action62 ensures accountability. It is also beneficial to establish measurement guidelines to follow the process through the project period into daily operations.4 This continuous sharing of numbers leads53 to motivation and boosts sustainability.4 The training organization can also provide external accountability33 and ensure participants see projects to completion.41
Focus on capacity building also improves sustainability. This includes training staff for specialized QI roles such as QI champion,28 process coach,31 and QI advisor.33 A mentorship framework to support those interested in developing QI skills and encouraging permanent staff to develop coaching skills improves sustainability.47 Effective communication contributes to sustainability. Recognizing the efforts of QI teams26,34 by showcasing success stories28 through ongoing promotional activities56 is a rewarding strategy. Senior leader communication through board letters31 also supports sustainability. Formal and informal dissemination are vital to communication and sustainability. Formal dissemination can include internal dissemination,31 dissemination to local, national, and international audiences2 and toolkits.55 Informal dissemination can include enthusiastic employees53 and other informal contacts.31 Similarly, visual display of data and progress helps in disseminating the message of improvement.59
Learning networks are an important enabler.53 Learning from peers by sharing ideas18,21 and building relationships creates a strong learning community for idea exchange.33 These learning platforms serve as venues for knowledge transfer57 and repositories for QI.27 Development of collaborations between organizations leads to networking56 and solution sharing.33 Another area in staff engagement is generating awareness about QI18 beyond the project team2,26,42 and its impact on career.22 Extended support from coaches for implementation sequencing48 improves sustainability.
Support from leaders is crucial to sustainability.53 This involves improving leaders' QI skills so they can develop infrastructure for QI in their organizations28 such as establishing QI teams.53 Senior leadership support22 including board executives and chief of the medical staff provide legitimacy to QI.31 A strong leadership structure championing QI on a daily basis34 sends a message for sustainability. Leadership support allows staff to try new ideas in a safe environment that does not punish risk-taking.56 Organizational support plays a role through various strategies such as incentivizing diffusion48 and providing resources and autonomy to innovate.3
Themes in barriers to sustainability include lack of accountability, poor communication, lack of leadership support, lack of staff engagement, lack of organizational support, absence of learning networks, and not having a strategic approach. When timelines, roles, and responsibilities are not established, the plan of actions can evaporate leading to slippage in agreed timeframes and a loss of momentum.50 Because of poor institutional communication, staff lack a shared perception of problems51 and often lack institutional knowledge to approach the relevant individuals for QI work.60 Lack of leadership support manifests in the form of a lack of interest from top management4 and variations in the readiness of senior leaders to engage in QI.46
Learning networks play a vital role in sustainability; however, they are challenging to establish because few practices reach out to others to learn from them62 and may also face difficulty in learning from practices with dissimilar QI capacity and patients.29 Lack of organizational support is a major barrier to sustainability53 because it represents a culture that is not conducive to making or sustaining change.33 Presence of administrative red tape3,64 can inhibit innovation and indicate that QI is not a priority for the organization.55
Poor data infrastructure,4,33,53 data quality, and access to data4 decrease sustainability. Repeated data collection can be cumbersome and labor intensive in the long run.52 Lack of resource availability3,26,46 for QI projects is another barrier. Programs that lack ongoing organizational support are likely to be unsuccessful.65 Health care staff have competing demands on their time,46,62,63,63 which interferes with team's ability to meet and work.3 Because QI teams are a disparate group of staff,50 a lack of dedicated time for QI work57 can be a barrier. Failing to engage staff, and leadership effectively and not focusing on motivation and behavior change can be a barrier to sustainability.53 It is also important to account for the high levels of stress and emotional demands experienced by front-line staff.1
DISCUSSION
The purpose of the systematic review is to define effectiveness and sustainability of QI programs with a significant data skills component and to identify the relevant barriers and enablers. Fifty-three studies were included in the review. There was heterogeneity in the content, teaching methods, and program design in the included studies and variability in the way effectiveness, sustainability, and spread were defined and measured in the context of QI programs. The review also highlighted variation in the ability of the programs to achieve desired outcomes. These inconsistencies in program success were attributed to various barriers and enablers to effectiveness and sustainability.
The lack of staff engagement, lack of a strategic approach, and lack of organizational support are barriers common between effectiveness and sustainability, which implies that these factors have implications for the short-term and long-term success of the programs. Poor intervention design affects the effectiveness of the program while poor communication, lack of accountability, and lack of leadership support can plague the ability to sustain the skills and results in the long-term. In enablers, intervention design, supportive leadership, engaged staff, and organizational support can affect positively on both effectiveness and sustainability of programs. Enablers that are relevant to sustainability are learning networks, communication, accountability, and a strategic approach to QI. The barriers and enablers highlight the importance of organizational,39 learner, teacher, curricular,35 and contextual factors3 in the success of QI programs.
The definitions derived for effectiveness and sustainability highlight the importance of measurement. Studies reported measurement as a key competency for clinical QI.21 Continuous measuring and remeasuring play an important role in maintaining62 and operationalizing improvements in the long run. Selecting appropriate measures,18,22 data collection34 and using charts to display data35 are essential to show effective change.64 QI programs therefore need to focus on training staff in QI methods and how to measure care and use data to drive change.62 There is an increasing expectation from health care professionals to measure, report, and continually improve the quality of care.62 This indicates the need for a cultural shift from traditional academic-focused programs toward programs focusing on measurement and results to develop the capability of health professionals in leading improvement.44
The findings of this systematic review also advocate for program evaluation to consider impact on participant behavior, patient outcomes, and supporting downstream learning beyond the direct participants of the programme.43 Instead of solely relying on measuring quantitative outcomes, evaluators should also use qualitative data to assess whether program outcomes are achieved.29
Measurement emerged as a critical element of QI training programs, which enables health care professionals and organizations to demonstrate effectiveness of improvement efforts and sustain improvements in the long run. Training health care professionals in data skills can have implications for improving health systems. However, health care systems are complex and various actors such as the health care authorities, training organizations, trainers, trainees, and trainee's organization have a collaborative role to play in ensuring effectiveness and sustainability of QI programs. Outputs of the thematic analysis in the form of effectiveness and sustainability barriers and enablers were broken down into inputs, outputs, and short- and long-term outcomes, which were then mapped onto a logic model. This was completed via consultation between three reviewers and presented in Figure 3.
FIGURE 3.
Intervention success factors. Summary of QI program inputs contributing to achievement of short-term and long-term outcomes.
Limitations
A limitation of this review is that there were no stand-alone measurement for improvement training studies. The reviewers overcame this by establishing the presence of measurement component in the QI programs as an inclusion criterion. Because no search strategy is perfect, there is a risk of missing relevant studies; however, we mitigated this risk using a search strategy focused on sensitivity and iteratively testing the search strategy in selected databases.
CONCLUSION
The review highlighted that measuring the improvement in outcomes and participant knowledge establishes effectiveness while remeasuring continuously helps in sustaining outcomes in the long-term for QI programs with a significant measurement skills component. The review identified staff engagement, strategic approach to QI, organizational support, intervention design, communication, accountability, leadership support, and learning networks as factors that affect effectiveness and sustainability. The review expands current knowledge about the importance of measurement in QI training programs. Ensuring effectiveness and sustainability of measurement for improvement programs requires a collective effort from trainers, trainees, the organizations in which the interventions are implemented and policy makers.
Lessons for Practice
■ Measurement has a central role in demonstrating improvements and maintaining desired improvement outcomes of QI programs in the short- and long-term.
■ Staff engagement, strategic approach to QI, organizational support, intervention design, communication, accountability, leadership support, and learning networks influence effectiveness and sustainability of QI programs.
■ Effectiveness, sustainability, and spread of QI programs with a measurement component requires a collective effort from trainers, trainees, the organizations in which the interventions are implemented, and policy makers.
Supplementary Material
Footnotes
The corresponding author receives a PhD funding from the Health Service Executive Ireland (Project reference 57399). The study is also supported by the Irish Health Research Board (RL-2015-1588).
Disclosures: The authors declare no conflict of interest.
Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal's Web site (www.jcehp.org).
Contributor Information
Aoife De Brún, Email: aoife.debrun@ucd.ie.
Jennifer Martin, Email: jennifer.martin@hse.ie.
Eilish McAuliffe, Email: eilish.mcauliffe@ucd.ie.
REFERENCES
- 1.White M, Wells JSG, Butterworth T. The impact of a large-scale quality improvement programme on work engagement: preliminary results from a national cross-sectional-survey of the Productive Ward. Int J Nurs Stud. 2014;51:1634–1643. [DOI] [PubMed] [Google Scholar]
- 2.Dolins JC, Powell J, Wise E, et al. Improving asthma care by building statewide quality improvement infrastructure. Pediatrics. 2017;140:e20161612. [DOI] [PubMed] [Google Scholar]
- 3.Butler A, Canamucio A, Macpherson D, et al. Primary care staff perspectives on a virtual learning collaborative to support medical home implementation. J Gen Intern Med. 2014;29:579–588. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Brandrud AS, Schreiner A, Hjortdahl P, et al. Three success factors for continual improvement in healthcare: an analysis of the reports of improvement team members. BMJ Qual Saf. 2011;20:251–259. [DOI] [PubMed] [Google Scholar]
- 5.Riley W, Parsons H, McCoy K, et al. Introducing quality improvement methods into local public health departments: structured evaluation of a statewide pilot project. Health Serv Res. 2009;44:1863–1879. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Hughes RG. Chapter 44: tools and strategies for quality improvement and patient safety. In: Hughes RG, ed. Patient Safety and Quality: An Evidence-Based Handbook for Nurses .Rockville, MD: Agency for Healthcare Research and Quality; 2008. [PubMed] [Google Scholar]
- 7.Mormer E, Stevans J. Clinical quality improvement and quality improvement research. Perspect ASHA Spec Interest Groups. 2019;4:27–37. [Google Scholar]
- 8.Margolius D, Ranji SR. Quality improvement, quality measurement and medical education: a brewing culture clash? BMJ Qual Saf. 2015;24:477–479. [DOI] [PubMed] [Google Scholar]
- 9.Boonyasai RT, Windish DM, Chakraborti C, et al. Effectiveness of teaching quality improvement to CliniciansA systematic review. JAMA. 2007;298:1023–1037. [DOI] [PubMed] [Google Scholar]
- 10.Starr SR, Kautz JM, Sorita A, et al. Quality improvement education for health professionals:A systematic review. Am J Med Qual. 2016;31:209–216. [DOI] [PubMed] [Google Scholar]
- 11.Peiris-John R, Selak V, Robb G, et al. The state of quality improvement teaching in medical schools: a systematic review. J Surg Educ. 2020;77:889–904. [DOI] [PubMed] [Google Scholar]
- 12.Nayak AK. Efficiency, effectiveness and sustainability: the basis of competition and cooperation. Paper presented at: XIMB Sustainability Seminar Series, Working Paper 2011.
- 13.Pimental S. Acquiring evidence-tips for effective literature searching. Permanente J. 2005;9:58–60. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Covidence Systematic Review Software [computer program]. Melbourne, Australia: Veritas Health Innovation. [Google Scholar]
- 15.Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLOS Med. 2009;6:e1000097. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Hong QN, Fàbregues S, Bartlett G, et al. The Mixed Methods Appraisal Tool (MMAT) Version 2018 for Information Professionals and Researchers. Educ Inform. 2018;34:1–7. [Google Scholar]
- 17.Souto RQ, Khanassov V, Hong QN, et al. Systematic mixed studies reviews: updating results on the reliability and efficiency of the mixed methods appraisal tool. Int J Nurs Stud. 2015;52:500–501. [DOI] [PubMed] [Google Scholar]
- 18.Adams D, Hine V, Bucior H, et al. Quality improvement collaborative: a novel approach to improve infection prevention and control. Perceptions of lead infection prevention nurses who participated. J Infect Prev. 2018;19:64–71. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Bardfield J, Agins B, Akiyama M, et al. A quality improvement approach to capacity building in low- and middle-income countries. AIDS. 2015;29:S179–S186. [DOI] [PubMed] [Google Scholar]
- 20.Barker P, Barron P, Bhardwaj S, et al. The role of quality improvement in achieving effective large-scale prevention of mother-to-child transmission of HIV in South Africa. AIDS. 2015;29:S137–S143. [DOI] [PubMed] [Google Scholar]
- 21.Berry L. Report of an Evaluation of the Clinical Quality Improvement Programme (CQIP). Saskatchewan, Canada: Health Quality Council; 2018. [Google Scholar]
- 22.Bidassie B, Davies ML, Stark R, et al. VA experience in implementing patient-centered medical home using a breakthrough series collaborative. J Gen Intern Med. 2014;29:563–571. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Bundy DG, Morawski LF, Lazorick S, et al. Education in quality improvement for pediatric practice: an online program to teach clinicians QI. Acad Pediatr. 2014;14:517–525. [DOI] [PubMed] [Google Scholar]
- 24.Chinthammit C, Rupp MT, Armstrong EP, et al. Evaluation of a guided continuous quality improvement program in community pharmacies. J Pharm Pol Pract. 2017;10:26. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Cosimi LA, Dam HV, Nguyen TQ, et al. Integrated clinical and quality improvement coaching in Son La Province, Vietnam: a model of building public sector capacity for sustainable HIV care delivery. BMC Health Serv Res. 2015;15:269. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Cranley LA, Hoben M, Yeung J, et al. SCOPEOUT: sustainability and spread of quality improvement activities in long-term care- a mixed methods approach. BMC Health Serv Res. 2018;18:174. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Davis MV, Vincus A, Eggers M, et al. Effectiveness of public health quality improvement training approaches: application, application, application. J Public Health Manag Pract. 2012;18:E1–E7. [DOI] [PubMed] [Google Scholar]
- 28.Davis MV, Cornett A, Mahanna E, et al. Advancing quality improvement in public health departments through a statewide training program. J Public Health Manag Pract. 2016;22:E21–E27. [DOI] [PubMed] [Google Scholar]
- 29.Devers KJ, Foster L, Brach C. Nine states' use of collaboratives to improve children's health care quality in medicaid and CHIP. Acad Pediatr. 2013;13(6 supplement):S95–S102. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Doyle C, Howe C, Woodcock T, et al. Making change last: applying the NHS institute for innovation and improvement sustainability model to healthcare improvement. Implement Sci. 2013;8:127. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Dückers ML, Wagner C, Vos L, et al. Understanding organisational development, sustainability, and diffusion of innovations within hospitals participating in a multilevel quality collaborative. Implement Sci. 2011;6:18. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Dunbar AE, Cupit M, Vath RJ, et al. An improvement approach to integrate teaching teams in the reporting of safety events. Pediatrics. 2017;139:e20153807. [DOI] [PubMed] [Google Scholar]
- 33.Fernald D, Wearner R, Dickinson WP. Supporting primary care practices in building capacity to use health information data. EGEMS (Wash DC). 2014;2:1094. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Fieldston ES, Jonas JA, Lederman VA, et al. Developing the capacity for rapid-cycle improvement at a large freestanding children's hospital. Hosp Pediatr. 2016;6:441–448. [DOI] [PubMed] [Google Scholar]
- 35.Fok MC, Wong RY. Impact of a competency based curriculum on quality improvement among internal medicine residents. BMC Med Educ. 2014;14:252. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Gaetke-Udager K, Brown RKJ, Centonze CP, et al. Advanced quality training in radiology: inaugural report of a 2-year program. Am J Roentgenol. 2019;212:1082–1090. [DOI] [PubMed] [Google Scholar]
- 37.Glasgow JM, Davies ML, Kaboli PJ. Findings from a national improvement collaborative: are improvements sustained? BMJ Qual Saf. 2012;21:663–669. [DOI] [PubMed] [Google Scholar]
- 38.Godfrey MM, Thor J, Nilsson M, et al. Testing a Team Coaching Model to develop improvement capability of front line teams: a comparative intervention and process evaluation pilot study. Jönköping: Quality Improvement and Leadership in Health and Welfare. Jönköping, Sweden: Jönköping University, School of Health Science, HHJ; 2013. [Google Scholar]
- 39.Godfrey MM, Andersson-Gare B, Nelson EC, et al. Coaching interprofessional health care improvement teams: the coachee, the coach and the leader perspectives. J Nurs Manag. 2014;22:452–464. [DOI] [PubMed] [Google Scholar]
- 40.Gustafson DH, Quanbeck AR, Robinson JM, et al. Which elements of improvement collaboratives are most effective? A cluster-randomized trial. Addiction. 2013;108:1145–1157. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Hajjar-Nejad MJ, Kubicki N, Morales D, et al. Multilevel quality improvement teams: an alternative approach for surgical academic training programs to meet ACGME core competency milestones. J Surg Educ. 2019;76:785–794. [DOI] [PubMed] [Google Scholar]
- 42.Joly BM, Booth M, Shaler G, et al. Assessing quality improvement in local health departments: results from the multi-state learning collaborative. J Public Health Manag Pract. 2012;18:79–86. [DOI] [PubMed] [Google Scholar]
- 43.Kamal AH, Quinn D, Gilligan TD, et al. ReCAP: feasibility and effectiveness of a pilot program to facilitate quality improvement learning in oncology: experience of the American society of clinical oncology quality training program. J Oncol Pract. 2016;12:177.e215–123. [DOI] [PubMed] [Google Scholar]
- 44.Kaminski GM, Britto MT, Schoettker PJ, et al. Developing capable quality improvement leaders. BMJ Qual Saf. 2012;21:903–911. [DOI] [PubMed] [Google Scholar]
- 45.Laing BY, Dixit RK, Berry SH, et al. A quasi-experimental evaluation of performance improvement teams in the safety-net: a labor-management partnership model for engaging frontline staff. J Public Health Manag Pract. 2016;22:E1–E7. [DOI] [PubMed] [Google Scholar]
- 46.McLinden DJ, Charns M. To Achieve the Best: Evaluating Quality Improvement Training as a Means to an End, Cincinnati Children's Hospital Medical Center. New Jersey: Robert Wood Johnson Foundation; 2012. [Google Scholar]
- 47.McNamara DA, Rafferty P, Fitzpatrick F. An improvement model to optimise hospital interdisciplinary learning. Int J Health Care Qual Assur. 2016;29:550–558. [DOI] [PubMed] [Google Scholar]
- 48.Mold JW, Aspy CB, Smith PD, et al. Leveraging practice-based research networks to accelerate implementation and diffusion of chronic kidney disease guidelines in primary care practices: a prospective cohort study. Implement Sci. 2014;9:169. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Morganti KG, Lovejoy S, Beckjord EB, et al. A retrospective evaluation of the perfecting patient care university training program for health care organizations. Am J Med Qual. 2014;29:30–38. [DOI] [PubMed] [Google Scholar]
- 50.New S, Hadi M, Pickering S, et al. Lean participative process improvement: outcomes and obstacles in trauma orthopaedics. PLoS One. 2016;11:e0152360. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.O'Connor ES, Mahvi DM, Foley EF, et al. Developing a practice-based learning and improvement curriculum for an academic general surgery residency. J Am Coll Surg. 2010;210:411–417. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.O'Leary KJ, Fant AL, Thurk J, et al. Immediate and long-term effects of a team-based quality improvement training programme. BMJ Qual Saf. 2019;28:366–373. [DOI] [PubMed] [Google Scholar]
- 53.Peden CJ, Stephens T, Martin G, et al. Health services and delivery research. In: A National Quality Improvement Programme to Improve Survival after Emergency Abdominal Surgery: The EPOCH Stepped-Wedge Cluster RCT. Southampton, United Kingdom: NIHR Journals Library; 2019. [PubMed] [Google Scholar]
- 54.Rask KJ, Blake SC, Kohler SS, et al. Evaluation of a Two-Pronged Training Program to Build Capacity for Quality. Atlanta, GA: Emory University, Rollins School of Public Health; 2011. [Google Scholar]
- 55.Rinke ML, Driscoll A, Mikat-Stevens N, et al. A quality improvement collaborative to improve pediatric primary care genetic services. Pediatrics. 2016;137:e20143874. [DOI] [PubMed] [Google Scholar]
- 56.Robert G, Morrow E, Maben J, et al. The adoption, local implementation and assimilation into routine nursing practice of a national quality improvement programme: the Productive Ward in England. J Clin Nurs. 2011;20:1196–1207. [DOI] [PubMed] [Google Scholar]
- 57.Rogers SJ, Hamdalla M, Little S. Evaluation of Training/Capacity Building. National Quality Center. New Jersey: Robert Wood Johnson Foundation; 2012. [Google Scholar]
- 58.Sarin E, Kole SK, Patel R, et al. Evaluation of a quality improvement intervention for obstetric and neonatal care in selected public health facilities across six states of India. BMC Pregnancy Childbirth. 2017;17:134. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Scott E, Downs S, Pottenger A, et al. Quality improvement learning collaborative improves timely newborn follow-up appointments. Joint Comm J Qual Patient Saf. 2019;45:808–813. [DOI] [PubMed] [Google Scholar]
- 60.Sellers MM, Hanson K, Schuller M, et al. Development and participant assessment of a practical quality improvement educational initiative for surgical residents. J Am Coll Surgeons. 2013;216:1207–+. [DOI] [PubMed] [Google Scholar]
- 61.Sepulveda D, Varaklis K. Implementing a multifaceted quality improvement curriculum in an obstetrics-gynecology resident continuity clinic setting: a 4 year experience. J Graduate Med Educ. 2012;4:237–241. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 62.Shaw JS, Norlin C, Gillespie RJ, et al. The national improvement partnership network: state-based partnerships that improve primary care quality. Acad Pediatr. 2013;13:S84–S94. [DOI] [PubMed] [Google Scholar]
- 63.Tudiver F, Click IA, Ward P, et al. Evaluation of a quality improvement curriculum for family medicine residents. Fam Med. 2013;45:19–25. [PubMed] [Google Scholar]
- 64.Vaughn NH, Hassenbein SE, Black KP, et al. Important elements in the quality improvement curriculum for orthopaedic residents. J Bone Joint Surg Am. 2019;101:e28. [DOI] [PubMed] [Google Scholar]
- 65.Wong BM, Goguen J, Shojania KG. Building capacity for quality: a pilot co-learning curriculum in quality improvement for faculty and resident learners. J Grad Med Educ. 2013;5:689–693. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 66.Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3:2006. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.