Introducing Quality Improvement Methods into Local Public Health Departments: Structured Evaluation of a Statewide Pilot Project

Abstract

Objective

To test the feasibility and assess the preliminary impact of a unique statewide quality improvement (QI) training program designed for public health departments.

Data Sources/Study Setting

One hundred and ninety-five public health employees/managers from 38 local health departments throughout Minnesota were selected to participate in a newly developed QI training program and 65 of those engaged in and completed eight expert-supported QI projects over a period of 10 months from June 2007 through March 2008.

Study Design

As part of the Minnesota Quality Improvement Initiative, a structured distance education QI training program was designed and deployed in a first large-scale pilot. To evaluate the preliminary impact of the program, a mixed-method evaluation design was used based on four dimensions: learner reaction, knowledge, intention to apply, and preliminary outcomes.

Data

Subjective ratings of three dimensions of training quality were collected from participants after each of the scheduled learning sessions. Pre- and post-QI project surveys were administered to collect participant reactions, knowledge, future intention to apply learning, and perceived outcomes. Monthly and final QI project reports were collected to further inform success and preliminary outcomes of the projects.

Principal Findings

The participants reported (1) high levels of satisfaction with the training sessions, (2) increased perception of the relevance of the QI techniques, (3) increased perceived knowledge of all specific QI methods and techniques, (4) increased confidence in applying QI techniques on future projects, (5) increased intention to apply techniques on future QI projects, and (6) high perceived success of, and satisfaction with, the projects. Finally, preliminary outcomes data show moderate to large improvements in quality and/or efficiency for six out of eight projects.

Conclusions

QI methods and techniques can be successfully implemented in local public health agencies on a statewide basis using the collaborative model through distance training and expert facilitation. This unique training can improve both core and support processes and lead to favorable staff reactions, increased knowledge, and improved health outcomes. The program can be further improved and deployed and holds great promise to facilitate the successful dissemination of proven QI methods throughout local public health departments.

Introducing quality improvement (QI) methods and techniques into local public health departments is a major strategy currently underway to enhance agency performance, prepare agencies for voluntary accreditation, and ultimately improve population health. However, there is very little empirical evidence on how to enhance workforce capacity through QI training and on QI efficacy in public health departments. Moreover, these techniques are noticeably absent from the long list of recommended conceptual foundations of public health practice-based research (ASPH Council of Public Health Practice Coordinators 2005). A fundamental question in public health systems research is how to promote evidence-based practice and accelerate the use of research findings in a timely way (Shortell 2006; Brownson 2008;). Few areas in bridging the gap between discovery and application are as relevant as the current efforts to diffuse QI into public health practice (McBride et al. 2008). In this paper, we report on the preliminary findings of a structured evaluation of a pilot statewide QI program delivered to local health departments through distance education and facilitation in the State of Minnesota.

BACKGROUND

It has been widely recognized that better methods are needed to improve public health process performance. Studies have found substantial gaps in the performance of public health providers (Mays et al. 2004). Several major initiatives have been undertaken over the past decade to introduce QI techniques into public health, including the National Public Health Performance Standards Program (Centers for Disease Control and Prevention 2008), the Turning Point initiative (Hassmiller 2002), and the activities sponsored by the Public Health Foundation (see http://www.phf.org). More recently, the partnership between the American Public Health Association, National Association of Local Boards of Health, the Association of State and Territorial Health Officials, and the National Association of County and City Health Officials to create the Exploring Accreditation Project aspired to develop a national accreditation system that is specifically intended to promote high performance through QI (Bender et al. 2007; Public Health Accreditation Board 2007;). Although the use of QI is gradually being introduced into public health (Mays and Halverson 2006; Seid et al. 2006;), the context for QI is still not well defined (Leonard 2007) and efforts to carry out QI in public health may be overestimated (Baker et al. 2007). Indeed, most QI projects undertaken in organizations fail to meet their intended goal (Leonard 2007) and very few organizations have succeeded in achieving transformational change with sustained improvement (VanDeusen Lukas et al. 2007). While numerous barriers have been identified (Riley and Nwoke 2007), there is limited evidence on public health organizations' readiness for change and ability to improve outcomes.

The Minnesota Quality Improvement Collaborative (MQIC) is an ongoing QI initiative funded by the Robert Wood Johnson Foundation through the Multi-State Learning Collaborative-2 (MLC-2) and Changes in Health Care Financing and Organization's programs. MQIC is a partnership between the Minnesota Department of Health (MDH), the School of Public Health at the University of Minnesota (UM), and the Minnesota Local Public Health Association (LPHA). Its mission is to provide resources, tools, technical assistance, and training on QI techniques to local public health departments and to rapidly build and deploy QI knowledge and practice through innovative and cost-effective programs.

QI TRAINING PROGRAM

In a pilot study, all 86 local health departments and tribal governments in the State of Minnesota were invited in April 2007 to submit a proposal to (a) join the MQIC, (b) train a team from their organization, and (c) conduct a QI project focused in the local agency. While projects focusing on core processes were preferred, projects focusing on support processes were also considered. Applicants were permitted to propose their own topic. Specific review criteria were established: problem statement, project management, level of potential impact, capacity for replication, achievement within 12 months, and overall feasibility. After a systematic review by an expert panel comprised of expert UM faculty members and QI experts from MDH and LPHA, a total of 33 local health departments and one tribal government were selected to participate in eight different QI projects and the MQIC program. Four of the projects were implemented by a single agency and four projects involved multiple agencies. Most of the participants (local public health staff, state public health staff, and other key community stakeholders) were new to QI methods and techniques.

In a recent study of the effectiveness of QI collaboratives in health care organizations, Schouten et al. (2008) noted the value of the collaborative approach but stressed the need for more studies. While several collaborative approaches exist, the MQIC approach is based on the Institute for Healthcare Improvement (IHI) Breakthrough Series model (IHI 2003), a framework for rapid, cooperative learning drawing on the expertise of outside experts. The MQIC program closely follows the IHI Breakthrough Series model and is depicted in Figure 1, which also shows the timing of the pilot program as well as the timing and type of data collection performed to evaluate the pilot.1

Figure 1.

The MQIC Pilot Program (Adapted from the Institute for Healthcare Improvement 2003)

Core elements of the collaborative include (1) a comprehensive didactic training in the methods and techniques of QI with specific public health application (delivered to a total of 195 participants), and (2) experiential learning (for a total of 65 team members) through the guided development, design, implementation, and successful completion of a QI project. There are three phases in the MQIC.

In the first phase, expert UM faculty members and expert MDH staff organized and delivered three 1 day-long learning sessions (July, August, and September 2007) to a total of 195 health departments participants using interactive videoconferencing covering QI foundational methods such as the Model for Improvement, the Plan-Do-Study-Act process, and essential tools such as cause-and-effect analysis, process mapping, and radar charts. Periodic conference calls with technical facilitation were conducted with all as the eight collaboratives began to setup the design of their respective projects. Between the learning sessions, the project teams engaged in “action periods” (see Figure 1) and were given specific assignments to further their work. By the end of the third learning session, each team had established the focus of their project, created measures, and developed a plan for implementing tests of change.

In the second phase (September 2007–March 2008), each collaborative extensively studied their process using various QI techniques and skills acquired during phase 1. Project teams came together on monthly conference calls with facilitation from the MQIC experts to share challenges and accomplishments. Each conference call also included a short training component focused on a new QI tool such as survey design and implementation and control charts. During phase 2, each QI project benefited from the formal active involvement of one UM graduate student formally trained to QI as well as one public health nurse consultant with QI expertise. Monthly reports were submitted by each project to document their tests of change, progress, and intermediary results. Expert MDH staff and UM faculty reviewed the reports and provided constructive feedback to project teams. As shown in Figure 1, ongoing technical support was available to each of the projects in the form of UM faculty advisors and MDH expert consultants.

The third and final phase, still in process and not reported here, consists of locking in the gains and proceeding to repetition and saturation to help each local health department move toward institutional change (VanDeusen Lukas et al. 2007).

EVALUATION METHODS

A mixed-method design and corresponding data collection were used to perform a structured evaluation of both the training component (learning sessions) and the experiential learning project (the QI project itself) of the pilot program. The study protocol was approved by the Institutional Review Board of the University of Minnesota. Three data collection methods were used: (1) a self-administered survey was given to all 195 training participants at the end of each of the three learning sessions; (2) an online pre- and postproject self-administered survey was given to all 65 QI project team members; and (3) monthly and final project reports included a standardized section to be filled out by each of the eight QI teams to capture structured process information, project milestones, and preliminary project outcome data. The timing of the data collection methods is shown in Figure 1.

Measurement

We used the four-level model developed by Kirkpatrick and Kirkpatrick (2006) to evaluate the effects of the program. The four levels are (1) reaction, (2) learning, (3) behavior, and (4) outcomes. The first level of evaluation, reaction, measures how those who participate in the program react to the training (Kirkpatrick and Kirkpatrick 2006). With the learning session questionnaires, we measured participant reactions by assessing length of the training session, pedagogical format, and satisfaction with training format. Reaction was also measured with the pre- and postproject questionnaire through four items assessing team member perception of relevance, perceived management's support for the project, perceived availability of resources, and overall experience/satisfaction with the program (postproject only).

The second level of evaluation, learning, measures the extent to which participants change attitudes, improve knowledge, or increase skill as a result of attending the program (Kirkpatrick and Kirkpatrick 2006). In the postproject questionnaire, team members were asked to rate (1) the perceived usefulness of 10 specific QI techniques that were presented, practiced, and applied during the training sessions and/or project implementation, and (2) the perceived future applicability of the 10 specific QI techniques.

The third level of evaluation, behavior, measures the extent to which change in behavior has occurred or will occur because the participant attended the training program (Kirkpatrick and Kirkpatrick 2006). In this pilot study, we could only measure intended behavior and did so in three different ways. In the pre- and postproject questionnaires, QI team members were asked to rate (1) the extent to which they would feel qualified to conduct a QI project and (2) whether they will conduct/participate in a QI project in the future. Third, in the postproject questionnaire, QI team members were asked to rate the extent to which they would apply/use each of the 10 QI techniques they learned.

The fourth level of evaluation, outcomes, measures the extent to which outcomes are attributable to the program. As mentioned earlier, in this pilot study, project milestones and preliminary project outcome data were collected from the monthly and final QI project reports. Outcome metrics varied and included increased production, increased quality, decreased costs or other output measures, depending on the nature of each project. In addition, in the postproject questionnaire, QI team members were asked to rate (1) the extent to which they perceived the project costs exceeding its benefits and (2) the perceived level of success of the QI project.

RESULTS

In total, over 230 individuals from the Minnesota Department of Health participated in some or all aspects of the MQIC pilot program. In all, 34 local public health agencies were trained in 10 specific QI methods and techniques. A total of 195 individuals (staff and managers) participated in at least one of the three learning sessions, as the sessions were not restricted to individuals selected to participate in the eight collaboratives. In addition, a total of 81 QI team members participated in one of eight QI projects. Of those, 65 were individuals who also participated in the QI learning sessions. The remaining 16, as mentioned earlier, consisted of 8 trained UM graduate students and 8 MDH public health nurse consultants with expertise in QI, all assigned as on-going technical support participant to the QI projects.

Table 1 provides details for the eight QI projects selected for the pilot program and lists, for each project, the aim, the location (metro/suburban/rural) and the number of local health departments involved, the number of individuals on the team, the type of process addressed, the QI techniques used, and the preliminary outcomes achieved.

Table 1.

Description and Preliminary Outcomes of Eight QI Projects

Project Title	Aim	Location (No. of Counties Involved)	Team Size	Process Type	QI Techniques Used	Preliminary Outcomes	Perceived Overall Success
Decreasing clinic “no show” rates	By February 2008, increase the rate of appointments kept at Women, Infant and Children (WIC) clinics	Metro/Rural (9)	13	Core	1, 2, 3, 6, 9, 10,12	While some counties implemented a reminder protocol and maintained fairly high rates of kept appointments, other counties experienced initial increase but those increases were not maintained. Overall, improvement in rates was not demonstrated	Low
Improving reaction time necessary to respond to and deploy health alerts	By March 2008, decrease by 50% the staff time required to respond and deploy a health alert through the Health Alert Network (HAN)	Suburban (1)	10	Support	1, 2, 3, 5, 6, 9, 10, 12	Several interventions were designed and implemented, resulting in a 70% net decrease in staff time dedicated to HAN testing events	High
Improving communication in latent TB clinics	Reduce time spent documenting treatment of patients with latent tuberculosis (TB) by 10% or 8 minutes per patient	Metro/Rural (3)	9	Core	1, 2, 3, 6, 9, 10	Preliminary assessment of the program shows a reduction in documentation time by 9.7 minutes per patient. Additional testing is under way	High
Improving access to children mental health services	Establish program to offer mental health screening to 100% children with mental health needs	Rural (1)	9	Core	1, 2, 3, 6, 9, 10, 12	From 11/28/07 to 2/23/08 all 254 parents (100%) of children ages 6 months to 16 years were offered screening, 66.5% completed the screening, and 27% of those were diagnosed and referred to treatment	High
Improving immunization rates through data integration	By March 2008, improve immunization rates for children enrolled in the Women, Infant, and Children (WIC) Program and born 11/1/05 to 2/28/06	Metro/Rural (11)	9	Support	1, 2, 3, 9, 10, 12	Preliminary outcomes show that immunization rates increased significantly from 75% to 83%. Routine medical exams also increased	Medium
Overcoming barriers to dental varnishing	Increase use of dental fluoride treatments for children enrolled in the Women, Infant, and Children (WIC) Program. Between 09/01 and 11/30/07, have at least 20 WIC children treated	Metro/Rural/Tribal (7)	11	Core	1, 2, 3, 9, 10, 11, 12	A new program was designed and established. Between 09/01 and 11/30/07, 104 children treated, largely exceeding initial aim	High
Incorporating public health (PH) competencies into staff performance assessments	Develop standardized measures of public health competency and ensure that 100% of leadership council members achieve a minimum rating of 3 on a 1–5 scale	Metro (1)	12	Support	1, 2, 3, 7, 9, 10, 12, 13	Understanding of public health core competencies and their use in staff development and appraisals increased from 28% to 62%. Aim was not achieved but is well underway	Low
Preventing gaps in personal care assistance (PCA)	Increase the percentage of PCA reassessments from 62% to 80%	Metro/Rural (1)	8	Core	1, 2, 3, 6, 9, 10, 12	The initial goal was exceeded. An initial review shows that 100% of all assessments between 10/1 and 12/31/2007 were on time. Team awaits independent review by the MN Department of Human Services.	High

Key: 1, Model for Improvement; 2, PDSA Cycle; 3, Process Mapping; 4, Control Chart; 5, Run Chart; 6, Cause and Effect Diagram; 7, Memory Jogger; 8, Radar Chart; 9, Aim Specification; 10, Measure Development; 11, Force Field Analysis; 12, Nominal Group Techniques; 13, Logic Model.

Table 2 displays participants' immediate reactions to the three QI learning sessions. Overall, 73 percent of participants were satisfied with the length of the training. However, 23 percent indicated that the training sessions were too long, while 4 percent indicated that the training sessions were too short. Secondly, 92 percent of the participants were satisfied with the videoconferencing format, while 8 percent were dissatisfied with the approach. Finally, 55 percent of the participants were satisfied with the length of time allocated for applied teamwork exercises, while 24 percent felt that too little time was spent and 21 percent felt that too much time was spent.

Table 2.

Learning Sessions Assessment Results

	Session 1	Session 2	Session 3	Overall
Participants (N)	123	107	85
Satisfied with length of training (%)
Yes, just right	78	65	75	73
Too long	15	31	23	23
Too short	7%	4	2	4
Satisfied with video conference format (%)
Yes, worked	94	87	96	92
No	6	13	4	8
Satisfied with the length of time allocated for team work
Yes, just right	N/A	45	65	55
Too long	N/A	10	32	21
Too short	N/A	45	3	24

N/A, not applicable (no team work conducted as part of Learning Session 1).

Table 3 shows the pre- and post-QI project survey results. Forty-six individuals (out of 65 QI team members) completed the survey for a response rate of 70.8 percent. With respect to reactions, the results show a significant improvement from pre- to postassessment in all dimensions—perceived relevance of QI methods, perception of management's interest in the QI project, and availability of resources needed to support the QI project. In addition, in the postsurvey, participants reported a high overall satisfaction level with the experience (4.17 out of 5).

Table 3.

Pre- and Post-QI Project Survey Results (N=46, 70.8% Response Rate)

Reaction	Pre	Post	t-Statistics
Relevance of QI	3.33	4.65	<0.01
Management interest in project	4.31	4.72	<0.01
Availability of resources needed	2.98	3.53	<0.01
Satisfaction working on project	—	4.17	—
Learning	Usefulness	Future Applicability	t-Statistics
Model for improvement	4.40	4.53	NS
PDSA cycle	4.49	4.70	<0.05
Process mapping	4.44	4.65	<0.01
Control charts	3.98	4.33	<0.05
Run chart	4.21	4.47	<0.05
Cause and effect diagram	4.33	4.62	<0.05
Memory jogger	4.37	4.53	NS
Radar chart	4.14	4.30	NS
Aim specification	4.44	4.65	<0.05
Measure development	4.28	4.69	<0.05
Behavior	Pre	Post	t-Statistics
Qualified to implement a QI project	2.78	4.19	<0.01
Likelihood of future participation	3.61	4.47	<0.01
Outcome	Pre		Post
Concern that costs exceed benefits	—		3.02
Level of success of QI project	—		4.29

Note. Scale: 1 (low) to 5 (high).

NS, not significant.

In terms of learning, Table 3 shows a high level of perceived usefulness of the 10 QI techniques learned (ranging from 3.98 for control charts to 4.49 for PDSA cycles), and a very high level of the perceived applicability of the techniques in the future (ranging from 4.3 for radar charts to 4.7 for PDSA cycles). Table 3 also shows that participants consistently rate future applicability higher than usefulness; also a paired t-test demonstrates statistical significance for 7 out of 10 of the QI techniques. In terms of behavior, Table 3 indicates a large significant improvement from pre- to postassessment in participants' perception of their qualification to participate in QI projects (increase from 2.78 to 4.19 out of 5) as well as in participants' likelihood of participation in a future QI project (increase from 3.61 to 4.47). Finally, in terms of outcomes, Table 3 shows a moderate concern on the part of participants that costs may exceed benefits (3.02 out of 5) but a high level of perceived success of the QI projects (4.29 out of 5). Furthermore, as reported in Table 1, an examination of preliminary outcomes reported by the QI teams in their final reports shows that five out eight projects report a high level of perceived overall success, one project shows moderate perceived success, and the remaining two projects report low perceived success. In the first of the two low-success projects—decreasing “no show” rates at the women, infant, and children (WIC) clinics—the QI techniques led to positive actions on the part of the nine participating local health departments and improved the WIC clinic experience and relationships with clients. However, the timing of the implementation of the recommended strategies varied greatly across the departments, thus leading to little improvement overall. With time, it is believed that success can be achieved. In the second of the two low-success projects—incorporating public health competencies into staff performance assessments—the local health department struggled with the complexity of the project and the number of stakeholders that needed to participate. As a result, the team underestimated the time required to study the process and implement the newly designed performance management system, and the project did not achieve its initial aim. However, the team strongly felt that the QI techniques and the collaborative approach were highly beneficial.

DISCUSSION

In this study, we examined the impact of a statewide pilot QI training program designed for public health departments. Using a mixed-method design, we assessed participant reaction, learning, behavior, and outcomes. Our findings strongly indicate that the QI collaborative model using videoconferencing, webinars, and expert technical facilitation can be undertaken to successfully implement and complete QI projects in local health departments on a statewide basis. The participants reported (1) high levels of satisfaction with the training sessions, (2) increased perception of the relevance of the QI techniques, (3) increased perceived knowledge of all specific QI methods and techniques, (4) increased confidence in applying QI techniques on future projects, (5) increased intention to apply techniques on future QI projects, and (6) high perceived success of, and satisfaction with, the QI projects. Finally, preliminary outcomes data show moderate to large improvements in quality and/or efficiency for six out of eight projects.

It should be noted that these results are subject to a number of limitations. The external validity of this pilot study may be limited by self-selection of the participants. The agencies that applied to participate in the collaborative may have had a predisposition for change and the learners may have been predisposed to liking the learning. This decision to participate in a QI program is revealing (Wilson, Berwick, and Cleary 2003; Shortell 2006;). It is also possible that participating agencies had leaders who were more supportive of QI than nonparticipating agencies. Finally, while selection criteria ensured that the collaboratives chose a significant project, it is possible that attention to more complex processes may have been avoided. Nevertheless, the projects were a good starting point for all participating agencies. In fact, the process used in this first cycle of the MQIC program mirrors the initial efforts of the National Demonstration Project for QI in Health Care (NDP), which allowed participating organizations to choose their own project (Kenney 2008). Subsequently, the NDP led to the formation of the IHI. MQIC is following a similar path and the next cycle of training and projects has begun under the Multi-State Learning Collaborative-3 (MLC-3) funded by the Robert Wood Johnson Foundation.

The findings from this research provide strong preliminary evidence that QI methods can be successfully implemented in local public health agencies on a statewide basis through distance training and expert facilitation. While previous evidence on the impact of QI collaboratives is limited (Schouten et al. 2008), this study contributes to understanding the use of QI in public health practice-based research. Advancing QI in public health should identify points of leverage to move from sporadic efforts to spreading performance management systems that lead to improved health outcomes (Davis 2008). The results of this study provide preliminary support for public health agencies to adopt QI and suggest avenues for future public health systems research into sustained performance improvement, leadership commitment, and aligning goals with resources. The MQIC pilot program is being further improved and deployed and holds great promise to facilitate the successful dissemination of proven QI methods throughout local public health departments.

CONCLUSIONS

Improving the quality of the health of communities and populations requires a commitment to delivering quality public health services based on sound scientific evidence (Sumaya 2006). Public health practice-based research is needed to build an evidence base of applied studies on how to best organize, finance, and deliver public health services across U.S. communities (Cioffi, Litchtveld, and Tilson 2004; Mays et al. 2004;). However, the dearth of evidence supporting effective public health practices results in inefficient models and processes within the public health infrastructure (Sumaya 2006). While there are numerous areas of public health practice where evidence about the most effective ways of service deliveries do not exist (Lenaway et al. 2006; AcademyHealth 2007;), QI studies are essential for better understanding effective ways to improve population health. Identifying effective ways to educate public health professionals for the 21st century is also a pressing agenda for public health (Institute of Medicine 2003). This study indicates that public health professionals can be effectively trained, become competent in using QI techniques and result in improved outcomes. This study helps contribute to the expanding volume and quality of research on public health services (Scutchfield et al. 2007).