Longitudinal Evaluation of Quality Improvement and Public Health Accreditation Readiness in Nebraska Local Health Departments, 2011-2016

Abstract

Objectives:

Public health accreditation is intended to improve the performance of public health departments, and quality improvement (QI) is an important component of the Public Health Accreditation Board process. The objective of this study was to evaluate the QI maturity and accreditation readiness of local health departments (LHDs) in Nebraska during a 6-year period that included several statewide initiatives to progress readiness, including funding and technical assistance.

Methods:

We used a mixed-methods approach that consisted of both online surveys and key informant interviews to assess QI maturity and accreditation readiness. Nineteen of Nebraska’s 21 LHDs completed the survey in 2011 and 2013, 20 of 20 LHDs completed the survey in 2015, and 19 of 20 LHDs completed the survey in 2016. We facilitated a large group discussion with staff members from 16 LHDs in 2011, and we conducted key informant interviews with staff members from 4 LHDs in 2015.

Results:

Both QI maturity and accreditation readiness improved from 2011 to 2016. In 2011, of 19 LHDs, only 6 LHD directors agreed that their LHD had a culture that focused on QI, but this number increased every year up to 12 in 2016. The number of LHDs that had a high capacity to engage in QI efforts improved from 3 in 2011 to 8 in 2016. The number of LHDs with a QI plan increased from 3 in 2011 to 10 in 2016. The number of LHDs that were confident in their ability to obtain Public Health Accreditation Board accreditation improved from 6 in 2011 to 13 in 2016. Although their QI maturity generally increased over time, LHDs interviewed in 2015 still faced challenges adopting a formal QI system. External financial and technical support helped LHDs build their QI maturity and accreditation readiness.

Conclusion:

Funding and technical assistance can improve LHDs’ QI maturity and accreditation readiness. Improvement takes time and sustained efforts by LHDs, and support from external partners (eg, state health departments) helps build LHDs’ QI maturity and accreditation readiness.

With the introduction of the Public Health Accreditation Board (PHAB) national accreditation in 2011, many local health departments (LHDs) in the United States began preparing their departments for accreditation. One step toward achieving PHAB accreditation is incorporating a quality improvement (QI) system; LHDs need to have a continuous QI system, which involves valuing and routinely practicing QI, in place to achieve accreditation.¹ Quality improvement in public health is defined as “a deliberate and defined improvement process, such as Plan-Do-Check-Act, which is focused on activities that are responsive to community needs and improving population health. It refers to a continuous and ongoing effort to achieve measurable improvements in the efficiency, effectiveness, performance, accountability, outcomes, and other indicators of quality in services or processes that achieve equity and improve the health of the community.”²

In 2011, the University of Nebraska Center for Rural Health Research began evaluating QI implementation in Nebraska LHDs. Nebraska created a regional public health system in 2001, and 20 LHDs currently provide the 10 essential public health services³ for Nebraska’s 93 counties. The public health system in Nebraska is decentralized; that is, local government has both fiscal and programmatic control over most LHD activities.

Nebraska received funding from the Centers for Disease Control and Prevention through the National Public Health Improvement Initiative grant from 2011 through 2014. Funding from this grant was used to provide education and technical assistance to LHDs on both QI and PHAB accreditation preparation activities, QI, and performance management. These funds were also used to provide small multiyear grants to LHDs to support the completion of accreditation prerequisites (eg, the community health improvement plan) or to develop a workforce or QI plan. From May 2014 to May 2016, the Nebraska Department of Health and Human Services also received a Gaining Ground grant from the Robert Wood Johnson Foundation. This grant, which is managed by the National Network of Public Health Institutes, was designed to advance QI and PHAB accreditation of health departments.⁴

Nebraska was 1 of 7 states that received Gaining Ground funding. Using this funding, Nebraska provided LHDs with training opportunities related to accreditation, such as branding, performance management, and public health law. Before Gaining Ground, only a few LHDs in Nebraska participated in a “community of practice,” which involved LHDs sharing their accreditation and QI experiences and lessons learned with each other. Gaining Ground funding was used to expand this community of practice into a more structured technical assistance sharing opportunity whereby all LHDs in Nebraska were invited to participate in scheduled QI and accreditation conference calls. For example, during these calls, LHDs shared examples of documentation they were creating to meet PHAB requirements and identified obstacles that individual LHDs were experiencing.

Nebraska also used Gaining Ground funding to give LHDs accreditation preparation grants of varying amounts based on the LHD’s level of readiness and preparation for accreditation. Larger grants were given to LHDs that had submitted a PHAB application; smaller grants were given to LHDs that had completed all prerequisites and intended to apply for PHAB accreditation and to LHDs that had completed some prerequisites but were uncertain about their intention to apply for PHAB accreditation. Seventeen of 20 LHDs participated in the grant. All 20 LHDs were offered the opportunity to participate in the community of practice and all trainings. We describe the results of our evaluation on QI implementation and accreditation readiness among Nebraska LHDs from 2011 through 2016.

With the implementation of the aforementioned state-level activities, it was important to evaluate how QI maturity and accreditation readiness in Nebraska LHDs changed from 2011 to 2016. Quality improvement maturity refers to an organization’s (1) culture that has values and norms supportive of QI, (2) internal capacity to assess and improve quality, (3) ability to align and diffuse QI efforts throughout the organization, and (4) practice of QI projects.⁵ These 4 domains of QI maturity formed the basis for evaluating progress in Nebraska LHDs: the first 3 domains evaluated the LHDs’ culture and competency, and the fourth domain evaluated QI practice. Both QI and accreditation readiness are processes that require an investment of resources,⁶ leadership support,⁷ and time⁸ to shape the LHD’s culture to one that is receptive to and capable of continuous improvement and performance measurement.⁹

Methods

We used a longitudinal, mixed-methods approach to evaluate progress in QI maturity and accreditation readiness among LHDs in Nebraska from 2011 through 2016 using a validated measurement tool. The Nebraska Center for Rural Health Research collaborated with the Nebraska Public Health Practice-Based Research Network to collect data on QI maturity and accreditation readiness from Nebraska LHDs. We obtained institutional review board approval from the University of Nebraska Medical Center for all data collection efforts.

Quantitative Methods

Nebraska LHDs completed online surveys about their QI maturity and accreditation readiness at 4 time points: 2011, 2013, 2015, and 2016. LHDs were invited to participate in the surveys by email. The surveys incorporated questions from the 10-item validated version of the Quality Improvement Maturity Tool.¹⁰ The 10-item Quality Improvement Maturity Tool measures the first 3 aforementioned domains of QI maturity scored on a 5-point Likert scale (from 1 = LHD strongly disagrees with the statement to 5 = LHD strongly agrees with the statement). The survey also included questions about the fourth QI domain of practice of QI projects, and in 2011 and 2015, the survey asked LHDs about QI models, techniques, and metrics they used.¹¹ Surveys in each year asked questions about accreditation readiness, although not all questions were asked each year. In 2011 and 2013, a total of 19 of 21 LHDs completed the survey. One LHD disbanded in 2014; in 2015, all 20 LHDs completed the survey, and in 2016, a total of 19 LHDs completed the survey. We used the number of surveys completed as the denominator for each year of data. LHD directors completed most surveys, but in a few instances, a staff member who was knowledgeable about the LHD’s QI and accreditation readiness activities completed the survey.

Qualitative Methods

Staff members at the Nebraska Center for Rural Health Research interviewed LHDs about their QI activities and accreditation readiness in 2011 and 2015. In 2011, the Nebraska Center for Rural Health Research facilitated a group discussion with staff members and directors from 19 LHDs to discuss challenges of QI implementation and the QI strategies that were most effective. In 2015, after completion of the online survey that was administered 1 year into the Gaining Ground initiative, the Nebraska Center for Rural Health Research selected 4 LHDs for interviews. We selected these 4 LHDs because they varied in size (ie, number of people on staff and population served) and in readiness to apply for accreditation. The Nebraska Center for Rural Health Research invited LHD directors and staff members involved in QI and/or accreditation to participate in these interviews, and each LHD was interviewed separately by telephone. Research assistants at the Nebraska Center for Rural Health Research transcribed the 2011 facilitated group discussion and the 2015 key informant interviews, and multiple coders discussed emergent codes and themes until they reached consensus.

Results

Quality Improvement Maturity and Accreditation Readiness

In 2011, only 6 of 19 LHD directors agreed that their LHD had a culture that focused on QI, but this number increased every year, to 10 in 2013, 11 in 2015, and 12 in 2016. Most directors in every survey year agreed that involving all staff members in contributing to decisions was important (19 in 2011, 18 in 2013, 18 in 2015, and 18 in 2016) (Figure 1).

Figure 1.

Ten-item questionnaire on quality improvement (QI) maturity in public health accreditation at local health departments (LHDs) in Nebraska, 2011-2016. Nineteen LHDs responded to the online survey in 2011, 2013, and 2016, and 20 LHDs responded to the online survey in 2015.

All 3 measures of QI capacity and competency improved from 2011 to 2016. The number of LHDs that had a high capacity to engage in QI efforts improved from 3 in 2011 to 8 in 2016. The number of LHDs with a QI plan increased from 3 in 2011 to 10 in 2016 (Figure 1).

LHDs also improved in aligning their overall activities and protocols with continuous performance improvement. The number of LHDs with policies and plans that reflected their commitment to QI rose from 11 in 2011 to 14 in 2016, and the number of key decision makers who believed that QI was very important to the LHD rose slightly, from 16 in 2011 to 17 in 2016. Most LHDs in every survey year allowed staff members to work within and across program boundaries to make improvements, although the number of LHDs that did so dropped from 17 in 2011 to 16 in 2016. The number of LHD directors who agreed that customer satisfaction information was routinely used to improve programs and services varied by survey year, from 10 in 2011, to 15 in 2013, to 14 in 2015, and to 12 in 2016. Only 9 LHD directors agreed that job descriptions currently included responsibilities related to measuring QI in 2016 (Figure 1).

The number of LHDs reporting that they had been involved in QI for at least 5 years dropped from 9 in 2011 to 3 in 2016. The number of LHDs reporting no systematic QI efforts increased from 3 in 2011 to 4 in 2016 (Figure 2). Lean and Model for Improvement were the 2 most frequently used models in 2011 and 2015; however, the number of LHDs using Lean decreased from 7 in 2011 to 4 in 2015, whereas the number of LHDs using Model for Improvement rose from 5 in 2011 to 10 in 2015. The most commonly used QI techniques in 2015 were brainstorming (n = 19), cause-and-effect diagrams (n = 11), and run charts (n = 9) (Table).

Figure 2.

Length of time local health departments (LHDs) engaged in quality improvement (QI) practice toward public health accreditation, Nebraska, 2011-2016. Nineteen LHDs responded to the online survey in 2011, 2013, and 2016, and 20 LHDs responded to the online survey in 2015. LHDs responded to the survey question: How long has your LHD been engaged in established and consistent efforts to improve the quality of services?

Table.

Number of local health departments in Nebraska using quality improvement models and techniques toward achieving public health accreditation, 2011 and 2015

Model or Technique	2011, No. (n = 19)	2015, No. (n = 20)
Model
Leana	7	4
Model for Improvementb	5	10
Baldridgec	4	1
Six Sigmad	2	0
Other	0	3
Technique
Brainstorminge	15	19
Prioritization matrixf	10	7
Run chartsg	8	9
Cause-and-effect diagramsh	5	11
Control chartsi	5	5
Spaghetti mapsj	3	3
Radar chartsk	2	5
5Sl	2	0

^aLean Enterprise Institute.¹²

^bInstitute for Healthcare Improvement.¹³

^cNational Institute of Standards and Technology.¹⁴

^dAmerican Society for Quality.¹⁵

^eAmerican Society for Quality.¹⁶

^fSix Sigma Online.¹⁷

^gInstitute for Healthcare Improvement.¹⁸

^hAmerican Society for Quality.¹⁹

ⁱAmerican Society for Quality.²⁰

^jAmerican Society for Quality.²¹

^kMinnesota Department of Health.²²

^lAmerican Society for Quality.²³

The number of LHDs that were confident in their ability to obtain PHAB accreditation improved from 6 in 2011 to 13 in 2016. In 2016, most LHDs also agreed that accreditation would enhance their credibility with stakeholders (n = 14) and improve the quality of their services (n = 14), and these measurements improved from the 2011 survey: 13 for enhancing credibility with stakeholders and 11 for improving the quality of their services. LHDs that indicated they would see PHAB accreditation in the future improved from 13 in 2011 to 18 in 2016 (Figure 3).

Figure 3.

Measures of public health accreditation readiness of local health departments (LHDs) in Nebraska, 2011-2016. Nineteen LHDs responded to the online survey in 2011, 2013, and 2016, and 20 LHDs responded to the online survey in 2015.

Qualitative Research Results

Facilitated discussion with 20 LHD staff members from 16 LHDs in 2011 revealed that QI implementation was limited by a lack of knowledge about QI and the culture, capacity, and resources of the LHDs. LHDs struggled with implementing QI because most staff members were not trained in public health QI methodologies; however, staff members felt that they were capable of using informal QI models and techniques. LHDs also reported a need to change the culture in their agencies to be more supportive of QI and for staff members to understand the need for them to point out ways to make improvements in their departments. Another limiting factor was that the public health system in Nebraska was still young, and other tasks besides QI were more important in the beginning stages of establishing a health department. In addition, most Nebraska LHDs employed fewer than 10 staff members, so staff members were often overwhelmed with the job duties associated with the 10 essential public health services, leaving little time to tackle QI initiatives. Some LHDs also noted that training staff members and dedicating time for them to participate in QI activities would overextend their budgets.

Directors and staff members with knowledge about their LHD’s QI and accreditation activities from 4 LHDs at various levels of QI maturity were interviewed again in 2015. For these LHDs, formalizing the QI process was a barrier despite having implemented QI activities for several years. Staff members at the LHDs were uncertain about the appropriate QI model or technique to use and had difficulty understanding the QI process. To address the challenge of formalizing QI activities, LHDs approved agency-wide policies promoting QI and formed internal QI committees. One LHD seeking PHAB accreditation made substantial changes to the way staff members documented their work and enacted more than 30 new forms in a single year, which allowed the LHD to track functions in a standardized way for both accreditation and QI purposes. The results of QI activities and progress toward reaching PHAB accreditation milestones were shared in LHD newsletters, which provided accountability for maintaining progress, a venue for sharing new information, and a way to normalize both QI and accreditation as a vital aspect of the LHD’s activities. LHDs fully committed to both QI and PHAB accreditation made participation in QI officially part of job descriptions and, when possible, reclassified positions that were responsible for these activities.

One of the lessons learned by LHDs about QI implementation was that to integrate QI into all facets of the LHD, staff members need to understand that QI is for everyone and a part of every program, rather than extra work in addition to their regular responsibilities. Because staff members had varying levels of QI expertise and comfort with change processes and a lack of resources to formally train all staff members, LHDs dedicated time for individual consultations on QI projects and organized group activities, such as peer-to-peer review, to facilitate successful experiences implementing QI. Although LHD staff members might have initially been hesitant to discuss areas needing improvement or their confusion about a QI tool with their supervisor, peer-to-peer review made staff members more comfortable discussing their ideas and questions about QI. Even in large health departments that had greater institutional knowledge and more resources available for formal training than small health departments, familiarity with QI differed among staff members. The LHDs also reported that the training opportunities and financial assistance provided by the Nebraska Division of Public Health were crucial to advancing their QI maturity and accreditation readiness. For example, the division made external QI experts available to educate LHD staff members about the overall value of QI and provided information about QI techniques. The division also had internal QI experts work with individual programs and service units to help design QI projects and document the impact of these projects on the Public Health Quality Improvement Exchange (an online resource for public health practitioners to interact around public health QI).²⁴ Finally, the state agency provided small grants to a few LHDs to encourage the implementation of QI projects and build QI expertise among LHD staff members.

Lessons Learned

Several lessons can be learned from this evaluation of Nebraska’s QI and accreditation readiness activities that can be applied to LHDs nationally. First, it is important to provide various training and educational opportunities to LHD staff members, and these activities need to span a long period of time. For example, Nebraska’s initial QI training activities focused on providing a general overview, the potential impact, and the relationship between QI and accreditation readiness. Subsequent training sessions discussed QI techniques. It is critical that the training be delivered continuously, not in short bursts of activity. Once a general understanding of QI and its importance exists, LHD staff members must be encouraged to apply QI techniques to QI projects. Small grants can be used as an incentive, but in initial QI projects, it is important to make technical assistance available. When issues arise in the design of QI projects (eg, selecting the appropriate methods, collecting and analyzing the data, and interpreting the results), it is essential to have QI experts who can answer questions and help guide the effort.⁷ Without this assistance, LHD staff members can become frustrated and sometimes fail to finish projects.

A second lesson learned is that to create a QI culture throughout an organization, all staff members must understand and apply QI concepts and techniques.²⁵ Although most directors in every survey year reported that their leaders were trained in basic QI methods, the low capacity to engage in QI indicates the need to provide training to all public health staff members, not just department leaders. Although not all staff members will have the same expertise with QI, peer-to-peer reviews can be useful. Continuous and updated educational programs can reinforce and build on past experiences. A related lesson is that LHDs can learn from each other; as such, it is important to create opportunities not only to share the results of QI projects but also to show how challenges were overcome. A community of practice that was formed in Nebraska provided an opportunity to share results and best practices. Quality improvement sessions were also held at conferences and other venues across the state.

Another lesson learned was that both large and small LHDs can master and implement QI projects. This finding is important given that nationally, 37% of LHDs employ fewer than 10 full-time equivalent employees.²⁶ Although smaller LHDs have fewer resources and fewer staff members than larger LHDs, strong leadership by both the director and the board can influence the culture and integration of QI throughout the organization.

Finally, this evaluation confirms that changing the QI culture and increasing accreditation readiness beliefs at LHDs takes time and may not always progress linearly, particularly when limited knowledge and understanding about the QI process and techniques exist. For example, we found that fewer LHDs agreed that they had been conducting systematic QI for 5 or more years in 2016 than in 2011. This phenomenon was found in other studies and can be explained either by the LHD’s enhanced exposure to formal QI between 2011 and 2016 or by discontinuation of QI during this time period.²⁷ LHD directors may have thought that their staff members were conducting QI improvement in 2011, but after several years of formal training provided by the state health agency and other partners, the directors may have realized that their activities did not meet the true definition of QI. During this time, several LHDs had changes in key administrative staff members, including directors, which could have contributed to a loss of knowledge about the LHD’s efforts and commitment to QI.

Limitations

This evaluation had several limitations. First, the small sample size limited our ability to conduct significance testing for changes in QI and accreditation readiness over time. Second, social desirability bias could have resulted in LHDs overreporting their QI and accreditation readiness activities. Finally, only 1 person from each LHD responded to the survey each year, and the respondent was not always the same person each year. For example, a new LHD director or staff member in charge of QI and accreditation might have completed the survey because he or she was most knowledgeable about current activities at the LHD but might have lacked historical knowledge about relevant activities that occurred before their employment. However, multiple people from each LHD participated in the qualitative portion of this evaluation.

Conclusions

Nebraska used multiple funding opportunities to build the capacity of LHDs to move forward on QI and accreditation readiness. Several years of funding allowed for the Nebraska Division of Public Health to provide small grants to LHDs in various years to sustain and grow their accreditation readiness and QI. Several years of funding at the state level also paid for multiple training opportunities and growth of a community of practice that provided education and technical assistance that was responsive to the LHDs’ needs. Because building momentum takes time, and because staff turnover in LHDs results in loss of institutional knowledge of QI,⁹ it is important to have long-term funding, technical assistance, and learning communities, such as Nebraska’s QI and accreditation community of practice. To accelerate this capacity building, the Centers for Disease Control and Prevention and other federal agencies, private foundations, and states should support QI and accreditation readiness initiatives even when budgets are tight. Improvements in quality and performance of accredited health departments will lead to more effective, efficient programs and practices²⁸ and, ultimately, to better population health outcomes.