Advances in Public Health Accreditation Readiness and Quality Improvement: Evaluation Findings From the National Public Health Improvement Initiative

Abstract

Introduction

Continuous quality improvement is a central tenet of the Public Health Accreditation Board’s (PHAB) national voluntary public health accreditation program. Similarly, the Centers for Disease Control and Prevention launched the National Public Health Improvement Initiative (NPHII) in 2010 with the goal of advancing accreditation readiness, performance management, and quality improvement (QI).

Objective

Evaluate the extent to which NPHII awardees have achieved program goals.

Design

NPHII awardees responded to an annual assessment and program monitoring data requests. Analysis included simple descriptive statistics.

Setting

Seventy-four state, tribal, local, and territorial public health agencies receiving NPHII funds.

Participants

NPHII performance improvement managers or principal investigators.

Main Outcome Measure(s)

Development of accreditation prerequisites, completion of an organizational self-assessment against the PHAB Standards and Measures, Version 1.0, establishment of a performance management system, and implementation of QI initiatives to increase efficiency and effectiveness.

Results

Of the 73 responding NPHII awardees, 42.5% had a current health assessment, 26% had a current health improvement plan, and 48% had a current strategic plan in place at the end of the second program year. Approximately 26% of awardees had completed an organizational PHAB self-assessment, 72% had established at least 1 of the 4 components of a performance management system, and 90% had conducted QI activities focused on increasing efficiencies and/or effectiveness.

Conclusions

NPHII appears to be supporting awardees’ initial achievement of program outcomes. As NPHII enters its third year, there will be additional opportunities to advance the work of NPHII, compile and disseminate results, and inform a vision of high-quality public health necessary to improve the health of the population.

Standards, quality improvement (QI), and performance management (PM) are well-established concepts that have the potential to strengthen organizational performance and increase efficiency and effectiveness.¹ With the introduction of the Public Health Accreditation Board (PHAB) in 2007 and the launch of its national voluntary accreditation program in 2011, PHAB expanded upon existing efforts to define and promote public health standards such as those by the National Public Health Performance Standards Program (NPHPSP)² and Project Public Health Ready.³ Intended to drive continuous QI,⁴ the PHAB accreditation program provides an opportunity to promote PM and QI in public health practice. In turn, QI and PM activities are critical to accreditation readiness. In recognition of this interplay, the Centers for Disease Control and Prevention (CDC) applied them as a mutually reinforcing framework for its National Public Health Improvement Initiative (NPHII).

Background

PHAB’s national voluntary public health accreditation program, funded by the CDC, the Robert Wood Johnson Foundation (RWJF), and fees paid by participating health departments, aims to improve and protect the public’s health by advancing the performance and quality of the nation’s state, tribal, local, and territorial (STLT) health agencies.⁵ With accreditation, there are now nationally recognized standards to foster organizational efficiency and effectiveness and promote accountability and continuous QI in public health agencies. By March 4, 2013, 11 public health departments had already received 5-year accreditation status,⁶ with many more public health agencies preparing to meet the national standards and seek accreditation.^7,8

Accreditation and QI share mutual goals of strengthening public health agencies and transforming public health practice. Both efforts have the potential to enable organizations to fill important performance gaps in meeting the essential public health services, to respond quickly and strategically to emerging challenges, and to demonstrate results in areas such as program operations, service delivery, and health outcomes.⁹ Yet, published findings on the near- and long-term public health impact of these activities are limited. A study on North Carolina’s state-based accreditation program found that two-thirds of the accredited local health departments conducted QI projects following accreditation.¹⁰ Studies specific to QI and PM also yield positive results. Findings from national initiatives such as the RWJF-sponsored Multi-State Learning Collaborative,¹¹ the NPHPS,¹² and the Turning Point Performance Management Collaborative¹³ illustrate the application of PM and QI practices among participating agencies, with, in the case of Turning Point, improvements in structures, processes, and in some instances health-related outcomes. Similarly, in a study of Florida’s public health departments, QI and PM were associated with improvements in selected health status indicators.¹⁴

Building on the growing momentum to engrain QI in public health practice, the CDC launched NPHII. A 5-year cooperative agreement funded through the Prevention and Public Health Fund of the Affordable Care Act, NPHII provides financial and technical support to STLT public health agencies. Seventy-six STLT awardees received a total of $42.5 million in the program’s first year (September 30, 2010-September 29, 2011). Seventy-four of those agencies were awarded an additional $33.5 million in its second year (September 30, 2011-September 29, 2012), including 48 state health departments, the District of Columbia health department, 8 American Indian/Alaska Native tribes/organizations, 9 local health departments, and 8 territories and freely associated states of the Pacific.¹⁵

The first year of the NPHII program required awardees to hire a performance improvement manager (PIM) with the responsibility to foster organizational QI. In its second year, NPHII clarified its intended outcomes to include achievement of public health standards; implementation of organization-wide PM and QI; and greater efficiency and effectiveness in public health operations, programs, and services.

Methods

The evaluation employs a utilization-focused,¹⁶ mixed-method design, incorporating quantitative and qualitative data collected directly in support of the evaluation, as well as data collected for program monitoring purposes. All data included in this article were collected in late 2012 or early 2013 and represent the status of NPHII awardee activities as of the end of the program’s second year (September 29, 2012).

NPHII annual assessment of performance management and improvement practices

An online assessment was implemented between November 2012 and January 2013 to measure awardees’ progress toward achieving NPHII outcomes in the areas of accreditation readiness, PM, and QI as of the end of the second program year. The awardee organization’s PIM, or NPHII principal investigator if there was no PIM, completed the self-reported assessment.

Awardee project plans and progress reports

For program monitoring purposes, NPHII awardees are required annually to submit project plans prior to the program year and annual progress reports (APRs) at the end of each year. Project plans document awardees’ annual intentions for meeting cooperative agreement expectations. A new requirement was introduced in the third year of NPHII funding (September 30, 2012–September 29, 2013) for awardees to complete an organizational assessment against the PHAB Standards and Measures, Version 1.0; the associated project plan required awardees to report whether or not they had already completed such an assessment as of August 2012 and, if completed, to indicate any gaps identified and progress made by their organization in meeting PHAB standards. APRs outline awardee accomplishments and challenges related to their organization’s proposed activities during the funding period. The APR data for the second NPHII program year were collected from November 2012 through January 2013. All data in the project plans and progress reports were self-reported.

Data analysis

Quantitative analysis of data from all sources included simple descriptive statistics. When appropriate, more detailed comparisons by awardee STLT type were conducted to identify patterns or differences. Cross-tabulations were performed to identify any associations between various organizational factors and select outcomes. No meaningful differences were identified by awardee type or organizational factors, so all data are presented in aggregate. Qualitative, free-text responses in the assessments and progress reports were coded and analyzed to extract relevant themes.

Results

Data presented represent findings as of the end of the second year of NPHII funding for the 73 awardees responding to all data requests, unless otherwise noted.

Accreditation readiness

PHAB prerequisites

Data collected in the APR indicate that 15% of NPHII awardees (8 states, 3 locals) had completed all 3 prerequisites, including a health assessment, health improvement plan, and strategic plan. An additional 14% of awardees (9 states, 1 local) had completed 2 of the 3 prerequisites and were in the process of completing the third. However, 7% of awardees (1 state, 1 tribe, and 3 territories) indicated no progress toward any of the prerequisites. The Figure shows the status of each prerequisite among all awardees.

FIGURE. Status of PHAB (Public Health Accreditation Board) Prerequisites as of the End of NPHII (National Public Health Improvement Initiative) Year 2.

Awardees reported using their completed prerequisites to advance the work of their agencies or jurisdictions’ public health systems. Among the 31 NPHII awardees that completed a health assessment, 84% (n = 26) indicated using it to inform or update their health improvement plan or identify health priorities. Likewise, 68% (n = 13) of the 19 awardees with a completed health improvement plan reported using the plan to address strategic priorities and/or inform a formal strategic planning process and 37% (n = 7) reported that using the plan had allowed them to strengthen collaboration with partners. Among the 35 awardees that completed a strategic plan, 37% (n = 13) reported using it to align or prioritize agency activities, and 20% (n = 7) reported using it to develop, update, or track agency performance measures.

Organizational assessments against PHAB Standards and Measures, Version 1.0

By August 2012, 26% (n = 19) of awardees had completed an organizational self-assessment against the PHAB Standards and Measures, Version 1.0. Of those awardees, 11% (n = 2) indicated having already met all of the standards. The remaining 89% (n = 17) of awardees having completed an assessment were at various stages of addressing identified gaps. The Table presents the status of each domain and standard for those awardees that had completed a self-assessment.

TABLE. Status of Each Public Health Accreditation Board (PHAB) Domain and Standard From the PHAB Standards and Measures, Version 1.0 for Awardees With a Completed Self-Assessment as of August 2012.

Domain	Domain Complete n	Standard	Standard is Met n	Gap Identified; Addressing n	Gap Identified; Not Yet Addressed n
Domain 1.	7	1.1	11	7	1
Conduct and disseminate assessments focused on  population health status and public health issues  facing the community		1.2	8	8	3
		1.3	11	6	2
		1.4	10	5	4
Domain 2.	6	2.1	12	4	3
Investigate health problems and environmental public  health hazards to protect the community		2.2	8	9	2
		2.3	11	5	3
		2.4	10	7	2
Domain 3.	10	3.1	13	4	2
Inform and educate about public health issues and  functions		3.2	10	7	2
Domain 4.	10	4.1	13	4	2
Engage with the community to identify and address  health problems		4.2	14	5	0
Domain 5.	3	5.1	14	2	3
Develop public health policies and plans		5.2	7	10	2
		5.3	7	10	2
		5.4	10	7	2
Domain 6.	9	6.1	12	4	3
Enforce public health laws		6.2	13	4	2
		6.3	9	5	5
Domain 7.	11	7.1	13	4	2
Promote strategies to improve access to health care  services		7.2	13	5	1
Domain 8.	2	8.1	11	5	3
Maintain a competent public health workforce		8.2	3	14	2
Domain 9.	2	9.1	3	14	2
Evaluate and continuously improve health  department processes, programs, and interventions		9.2	4	14	1
Domain 10.	9	10.1	10	7	2
Contribute and apply the evidence base of public  health		10.2	9	8	2
Domain 11.	5	11.1	5	13	1
Maintain administrative and management capacity		11.2	13	4	2
Domain 12.	7	12.1	11	5	3
Maintain capacity to engage the public health  governing entity		12.2	9	8	2
		12.3	8	8	3

Other accreditation-readiness activities

Annual assessment data show that 93% (n = 68) of NPHII awardees engaged in various accreditation-readiness activities. The most frequently cited activities were conducting communications or meetings with leadership (86%, n = 63) or staff (84%, n = 61), participating in an accreditation planning or advisory group (75%, n = 55), designating individuals to coordinate accreditation readiness activities (66%, n = 48), and implementing activities to complete the PHAB readiness checklist (60%, n = 44). Awardees were less likely to report the following activities: submitting a statement of intent to pursue PHAB accreditation (21%, n = 15), promoting accreditation readiness activities among other health departments in the awardee’s jurisdiction (33%, n = 24), and organizing agency documentation for accreditation (38%, n = 28).

Performance management and quality improvement

Development of an organization-wide performance management system

Awardees reported in the annual assessment whether they had established any or all of the components of a PM system, including performance standards, performance measures, routine performance reporting, and organization-wide processes for QI as defined by the Turning Point Performance Management National Excellence Collaborative.¹⁷ Twenty-two percent (n = 16) reported having all 4 components in place (9 states, 4 locals, and 3 tribes) and 72% (n = 52) indicated having established at least 1 of the 4 components. Approximately half of awardees reported establishing performance measures and routine performance reports (51%, n = 37 and 48%, n = 35, respectively). Fewer reported having established organization-wide processes for QI (41%, n = 30) or performance standards (42%, n = 30).

Conducting QI initiatives

In the same assessment, 90% (n = 66) of awardees indicated that they had conducted QI activities focused on increasing efficiencies and/or effectiveness. Seventy-nine percent (n = 58) of awardees focused their activities on increasing efficiencies, with 71% (n = 52) of awardees either having completed or working toward completing activities aimed at saving time and 70% (n = 51) focused on reducing the number of steps required to complete a process or deliver a service. For example, 1 awardee indicated using process mapping, flow charting, and plan-do-study-act to reduce the number of steps required to mail penalty letters to health care facilities from the program from 28 to 14 process steps, and another awardee reported using Lean/Six Sigma and plan-do-study-act to reduce payment time for AIDS Drug Assistance Program invoices from their drug wholesaler, thereby reducing the number of payment days from a mean of 40 to a mean of 12 and avoiding approximately $144 000 a year in costs.

Similarly, 78% (n = 57) of awardees focused their QI efforts on improving the effectiveness of programs, services, or processes. For example, 1 awardee used the plan-do-check-act model for improvement to decrease early elective deliveries by 66% over the previous year. Another awardee demonstrated the combination of multiple strategies to address both efficiency and effectiveness by leveraging non–registered nurse staff to increase sexually transmitted disease clinic efficiency and testing capabilities. As a result, wait time for sexually transmitted disease evaluations decreased from an average of about 1 hour to an average of 28 minutes. Among the awardee’s 4 pilot sites, QI activities increased appointment capacity by 244 appointments and made available an additional 161 diagnosis/treatment slots. The improvements also resulted in high levels of both customer and staff satisfaction (99% and 100%, respectively).

Barriers to organization-wide performance management

Awardees identified, in the annual assessment, the top challenges that their organizations experienced in implementing PM on an organization-wide basis. The 3 most frequently cited challenges included competing priorities (75%, n = 55), limited numbers of staff trained in PM or QI (67%, n = 49), and limited numbers of staff available to conduct PM activities (61%, n = 45). Other frequently cited challenges are related to budgetary constraints such as staff cuts or elimination of services (36%, n = 26) and turnover in staff or leadership (27%, n = 20).

Discussion

This study marks the first evaluation of a national initiative focused on PM, QI, and accreditation readiness. NPHII is currently at its mid-point and, to date, appears to be supporting awardees’ advancement of outcomes in these areas. Most notably, nearly all awardees are conducting accreditation readiness activities and implementing QI projects, both of which are critical to achieving key program goals of meeting public health standards and improving organizational effectiveness and efficiency.

Limitations

All data are self-report and to date verification of the data has been limited. Instability in question wording over time, due to the evolution of NPHII programmatic requirements as well as the evolution of the fields of accreditation and PM, has made cross-time comparisons difficult and at times inappropriate. Several awardee characteristics also affect the quality of the data. Participating local and tribal health departments were not randomly selected for funding. Therefore, data on locals and tribes may not be representative of those populations as a whole. Finally, a number of health departments have funding other than NPHII to support accreditation and QI activities. It is possible that the data reflect activities that were supported in part by other funds.

Opportunities to demonstrate impact

Because continuous QI is central to public health accreditation,⁵ a greater understanding of QI-associated outcomes remains an important step in building the evidence for both NPHII and accreditation. Defining measurable outcomes of efficiency and effectiveness for QI will place the focus not just on how QI is implemented, but also on what it can achieve—an area not yet fully explored in public health.¹⁸ The CDC has begun work in this area by defining a core set of efficiency- and effectiveness-related outcomes for the third funding year of NPHII, along with a framework to standardize their measurement.

Similar opportunities exist with the measurement of accreditation readiness. For example, it is difficult to meaningfully assess the status of PHAB prerequisites. Each prerequisite represents a fluid set of processes and products that are expected to be developed, implemented, and revised. A method for measuring when an organization has sufficiently developed the prerequisites has not been well established. The release of the PHAB Standards and Measures, Version 1.0 has addressed several of these challenges by providing clarity on the expectations associated with each domain, including clarifying processes and documentation associated with the 3 prerequisites. Further efforts to standardize data collection and clarify language about completion of prerequisites would increase confidence in the accuracy and interpretation of these data.

A better understanding of the aggregate impact of QI efforts and the true status of accreditation-preparation activities should result in more robust findings that will help the CDC understand and demonstrate NPHII’s impact and inform the field to advance QI and accreditation readiness outcomes. By funding STLT health agencies across the nation that are at varying stages of accreditation readiness, and QI and PM capacity, the NPHII program has the unique ability to advance the work of these agencies, compile and disseminate results from their efforts, and inform a vision of high-quality public health necessary to improve the health of the population.