Defining and Assessing Quality Improvement Outcomes: A Framework for Public Health

Abstract

We describe an evidence-based framework to define and assess the impact of quality improvement (QI) in public health. Developed to address programmatic and research-identified needs for articulating the value of public health QI in aggregate, this framework proposes a standardized set of measures to monitor and improve the efficiency and effectiveness of public health programs and operations.

We reviewed the scientific literature and analyzed QI initiatives implemented through the Centers for Disease Control and Prevention’s National Public Health Improvement Initiative to inform the selection of 5 efficiency and 8 effectiveness measures.

This framework provides a model for identifying the types of improvement outcomes targeted by public health QI efforts and a means to understand QI’s impact on the practice of public health.

At a time when taxpayer resources are scarce, government agencies are expected to deliver on broader missions while reducing operating costs.^1–6 As stewards of public funds, agencies must implement programs and deliver services as effectively and efficiently as possible on the basis of the best evidence available. Federal programs are required to engage in rigorous measurement and evaluation and use the findings to facilitate continuous improvement and understand the value of services and programs for improved accountability and decision making.⁷ This approach relies on the adoption of valid measures that track progress toward goals, identify areas for improvement, and assess achievement of outcomes.^7,8

In the public health field, quality improvement (QI) is an increasingly recognized approach to maximizing the effectiveness of services while minimizing costs. As defined by Riley et al., public health QI “refers to a continuous and ongoing effort to achieve measurable improvements in the efficiency, effectiveness, performance, accountability, outcomes, and other indicators of quality in services or processes which achieve equity and improve the health of the community.”^9(p6) To date, several initiatives have promoted the use of QI among public health agencies with the goals of reaching these outcomes and building the evidence base. Tools, such as the National Public Health Performance Standards, and initiatives, such as the Turning Point Performance Management Collaborative and the Robert Wood Johnson Foundation–supported Multi-State Learning Collaborative, represent some of the earliest efforts that encouraged health departments to adopt performance management and QI methods as a strategy to strengthen public health systems.^10–12

More recently, new initiatives aimed at integrating QI into the practice of public health have included the Robert Wood Johnson Foundation–funded Communities of Practice for Public Health Improvement, which serves as a forum for public health agencies to exchange best practices related to QI,¹³ and the Centers for Disease Control and Prevention’s National Public Health Improvement Initiative (NPHII), through which 73 state, tribal, local, and territorial public health agencies are funded to achieve public health standards and adopt and institutionalize cross-cutting performance management and QI approaches to improve the accountability, efficiency, and effectiveness of their public health programs and services.^14,15 Most recently, the establishment of the Public Health Accreditation Board and its release of version 1.0—and subsequently version 1.5—standards and measures have driven public health agencies to integrate performance management into their daily practice. The Public Health Accreditation Board has further highlighted QI as an important aspect of the performance management system,¹⁶ supporting the Turning Point initiative, which includes QI as a core component of its performance management framework as a demonstrated means to manage change and make improvements based on data.¹¹

As a result of these efforts, the body of evidence for public health QI is growing, with a focus on the extent to which public health agencies have adopted QI and the kinds of QI processes and tools implemented.^6,12,17–21 However, conceptualizing and assessing outcomes resulting from the implementation of public health QI has proven challenging, in large part because of the diversity of public health contexts^12,20 and the scarcity of evidence-based measurement methods.^5,22,23 Only recently have researchers and practitioners begun to describe or assess outcomes of public health QI in a way that has the potential to demonstrate the impact of this work on public health organizations and the public health system more broadly.^5,22,24,25 Recent studies have described the role of public health QI in addressing service and program processes as well as operational processes.^5,22 Another study highlighted that certain characteristics of QI initiatives correlate with an increased likelihood of attaining stated objectives, including clarity around select measurement parameters such as time frames, baselines, and targets.²⁴ Although progress has been made, these studies have acknowledged that the evidence base for what works in public health QI is still growing and standardized measures for improvement initiatives targeting operational or programmatic efficiency and effectiveness are lacking. To improve performance, public health practitioners and researchers need to clarify what we hope to achieve and continue to build the evidence base for what works.^5,22,25

In recognition of this need, the Public Health Services and Systems Research national research agenda has focused attention on the following QI research questions:

• What measures provide the most valid and reliable indicators of the implementation and impact of QI strategies in public health settings?

• What types of QI strategies have the largest effects on the effectiveness, efficiency, and outcomes of public health strategies delivered at local, state, and national levels?²⁶

To advance the science and practice of QI outcome measurement, we conceptualized a framework that proposes and defines a standardized way to assess public health QI outcomes related to efficiency and effectiveness. The primary purposes of this QI measurement framework are to (1) support public health agencies’ efforts to achieve demonstrable outcomes, (2) provide a means to aggregate the impact of individual QI initiatives, and (3) advance the science and practice of this emerging field.

METHODS

We based our identification of specific outcomes and the development of a standardized measurement approach on an iterative process that used both theoretical and grounded approaches, including a review by Centers for Disease Control and Prevention evaluators (A. W. M., C. T., and A. Y.) and a contractor (S. N.) of both the existing literature and data collected by the Centers for Disease Control and Prevention on QI initiatives reported by NPHII awardees.

We identified peer-reviewed journal articles in PubMed by means of a title–abstract search. The search terms public health and quality improvement were applied together and in combination with each of these additional terms: outcomes, efficiency, effectiveness, and evaluation. After removing duplicates, we identified 147 articles. We conducted an additional PubMed search, applying the terms measurement and public health in the title–abstract field and quality improvement as a text word. This search resulted in 34 articles. Once all duplicates were removed, 170 articles remained. Evaluators reviewed abstracts for these 170 articles and removed those that focused on health care settings, accreditation, or laboratory services, resulting in 35 articles that directly addressed the topic of public health QI. We identified an additional 3 articles through a manual review of a table of contents (volume 16, issue 1, of the Journal of Public Health Management and Practice), resulting in a final total of 38 articles.

Many studies documented the process of QI, including workforce development,^27,28 establishing a culture or environment conducive to QI,^12,29–31 integrating QI within a broader framework,^20,32,33 or describing QI implementation.^{17–19,34–37} Other articles referred to QI outcomes without specifying them.^24,38 Many articles described various types of public health QI efficiency-related outcomes, including cost reductions^{22,31,39–42} and time savings.^{5,9,21,22,40,41,43} Effectiveness-related outcomes were also described, including increased reach of, or access to, programs and services^{5,22,39,41,43,44}; improved quality of data,^5,43,45 programs, or services^21,31,41,45; increased customer or client satisfaction^5,42,44; changes to organizational structure³¹; increased preventive behaviors^{5,22,41–44,46}; and reduced disease incidence or prevalence.^22,43,45 The review also highlighted the need for a robust measurement system^{23,31,33,38,47} to accompany the articulation of outcomes.

Additional inputs included reports by the Institute of Medicine that focused on performance measurement, public health, and health care quality, as well as the US Department of Health and Human Services’ national framework for public health quality. In Crossing the Quality Chasm,⁴⁸ the Institute of Medicine highlighted dimensions of improvement in the personal health care delivery system that are also relevant to public health, including a focus on quality, timeliness, and cost of administrative and clinical or service-delivery processes. Other Institute of Medicine reports^49,50 emphasized the importance of measurement and of maximizing the efficiency and effectiveness of public health services and strategies as a means to make progress toward population health outcomes. The US Department of Health and Human Services’s framework identified efficiency and effectiveness as critical public health system characteristics and core components of successful QI.^8,51 Both the Institute of Medicine reports and the Department of Health and Human Services framework provided conceptual guidance for the organization of the QI measurement framework, yet neither source provided specific guidance on operationalizing concepts in a manner that would facilitate measurement of discrete QI initiatives.

To ensure the framework’s relevance to current practice, we also conducted a grounded review of measures for QI initiatives reported by 74 NPHII awardees to the Centers for Disease Control and Prevention at the end of the 2nd program year. We conducted this review to (1) determine the extent to which awardees’ efforts aligned with outcomes found in the literature, (2) identify additional outcomes to consider for inclusion in the framework, and (3) identify potential measurement challenges. The review confirmed the relevance of efficiency-related outcomes such as cost and time savings, the importance of a focus on health outcomes, and the need for a series of outcomes associated with business processes or program or service delivery improvements, such as standardization and enhancements to services or systems, and a focus on reducing steps associated with various processes. The review also highlighted measurement challenges, including the lack of baseline values or consistent units of measurement.

An initial version of the QI measurement framework was used by 73 awardees during the 3rd year of the NPHII program (September 30, 2012–September 29, 2013) to test its relevance to and utility for their efforts. This testing resulted in a more grounded and refined measurement framework by revealing additional nuances to existing outcomes, and new outcomes, that were subsequently incorporated into the final version. For example, recognizing that several awardees engaged in QI efforts to support work by other public health system partners, we defined a new outcome to capture how broadly QI products or practices are disseminated.

Defining Public Health Quality Improvement Outcomes

The QI measurement framework (Table 1) defines outcomes for 2 key constructs—efficiency and effectiveness—and provides standardized-measure language for each outcome. Specifically, 5 efficiency outcomes and 8 effectiveness outcomes were developed. We used 2 primary criteria in the selection and definition of these outcomes: (1) applicability to a wide variety of public health processes, programs, or services, and (2) relevance to public health agencies’ differing contexts and stages of familiarity with QI.

TABLE 1—

Efficiency and Effectiveness Outcomes in the Quality Improvement Measurement Framework

Outcome	Description of Associated Measure
Efficiency
Time saved	Time to complete a specific process or deliver a specific service
Reduced no. of steps	No. of steps required to complete a specific process or delivery of a specific service
Revenue generated from billable services	Revenue generated by changing the implementation of a billable process or service
Costs saved	Cost to complete a specific process or deliver a specific service
Costs avoided	Cost avoided because of changes in a specific process or delivery of a specific service
Effectiveness
Increased customer or staff satisfaction	Percentage of customers or staff who report being satisfied or extremely satisfied with a specific service or process
Increased reach to a target population	Percentage of target population that has been offered, received, or completed a specific public health service or program
Dissemination of information, products, or evidence-based practices	Percentage of individuals or public health partner organizations reached through dissemination of information, products, or evidence-based practices
Quality enhancement of services or programs	Description of issue or improvement opportunity and its resolution for a specific service or program
Quality enhancement of data systems	Description of issue or improvement opportunity and its resolution for a specific data or health information system
Organizational design improvements	Description of improvements to organizational operations, business processes, or service or program delivery resulting from specific organizational redesign efforts
Increased preventive behaviors	Percentage of preventive or health-promoting behavior or early indicators of preventive behaviors in a target population
Decreased incidence or prevalence of disease	Percentage of individuals with disease in the target population

By definition, efficiency outcomes typically reflect reductions in the amount of resources required to implement activities resulting from a QI initiative. Efficiency outcomes included in the framework are time saved, reduced number of steps, revenue generated from billable services, costs saved, and costs avoided. Compared with the other efficiency outcomes, reduced number of steps is process focused but is the first step to realizing other efficiency gains and may be a more realistic outcome for agencies new to QI.

Three outcomes track efficiencies based on dollar amounts. Revenue generated captures increases in resources, particularly revenue, resulting from expansion of coverage or increases in productivity. For example, if a QI initiative results in timely and accurate billing for services or more productive service delivery, a public health agency might experience increases in revenue. The costs-saved outcome focuses on investments made by the public health agency in labor, resources, and overhead to achieve monetary returns. Finally, costs avoided captures future costs that are offset by current investments in efficiencies. These offsets might occur because of improved allocation of staff or current investments in automation.

Effectiveness outcomes include results associated with improved service or program delivery or improved implementation of organizational processes to achieve agency or program goals. The 8 effectiveness outcomes are increased customer or staff satisfaction; increased reach to a target population; dissemination of information, products, or evidence-based practices; quality enhancement of services or programs; quality enhancement of data systems; organizational design improvements; increased preventive behaviors; and decreased incidence or prevalence of disease. Each of these outcomes may be short or long term with respect to the time frame required to demonstrate improvements. They are intended to represent a range of potential improvements that are feasibly achieved by a broad array of public health organizations and within myriad different programs or service delivery settings.

The existing literature has emphasized the need to link QI initiatives to programmatic successes or increased equity in service delivery, such as increased reach to a target population, and health outcomes, such as increased preventive behaviors (or, alternatively captured in this outcome, reduced risk factors) and decreased incidence or prevalence of disease.^5,8,12 However, in the early stages of QI efforts, public health agencies may not yet be able to detect improvements in these outcomes. Therefore, we included a range of outcomes to highlight more immediate QI successes. Dissemination of information, products, or evidence-based practices tracks results of public health agencies’ efforts to share products with or provide other forms of technical assistance to their community or regional partners. Quality enhancement of services tracks standardization of services, adoption of evidence-based practices, and compliance with established policies with the goal of improved service delivery, and quality enhancement of data systems captures improvements in data systems’ accuracy, functionality, and standardization. Finally, improved effectiveness may result as public health organizations reorganize or adjust their service delivery models for more effective use of human resources. These changes are captured under organizational design improvements.

Within the framework, each outcome is defined independently for purposes of clarity and simplicity, recognizing that any given QI initiative may address multiple outcomes either within or across the constructs of efficiency and effectiveness. Also, public health agencies may identify other outcomes of interest. To increase the framework’s usability, the outcomes are accompanied by a series of steps to consider at the outset of any QI initiative. First, practitioners are asked to determine what they hope to achieve if their QI initiative is successful: increased efficiency, increased effectiveness, or both. On the basis of the response to this first question, practitioners can identify the specific outcome of interest. Any initiative may have a primary intended outcome as well as additional intended benefits or outcomes that should be considered. The framework provides a series of guiding questions for consideration when deciding on outcomes, notably, Is the outcome relevant? Does it reflect the intent of the initiative given the problem or opportunity being addressed? Is the outcome achievable given the available resources and the given time period? Is the outcome measurable? Are data sources available?

Quality Improvement Measurement Framework

The framework has been implemented in the field for 2 years. For the framework to be relevant and useful, it had to improve the consistency of measurement of efficiency and effectiveness outcomes while simultaneously acknowledging and respecting the diversity of public health agencies and their QI initiatives. To this end, the framework guides practitioners through an approach to developing measures that is both standardized and customizable to individual agency priorities.

Standardizing measurement of public health quality improvement outcomes.

Each QI initiative is unique to each jurisdiction’s needs and context. Therefore, the measurement approach uses a standard set of generic measures (Table 1) that address each of the framework’s key outcomes associated with efficiency and effectiveness. This approach allows each organization to tailor the measures to the aims of its specific programmatic, service-oriented, or process-oriented QI initiative and facilitates a consistent approach to measurement despite the wide array of QI efforts.

To ensure common interpretation and application of this generic measurement language, the framework incorporates additional guidance regarding the calculation of these measures and considerations for other contextual information. Specifically, for each outcome and associated measure, the framework includes (1) a definition of the intended outcome and further clarification of the measure itself, including sample measures; (2) specific information about what should be considered when establishing baseline and target values for the measure and what should be captured after implementation of the QI initiative; (3) guidance on how to calculate the measurement specifications, such as the numerator and denominator, start and stop time, or criteria to consider for qualitative measures; and (4) when applicable, additional information that may provide context to the measure itself.

Given that some public health QI initiatives are more conducive to quantitative measurement than others,^5,24 the framework includes a combination of quantitative and qualitative measures, depending on the outcome. For example, measures for increased customer or staff satisfaction would be quantitative, specifically, the percentage of customers who were satisfied or extremely satisfied with a service. For the time-saved outcome, the measure would represent start and stop times to calculate the average time taken to complete a service or activity. An example of an outcome with a qualitative measure is quality enhancement of services. For this outcome, a baseline may describe gaps in effectiveness resulting from variability in services, and the postimplementation value would reflect gains achieved because of standardization or policy implementation.

Implementation of the framework.

After the first 1.5 years of implementation among 73 NPHII awardees, 97.3% of awardees (71 of 73) submitted measures for at least 1 QI initiative. This yielded 693 measures for 357 QI initiatives because several of the initiatives addressed more than 1 outcome and therefore resulted in development of more than 1 measure. A variety of data sources informed measures, including but not limited to process maps, customer satisfaction surveys, vital records, programmatic data, and electronic health records.

The most commonly addressed outcomes were quality enhancement of services (18.2%; n = 126), time saved (17.7%; n = 123), and increased customer or staff satisfaction (11.4%; n = 79). The outcomes least frequently addressed were revenue generated from billable services (0.7%; n = 5), costs avoided (0.9%; n = 6), and costs saved (1.2%; n = 8).

NPHII awardees reported both quantitative and qualitative measures. Of all measures, 83% (n = 575) tracked quantifiable improvements, and 16.7% (n = 116) tracked improvements qualitatively. The remaining measures (0.3%; n = 2) were somewhat ambiguous and difficult to categorize. Quality enhancement of services had the highest percentage of qualitative measures (5.6%; n = 39), followed by quality enhancement of systems (4.2%; n = 29). Examples of quantitative and qualitative measures are provided in Table 2.

TABLE 2—

Examples of Quantitative and Qualitative Measures Using the Quality Improvement Measurement Framework

Associated Outcome	Example Measure
Quantitative
Efficiency
Time saved	No. of days from award letter to contract start date
Costs saved	Cost per unit of pharmaceuticals wasted in 4 Public Health Center pilot sites
Reduced steps in process	No. of steps in accessing the most recent data from (state-based) information technology system
Revenue generated from billable services	Revenue generated by reducing the no. of preventable denials for claims submitted for clinical service by health department
Costs avoided	Cost of processing applications by the Environment Unit (through move from paper to online processing)
Effectiveness
Customer and staff satisfaction	% of nursing staff satisfied or extremely satisfied with the protocols in their refugee clinic
Increased preventive behaviors	% of babies born at hospital to moms with diabetes who are exclusively breastfed in the hospital
Decreased incidence and prevalence	% of individuals exposed to syphilis who are designated as unable to locate
Dissemination of Information	% of participating programs that are submitting meaningful measures to the agency Dashboard report
Organizational design improvements	% of job descriptions across the division that align with standard (responsibilities) domains identified for each job class
Qualitative
Effectiveness
Quality enhancement, services	Uniform standard policy and procedures for disposal of sharps used in HIV testing will be in place within 6 mo of development
Quality enhancement, systems	Extent to which health departments’ databases are compliant with standards for collection of race, ethnicity, and gender data

DISCUSSION

This standardized framework represents 1 approach to operationalizing and defining measures for QI efficiency and effectiveness outcomes that can be applied in a broad array of public health contexts. The framework identifies measures relevant to a range of public health programs, services, and operational processes. Although initially originated as a framework for specific QI initiatives, the close linkage between QI, especially at the organization level, and performance management³⁸ allows it to be useful within, or considered a part of, performance management efforts. For example, these outcomes and measures may be used to assess changes related to agency priorities or captured within an agency’s performance management system. Whether used specifically in the context of discrete QI projects or embedded in broader performance management efforts, this framework provides a unique balance between standardization and customization through a focus on outcomes without prescribing specific processes to achieve them and generic measures that can be tailored to agency- or program-specific initiatives and contexts.

According to the Public Health Services and Systems Research national research agenda and recent literature on the science of QI, the field of public health QI has grown in both visibility and attention, presenting opportunities for innovative approaches to practice and research.^5,22,24–26 To advance the science and practice of public health QI, the field needs more studies that use valid and reliable instruments and draw conclusions from representative samples.⁵² The field can be advanced by establishing a standardized set of QI measures that can be used to support individual project aims as well as systemwide initiatives. This framework has the potential to advance the dialogue around these needs by (1) presenting a parsimonious measurement model for collecting data on QI efficiency and effectiveness outcomes at the program and agency levels; (2) testing the face validity of the framework through implementation in the field across a variety of state, local, tribal, and territorial health departments; and (3) identifying the types and frequency of QI approaches used to improve public health programs’ and services’ efficiency and effectiveness.

This framework is unique in its articulation of a standard set of outcomes and measures uniquely applicable to public health QI that are responsive to needs identified in the literature and reflect current public health practice. However, a review of resulting measures and other information on QI initiatives is critical to determine whether other core outcomes of public health QI need to be considered. Similarly, an analysis of timeframes required for achievement of various outcomes may help inform improvements to guide the application of various outcomes and expectations to achieve results. Additional analysis of data derived from implementation of the framework will further test the validity and reliability of the measurement constructs across varying QI initiatives, programs, and organizations, as well as build an understanding of how context affects its use. Further research can build on the measurement framework and explore how it may be used to understand the impact of QI across multiple contexts and over time. The framework is intended to be a living document that can expand as understanding of the science and practice of QI in public health progresses, ultimately contributing to the “so what” of public health QI.