Improving Outcome Accountability of Block Grants: Lessons Learned From the Preventive Health and Health Services Block Grant Evaluation

Abstract

The flexibility federal block grants provide recipients poses challenges for evaluation. These challenges include aggregating data on wide-ranging activities grant recipients implement and the outcomes they achieve. In 2014, we began designing an evaluation to address the challenges of assessing outcomes and to improve outcome accountability for the Preventive Health and Health Services Block Grant. Through the use of evaluability assessment methodology, review of existing data and the literature, and key informant interviews, we developed a measurement framework to assess outcomes resulting from recipients’ ability to use grant funds to meet their locally prioritized needs. We argue our evaluation approach demonstrates that block grants, and other similarly flexible programs, can be evaluated through appropriately designed measures. Our efforts challenge the idea that flexibility presents an insurmountable barrier to evaluation and outcome accountability for federal block grants.

The Challenges of Federal Block Grant Accountability

Block grants have been used as federal funding mechanisms since the mid-1960s. A block grant is a form of federal aid that authorizes state and local governments to conduct a wider range of activities, compared with categorical grants and cooperative agreements. Block grant recipients are given flexibility to use funds based on their own priorities and to design programs and allocate resources as they determine to be appropriate (Dilger & Boyd, 2014; U.S. Government Accountability Office [GAO], 1998). While this flexibility presents opportunities for how grant recipients administer their funds, it poses challenges for evaluators in documenting overall outcomes, given the diversity of activities implemented across recipients (Department of Health and Human Services [HHS], Office of Inspector General [OIG], 1995; Dilger & Boyd, 2014; U.S. GAO, 1995). A potential trade-off between flexibility and the ability to account for outcomes is highlighted in a question posed in a federal GAO (1998) report focused on the accountability of block grants and similar programs—“Can grant programs be designed to promote flexibility at the state or local level as in traditional block grants, yet still provide the information needed to ensure accountability and support federal policy decisions?” (p. 1). As staff tasked with evaluating important outcomes of an existing block grant, a pertinent question for us was whether an evaluation could be designed to overcome these challenges and provide data and information useful for outcome accountability purposes.

Accountability (in the broad sense of demonstrating results and effectiveness to the public) has two measurement components: (1) monitoring progress on agreed-on plans and (2) assessing outcomes resulting from the execution of the plans (Institute of Medicine [IOM], 2011). The first measurement component, or process accountability, is often addressed through the use of process monitoring, such as reviewing block grant recipient annual work plans and progress reports. The second measurement component, outcome accountability, focuses on assessing short-term, intermediate, or long-term results of program implementation (Derose et al., 2002; Porterfield et al., 2015). This article describes how we designed an evaluation to meet the outcome accountability challenges of the Preventive Health and Health Services (PHHS) Block Grant through an evaluation approach tailored to the realities of this funding mechanism. By using multiple evaluation methods—including evaluability assessment, funding allocation review, and key informant interviews—we were able to create a measurement framework that is rooted in the flexible nature of the grant while providing a means to evaluate across grant recipients. Our evaluation approach highlights opportunities for evaluators to improve outcome accountability for other types of programs that involve a wide range of activities and broad programmatic goals.

The PHHS Block Grant

The PHHS Block Grant was established by the Omnibus Budget Reconciliation Act in 1981 (U.S. GAO, 1984) and is administered by the Centers for Disease Control and Prevention’s (CDC) Center for State, Tribal, Local, and Territorial Support. In federal fiscal year 2019, the PHHS Block Grant received $160 million to support 61 recipients—50 states, the District of Columbia, two American Indian tribes, five U.S. territories, and three freely associated states. Recipients address public health needs that are priorities within their jurisdictions in collaboration with public health agencies and community-based organizations. Current legislation requires recipients to establish and engage an advisory committee and align their public health priorities to relevant Healthy People national health objectives (Preventive Health Amendments of 1992). More than 1,200 objectives represent 42 Healthy People 2020 health topic areas (www.healthypeople.gov), such as public health infrastructure, heart disease and stroke, immunization and infectious diseases, and injury and violence prevention (HHS, Office of Disease Prevention and Health Promotion, 2014).

Background on Accountability Efforts for the PHHS Block Grant

The flexibility that allows recipients to use the PHHS Block Grant for a wide range of programs and activities makes it challenging to measure grant performance and assess the grant’s effectiveness (CDC, 2011). Throughout the PHHS Block Grant’s history, process accountability efforts have focused primarily on monitoring adherence of recipient activities to established work plans. This monitoring function is facilitated through CDC’s “Block Grant Management Information System” (BGMIS; Office of Management and Budget [OMB] Control No. 0920–0106, expiration August 31, 2022). BGMIS is a web-based grant management system designed to capture data from recipient work plans each fiscal year, monitor progress related to work plan implementation, and collect annual reports (CDC, 2019). In addition, BGMIS supports the submission of success stories—narratives that highlight the accomplishments of individual recipients. While BGMIS enables fiscal and process accountability, data currently collected by the system have limited use for assessing outcomes across all recipients.

In 2014, the Association of State and Territorial Health Officials (ASTHO, 2014) board, members, and Affiliate Council proposed guiding principles for effective implementation of the PHHS Block Grant. One of those principles focused specifically on ensuring accountability and reporting of results. Also, in 2014, the U.S. Congress, the federal OMB, and the then Advisory Committee to the CDC Director charged CDC with evaluating the PHHS Block Grant to improve its outcome accountability. With this direction, our team set out to design an evaluation of the PHHS Block Grant.

Designing the PHHS Block Grant Evaluation—Arriving at a Focus on Flexibility

We used CDC’s Framework for Program Evaluation (CDC, 1999) as a guide to initiate our evaluation activities in 2014. CDC’s evaluation framework served as a tool to ensure that our focus on outcome accountability would be achieved through an evaluation approach that met CDC’s core evaluation standards of utility, feasibility, propriety, and accuracy (CDC, 2017). This was accomplished through the application of the six steps outlined in the framework—engage stakeholders, describe the program, focus the evaluation design, gather credible evidence, justify conclusions, and ensure use and share lessons learned.

Evaluability Assessment

To assist our team in the first three steps of CDC’s evaluation framework (engage stakeholders, describe the program, and focus the evaluation design), we undertook an evaluability assessment in partnership with ASTHO and ICF, a professional services firm. An evaluability assessment is a systematic process for describing the structure of a program and for analyzing the plausibility and feasibility of achieving its objectives; the suitability of the objectives for indepth evaluation; and their acceptability to program managers, policy makers, and program operators (Smith, 2005). Given our directive to evaluate the PHHS Block Grant, we used the evaluability assessment to identify where we could most feasibly and meaningfully initiate evaluation activities, taking into consideration the challenges of measuring and aggregating outcomes for a block grant. Key objectives of the evaluability assessment included (1) understanding and assessing the current state of the program in terms of program design, operations, monitoring and evaluation systems, and data quality; (2) describing the intended program theory or logic; and (3) identifying opportunities for useful evaluation and selection of a recommended evaluation approach.

To accomplish these objectives, we engaged a variety of PHHS Block Grant program stakeholders, first via an evaluability stakeholder work group and the convening of a Think Tank meeting with representatives from CDC’s national partners (e.g., ASTHO, State Chronic Disease Directors, National Association of County and City Health Officials) and, subsequently, convening an ongoing evaluation stakeholder work group composed of PHHS Block Grant coordinators within recipient agencies. We also reviewed the literature on evaluating block grants to understand how other agencies had approached this complex subject. As part of our review, we identified a 2012 GAO report focused on designing evaluations of federal programs (U.S. GAO, 2012). In the report, GAO outlined specific methodological challenges for program evaluation, including programs for which outcomes might not be comparable across recipients given the diversity of funded activities—a key characteristic of the PHHS Block Grant. Table 1 reflects select characteristics of federal programs and associated challenges identified in the 2012 GAO report that were specifically relevant to us as we designed the PHHS Block Grant evaluation. The table includes examples of evaluation approaches some federal agencies have used to meet the associated challenge that we also employed in designing the PHHS Block Grant evaluation.

Table 1.

Selected Methodological Challenges to Evaluation.

Characteristic of Federal Program	Challenge	Evaluation Approach
Complex programs and initiatives	1. Benefits of flexible grant programs are difficult to summarize	1a. Describe national variation in local approaches
		1b. Measure national improvements in common outputs or outcomes
Outcomes are difficult to measure	2. Lack of common outcome measures	2a. Collaborate with others on a common reporting format
		2b. Conduct a special survey to obtain nationwide data
	3. Desired outcomes are infrequently observed	3a. Measure effects on short-term or intermediate goals

Source: Adapted from U.S. Government Accountability Office (2012).

The evaluability assessment was a key step in addressing the GAO-identified challenge of summarizing the benefits of complex flexible grant programs. The evaluability assessment increased understanding of the program context and its design and operations. It also identified analytic limitations of the BGMIS system (e.g., the system was not configured to allow for ease in exporting or extracting aggregate recipient data). As a result of this process, we were able to create a logic model that describes the intended program theory (Figure 1), formulate overarching evaluation questions, and a recommended approach for evaluation (which is described below). The approach involved three distinct, yet related evaluation phases.

Figure 1.

Preventive Health and Health Services block grant logic model.

Implementing the Phased Evaluation

The phased evaluation is centered on two evaluation questions: (1) How does the PHHS Block Grant support recipients in addressing their jurisdictions’ prioritized public health needs related to Healthy People 2020 objectives? and (2) How does the PHHS Block Grant contribute toward the achievement of organizational, systems, and health-related outcomes? To address these questions and evaluate the grant effectively, we needed to find a way to mitigate the second challenge GAO outlined—the lack of common outcome measures.

Phase 1: Funding Allocation Data Review

In the initial phase of the evaluation, we focused on identifying how best to prioritize evaluation efforts, given the breadth and variability of the PHHS Block Grant activities. To do so, we reviewed BGMIS data on recipients’ allocation of funds in their work plans by Healthy People 2020 topic areas to gain an objective understanding of the types of activities funded by the grant.

Reviewing the funding allocation data for fiscal years 2015 and 2016 and across all recipients helped us describe the national variation in local approaches as articulated in the GAO report. For example, in both fiscal years 2015 and 2016, recipients allocated funding to 33 of the 42 Healthy People 2020 health topic areas plus two additional health topic areas, Emergency Medical Services and Rape/Attempted Rape Prevention (Table 2). While most of the health topic areas supported with PHHS Block Grant funds remained consistent from 2015 to 2016, the breadth of approaches recipients might take to address a locally identified public health need within any given topic area is vast. This type of diversity poses a challenge for aggregating data and measuring outcomes across recipients over time. We knew we need to develop a measurement approach that accounted for this diversity.

Table 2.

FY 2015 and FY 2016 Health Topic Areas Funded by the Preventive Health and Health Services Block Grant.

Healthy People 2020 Health Topic Areaa	% of Total FY 2015 Fundingb	% of Total FY 2016 Fundingb
1. Public Health Infrastructure	26.2	27.8
2. Educational and Community-Based Programs	21.0	20.2
3. Injury and Violence Prevention	8.3	7.5
4. Nutrition and Weight Status	7.9	8.1
5. Heart Disease and Stroke	5.5	4.7
6. Environmental Health	3.3	2.1
7. Immunization and Infectious Diseases	3.2	3.3
8. Emergency Medical Servicesc	3.2	3.1
9. Oral Health	3.0	3.0
10. Physical Activity	2.7	2.7
11. Diabetes	2.5	3.0
12. Access to Health Services	2.0	1.6
13. Food Safety	1.8	1.9
14. Health Communication and Health Information Technology	1.5	1.8
15. Maternal, Infant, and Child Health	1.1	1.2
16. Respiratory Diseases	0.9	1.0
17. Sexually Transmitted Diseases	0.9	1.1
18. Tobacco Use	0.8	0.9
19. Rape or Attempted Rapec	0.6	0.8
20. Cancer	0.7	0.8
21. Hearing and Other Sensory or Communication Disorders	0.5	0.6
22. Occupational Safety and Health	0.5	0.5
23. HIV	0.4	0.5
24. Mental Health and Mental Disorders	0.3	0.4
25. Preparedness	0.3	0.3
26. Early and Middle Childhood	0.3	0.3
27. Substance Abuse	0.2	0.3
28. Family Planning	0.2	0.3
29. Dementias, Including Alzheimer’s Disease	0.1	0.1
30. Older Adults	0.1	0.1
31. Arthritis, Osteoporosis, and Chronic Back Conditions	0.1	0.1
32. Disability and Health	—	0.1
33. Healthcare-Associated Infectionsd	0.1	0.0
34. Visiond	0.0	—
Total	100.0	100.0

^aBolded topic areas appear in the top 10 health topic areas in FY 2015 and/or FY 2016. Environmental health is in the top 10 in FY 2015 and diabetes in FY 2016.

^bRecipients are mandated by legislation to set aside a portion of grant funding for sexual assault prevention, with the remaining funds to be allocated at the recipients’ discretion.

^cEmergency Medical Services and Rape or Attempted Rape are not health topic areas in Healthy People 2020 but are designated topic areas where recipients may allocate funds.

^dLess than 0.1% of total FY 2015 funding was allocated by recipients to vision and less than 0.1% of total FY 2016 funding was allocated by recipients to Healthcare-Associated Infections.

The review of funding allocation data also provided us with the top-funded topic areas across all recipients. In fiscal years 2015 and 2016, the top 10 Healthy People 2020 topic areas funded included public health infrastructure, educational and community-based programs, injury and violence prevention, nutrition and weight status, heart disease and stroke, environmental health, and immunization and infectious diseases, among others (Table 2). Identifying those top-funded topic areas across recipients allowed us to prioritize evaluation efforts to topic areas where the largest amount of funds were directed and focus measurement on areas that would be relevant to the majority of grant recipients.

Phase 2: Key Informant Interviews

In the second phase of the evaluation, exploratory key informant interviews were conducted to identify, from the perspective of funded health departments, the unique benefits or contributions of the PHHS Block Grant to addressing diverse public health needs and improving health outcomes in recipient jurisdictions. Nine PHHS Block Grant coordinators were selected to participate in the interviews (see Appendix for interview questions). These interviews provided strategic insight and perspective from another key stakeholder group that was not available through the grant’s existing monitoring data. All nine interviewees identified the flexibility of the grant to address their priorities as the primary difference and distinct value of the grant from categorical funding. Interviewees relayed that this flexibility allowed them to implement public health efforts that were unfunded or underfunded, develop innovative strategies to address local needs, respond to emerging public health issues, support organizational and public health infrastructure, develop crosscutting programs, and facilitate partnerships with internal and external partners.

Coupled with the funding allocation data, the qualitative study built our knowledge of the grant and made clear that the next phase of the evaluation needed to address the challenges presented by flexibility and articulate the collective results of the grant.

Phase 3: PHHS Block Grant Measurement Framework—Giving Structure to Flexibility

We designed the PHHS Block Grant Measurement Framework (“the measurement framework”) based on the top topic areas identified from the funding allocation data and the recipient perspectives from the key informant interviews on the importance of the grant’s flexibility. By drawing on the findings from these two phases of the evaluation, we were able to craft a measurement framework to address key challenges of block grant outcome accountability in quantifying selected outputs and outcomes resulting from recipients’ ability to use grant funds to meet their locally prioritized needs. In doing so, the measurement framework helped us further refine our evaluation design in order to address our evaluation questions and provide insight into the short-term, intermediate, and long-term outcomes from our logic model (CDC, 2018b). A representation of key components of the measurement framework is presented in Figure 2.

Figure 2.

Components of the Preventive Health and Health Services (PHHS) Block Grant measurement framework.

In the measurement framework, we positioned flexibility as a foundational element. Through the interview responses, we were able to operationalize the broad concept of flexibility into more concrete constructs included in the second and third nested layers of the framework. We shared the draft measurement framework with our internal stakeholders at CDC and our grant recipient evaluation work group. Both groups provided feedback that shaped the version presented in Figure 2. This collaboration with stakeholders promoted buy-in on the evaluation as well as created important touch points with the recipients to keep the evaluation grounded in the program. The measurement framework also provided a common reporting format for crosscutting outcomes and aligns with a suggested approach in the 2012 GAO report to address outcomes that are difficult to measure.

While designed with the most frequently funded Healthy People 2020 topic areas in mind, the measurement framework is intended to apply to recipient activities, regardless of how funds are invested or which Healthy People 2020 objectives are selected. We designed the measurement framework so that recipients could see alignment between their work and the measurement framework; however, depending on recipients’ priorities and activities, not every aspect of the measurement framework will necessarily be relevant to every recipient in any given reporting period. As such, the measurement framework captures a cross-sectional view of recipients’ achievements and complements other valuable data and information about the grant, such as the process monitoring data reported in BGMIS and individual success stories shared by recipients. The measurement framework consists of three components—flexibility, use of funds, and results.

Measurement framework component

Flexibility

The core of the measurement framework is the flexibility of the PHHS Block Grant. Recipients determine which jurisdictional public health needs to prioritize within the parameters of Healthy People 2020 objectives and select strategies to address those needs for each 12-month funding cycle. Grant recipients are able to address identified public health needs for which other categorical types of funding are insufficient, unavailable, or too directive as to how program funds can be used. The current version of the measurement framework operationalizes this flexibility through the second component—the use of funds.

Use of funds

The key informant interviews and funding allocation data revealed the different ways recipients use grant funds to implement public health efforts to address prioritized public health needs. The “Use of Funds” component of the measurement framework shows how flexibility in the use of PHHS Block Grant funds helps attain the results identified in the measurement framework. This component outlines four ways recipients may use PHHS Block Grant funds: (1) initiate new public health efforts, (2) maintain existing public health efforts, (3) enhance or expand existing public health efforts, and (4) sustain or restore efforts that have experienced a partial or complete loss in funding and are at risk for discontinuation. Table 3 provides examples for each of these uses of funds.

Table 3.

Examples of the Use of Preventive Health and Health Services (PHHS) Block Grant Funds.

Use of Funds	Examples
Initiate new public health efforts	• Testing new or innovative approaches to addressing needs (e.g., implementing a novel intervention to address the opioid epidemic) • Implementing programs, services, or activities that may have been conducted elsewhere but are new to the jurisdiction
Maintain existing public health efforts	• Providing ongoing support to longer term efforts (e.g., supporting oversampling of a subpopulation in a surveillance program) • Ensuring consistency and continuation of efforts
Enhance or expand existing public health efforts	• Fully implementing or scaling up pilots or smaller efforts • Establishing new or expanding existing partnerships or increasing integration across categorical programs • Advancing existing work (e.g., updating plans or assessments)
Sustain or restore public health effortsa,b	• Ensuring a program continues until other funding sources are identified (e.g., stopgap funding) • Institutionalizing public health efforts (e.g., restoring ongoing funding in the wake of funding loss)

^aPHHS Block Grant funds may not be used to supplant state or local funds.

^bOnce a public health effort is restored, it would move into the “maintain” category in subsequent years.

Results

As recommended by the GAO, outcomes should be defined in terms of the program’s intended purpose (U.S. GAO, 1998). In this case, the program is a block grant with the purpose of providing health departments with flexible funding to address jurisdiction-specific public health priorities. Through our review of the funding allocation data and themes from the interviews, we identified three crosscutting outcomes of health department performance and public health practice which result from using PHHS Block Grant funds. These crosscutting outcomes, or results, reflect the purpose of the PHHS Block Grant and, therefore, are not framed as traditional public health outcomes of categorical programming, such as reduced prevalence of risk factors or reduced disease rates. The results in the measurement framework are the following:

1. Public health infrastructure improved: Public health infrastructure includes the organizational capacity (i.e., the systems, workforce, partnerships, and resources) that enables health departments to perform their core functions and provide essential services. Improvements to infrastructure may occur within the recipient health department, either department-wide or within a specific program, or across the recipient jurisdiction’s public health system. There are several aspects of improving public health infrastructure, such as improving information systems capacity, quality, and communications strategies; strengthening the workforce; addressing public health standards; and supporting partnerships. For example, recipients use PHHS Block Grant funds to improve agency information systems. These systems include surveillance (e.g., Behavioral Risk Factor Surveillance System, Pregnancy Risk Assessment Monitoring System), administrative and business (e.g., performance management systems), financial management (e.g., state grants management systems), and program administration (e.g., case management systems). Recipients also use grant funds to employ established quality improvement methods (e.g., Lean, Plan Do Study Act) to improve the efficiency and effectiveness of health department operations, programs, and services. These types of improvements may take a variety of forms such as saving time on business operations, avoiding costs, making quality enhancements, and implementing organizational design improvements (McLees et al., 2015).

2. Emerging needs addressed: Emerging needs are public health issues that are beginning to present themselves as problems within the recipient’s jurisdiction. Emerging needs might be newly arising problems, reemerging problems, or existing problems that have developed new characteristics (e.g., that affect new populations or geographic areas). Public health emergencies, or unexpected natural or manmade events that cause an immediate risk to the public’s health, are also considered emerging needs. Emerging needs might occur in response to external factors or to changing priorities within a jurisdiction. Recipients have used PHHS Block Grant funds to address emerging public health needs such as substance abuse (e.g., opioid and prescription drug abuse), infectious diseases (e.g., viral infections, foodborne illnesses), injury prevention, and workforce development.

3. Evidence-based public health practiced: Evidence-based public health practice involves implementing interventions that are known to be effective. This is an indicator of recipient use of an evidence-based intervention, not a measure of intervention effectiveness. It also includes both building and using evidence (i.e., data and information) to

   • define the public health need to be addressed (e.g., surveillance data),
   • describe the effectiveness of specific interventions with respect to outcomes (e.g., evaluation data), and
   • describe how to effectively implement interventions with respect to relevant contextual factors such as setting, population, and social norms (e.g., translational research data; Brownson et al., 2009).

Examples of the types of evidence-based public health practice recipients support with PHHS Block Grant funds include conducting health assessments to determine public health needs, developing health improvement plans to prioritize public health needs and guide public health action, implementing evidence-based interventions for a wide array of Healthy People 2020 health topic areas, and developing practice-based evidence by assessing the effectiveness of public health interventions with weak or no supporting evidence.

While the measurement framework results do not capture all possible outcomes of the grant, these three crosscutting outcomes had the broadest applicability to the top-funded activities; align with the short-term, intermediate, and long-term logic model outcomes; and reflect important contributions of the grant from a recipient perspective. We developed framework measures for each of these results.

Measurement framework measures

The measurement framework includes four measures that assess specific aspects of the three results and enable the standardization of data collection on recipient achievements, thereby improving the outcome accountability of the grant. The measures focus on the aspects of each result that were considered by stakeholders as most important, relevant, measurable, and feasible for data collection. Table 4 includes a list of the measures.

Table 4.

Preventive Health and Health Services (PHHS) Block Grant Framework Measures.

Result	Measure
1. Public health infrastructure improved	Information Systems Capacity Improved
	1.1. Number of state, territorial, tribal, and local health departments whose capacity to collect or enhance data that provide information of public health importance was improved or maintained through the use of PHHS Block Grant funds
	Quality Improved
	1.2. Number of state, territorial, tribal, and local health departments in which the efficiency or effectiveness of operations, programs, or services was improved through the use of PHHS Block Grant funds
2. Emerging needs addressed	Emerging Public Health Needs Addressed
	2.1. Number of emerging public health needs that were addressed through the use of PHHS Block Grant funds
3. Evidence-based public health practiced	Evidence-Based Public Health Interventions Implemented
	3.1. Number of evidence-based public health interventions implemented through the use of PHHS Block Grant funds

In developing the measures, we ensured that they are relevant to recipient objectives and activities, lend themselves to accurate measurement, and allow for aggregating data across recipients. The framework measures are not intended to be used to limit or direct recipient activities to address public health priorities within their jurisdiction. Where feasible, the measures capture results both within the recipient health department and among local and tribal health departments indirectly receiving PHHS Block Grant support. The reach of PHHS Block Grant activities to local and tribal health departments, as relevant, and the results of those activities are important aspects for us to measure in an effort to account for outcomes the PHHS Block Grant contributes to across the public health system.

For each measure, we asked recipients to provide us with descriptive contextual information. These data provide a fuller picture of the result or achievement beyond a count. For example, recipients provided us information regarding the strength of the evidence for the evidence-based interventions they were implementing, where and how quality was improved (operations vs. services, etc.), and where and how information systems capacity was improved. These contextual data provide important but limited detail. In order to evaluate the quality of evidence or other aspects of the outcomes, additional investigation is needed. Table 5 summarizes the contextual data we gather for each result measure.

Table 5.

Preventive Health and Health Services (PHHS) Block Grant Framework Measures and Associated Contextual Data.

Measure	Contextual Data
Information Systems Capacity Improved 1.1. Number of state, territorial, tribal, and local health departments whose capacity to collect or enhance data that provide information of public health importance was improved or maintained through the use of PHHS Block Grant funds	• Type of health department that developed, improved, or maintained information system • Name and type of information system (e.g., surveillance, administrative, vital events database) • How funds were used to support the information system (e.g., initiated, maintained, enhanced, sustained) • Whether local or tribal health departments used or had access to the information system • Type of health department that used quality improvement methods to improve an operation, program, or service • Name of operation, program, or service for which efficiency or effectiveness was improved
Quality Improved 1.2. Number of state, territorial, tribal, and local health departments in which the efficiency or effectiveness of operations, programs, or services was improved through the use of PHHS Block Grant funds	• Type of efficiency and/or effectiveness improvement (e.g., time saved, reduced number of steps, costs saved, quality enhancements) • How funds were used to support the efficiency and effectiveness improvements • One descriptive example of an efficiency and/or effectiveness improvement including specific issue being addressed, quality improvement method used, and brief description of efficiency and/or effectiveness achieved
Emerging Public Health Needs Addressed 2.1. Number of emerging public health needs that were addressed through the use of PHHS Block Grant funds	• Name/title of emerging public health need • Whether the need is newly prioritized or newly developing • How the emerging need was identified • Approximate size of the population potentially affected by the emerging need • How funds were used to support the effort to address the emerging need
Evidence-Based Public Health Interventions Implemented 3.1. Number of evidence-based public health interventions implemented through the use of PHHS Block Grant funds	• Name/title of public health intervention • Level of evidence that supports the public health intervention (e.g., rigorous, strong, moderate, weak, no evidence)a • Whether the intervention was untested, new, or innovative and, if so, data were collected for the purposes of determining the intervention’s effectiveness • Healthy People 2020 health topic area(s) the intervention addresses • How funds were used to support implementation of the intervention

^aHealthy People 2020. Evidence-Based Resources (accessed August 16, 2019, www.healthypeople.gov/2020/Implement/EBR-glossary#selection-criteria).

Data for the measures were collected for the first time in 2017 via an online voluntary questionnaire (OMB Control No. 0920–0879, expiration January 31, 2021). Of the 61 recipients, 57 (93%) responded to the questionnaire. Ninety-one percent of recipients used PHHS Block Grant funds to achieve results associated with at least one measure. The majority of recipients reported data on each of the four measures (range = 67–84%; Table 6). The high percentages of recipients reporting on the measures indicate that the measures are relevant and capture achievements related to the results in the framework for most grant recipients (CDC, 2018a).

Table 6.

2017 Preventive Health and Health Services (PHHS) Block Grant Framework Measures Assessment Recipient Reporting by Measure.

PHHS Block Grant Measure	N	%
1.1. Information Systems Capacity Improved	42	73.7
1.2. Quality Improved	38	66.7
2.1. Emerging Public Health Needs Addressed	39	68.4
3.1. Evidence-Based Public Health Interventions Implementeda	47	83.9
Reported on at least one measure	52	91.2
Reported on all four measures	25	43.9
Did not report on any measure	4	7.0

Note. N = 57.

^aOne recipient excluded because of missing data.

Conclusion

The evaluation provides data for communicating about and reporting on what the PHHS Block Grant is achieving as a whole. Stakeholders at all levels—federal, state, tribal, territorial, and local/community levels—can use the information to inform future grant activity and resource strategies. This conceptual innovation could be applied to other block grants or programs with broad flexibility by varying the results and measures to align with the subject of the evaluation. This use of a purposeful, iterative process to designing an evaluation can help evaluators discover creative and useful ways to overcome evaluation design challenges focused on outcome accountability.

Overcoming the evaluation challenge of aggregating outcomes across recipients, when the grant affords a high level of flexibility in the use of awarded funds, is critical to demonstrating outcome accountability for federal block grant funds. Our use of a grounded understanding of the program, based on data and stakeholder input, strengthened our ability to design a realistic, feasible, and relevant evaluation for the PHHS Block Grant that focused on what recipients are achieving.

As a key component of the evaluation, the PHHS Block Grant Measurement Framework is an innovative approach to assessing selected outputs and outcomes resulting from recipients’ use of grant funds. It overcomes challenges associated with evaluating the overall merit of the block grant by acknowledging flexibility as a core component of the grant and balancing how recipients can achieve outcomes (through the use of funds component) with what those outcomes are (the results achieved component). By working with stakeholders and employing evaluability assessment tools to address identified challenges in articulating the benefits of flexible grant programs, we were able to identify common themes across the ways that funds are used by recipients, the types of public health needs they address, and the outcomes linked to their activities.

The key aspect of the measurement framework—and the most critical component for us to address in responding to the call for increased outcome accountability—are the results. The results in the measurement framework are intentionally broad and crosscutting to allow for the variety of ways public health infrastructure needs might manifest in a given jurisdiction; the variety of diseases, pathogens, outbreaks, and so on that might become an emerging need in a given locale; and the variety of evidence-based interventions that are employed to address public health priorities. How recipients are achieving the outcomes (process accountability) describes the variability in the use of funds as well as the existence of other funding at the recipient level, while what the recipients are achieving (outcome accountability) is the focus of the evaluation measurement framework. The measurement framework identifies four measures that are relevant to a broad range of public health practices, services, and organizational operations and uniquely focused on assessing results yielded from the grant. As such, the measurement framework enables standardized data collection and aggregation of leading indicators of public health agency and system performance, acknowledging the role of public health practice in achieving health outcomes and the importance of measuring relevant, program-specific outcomes or results of planned activities (DeGroff et al., 2010; IOM, 2011).

While it was our intention to design a measurement framework that assessed select outcomes of the grant that were considered most important, relevant, measurable, and feasible by our stakeholders, there are limitations to this approach. One limitation is that the measurement framework does not capture all possible results from recipients’ use of the funds nor is the intent to report individual recipient achievements. Our evaluation questions focus on how, in the aggregate, the grant supports recipients in addressing their prioritized public health needs as well as how the grant contributes to the achievement of organizational, systems, and health-related outcomes. However, there will likely be future refinements to the measurement framework. For example, if future funding allocation data show recipient investments are targeted in other health topic areas, we might identify and measure additional (or different) outcomes. Additionally, these data will need to be complemented by other recipient information, such as success stories and other process accountability data, to demonstrate individual recipient achievements.

Appendix

PHHS Block Grant Evaluation: Exploratory Key Informant Interview Questions

Q1. I would like to start by learning more about your level of engagement with the Block Grant. Please tell me a bit about your role as the Block Grant coordinator, including where you sit in your organization and your key responsibilities.

Probes:

• As the Block Grant Coordinator, do you direct activities or do you coordinate reporting from individual programs? Please explain.

• Prior to your current role, have you worked in a different position that was supported by the Block Grant? If so, what was the nature of that position and how long were you in that role?

Q2. How does the Block Grant aid your organization in addressing public health needs?

Probes:

• What’s unique about the Block Grant in supporting your organization’s efforts to address these needs?

• How important, if at all, is the flexibility of the Block Grant in supporting your organization’s activities?

• How has the flexibility of the Block Grant supported your organization’s activities as they relate to the public health needs? Please provide specific examples.

Q3. Is this type of noncategorical funding important? Why or why not?

Probes:

• In what ways, if any, does the Block Grant allow your organization to address program needs differently than through categorical funding?

• What does the Block Grant allow your organization to do that other federal funding sources do not allow?

• Why do you use the Block Grant for these organizational or program needs instead of a different funding source?

Q4. (Reviewing the roles that exist in BGMIS) When you select (insert Block Grant role [see bullet list]), what does it mean to you?

• Rapid response

• Start-up

• Supplemental

• No other existing federal or state funding

Probe:

• What are other roles that you see Block Grant funding playing in your organization that are not currently being captured in BGMIS, if any?

Q5. If Block Grant funding were eliminated, what effect would that have on your organization’s ability to address public health needs?

Probe:

• Specifically, what would your organization not be able to do by some other means such as categorical funding?

Q6. Are there any additional comments you would like to share regarding the value of the Block Grant that we have not touched upon today?