Measuring the Value of Public Health Systems: The Disconnect Between Health Economists and Public Health Practitioners

Abstract

We investigated ways of defining and measuring the value of services provided by governmental public health systems. Our data sources included literature syntheses and qualitative interviews of public health professionals. Our examination of the health economic literature revealed growing attempts to measure value of public health services explicitly, but few studies have addressed systems or infrastructure. Interview responses demonstrated no consensus on metrics and no connection to the academic literature. Key challenges for practitioners include developing rigorous, data-driven methods and skilled staff; being politically willing to base allocation decisions on economic evaluation; and developing metrics to capture “intangibles” (e.g., social justice and reassurance value). Academic researchers evaluating the economics of public health investments should increase focus on the working needs of public health professionals.

The value of governmental public health systems may seem obvious in light of progress in public health over the past century. The reality of chronic underfunding of these systems suggests that the general public is unaware of public health's value. It is thus essential that governmental public health systems demonstrate measurable contributions to improving the population's health. On a conceptual and practical level, however, measuring the payback from public health spending is a challenge. As public health officials battle for resources in constrained fiscal environments, the manner in which they measure and communicate the value of programs—both internally in budget discussions and externally to the public—is critical.

We examined how researchers and public health practitioners have defined and measured the value of services provided by governmental public health systems. First, we discuss the health economics literature on value measurement in public health. Next, we present the results of qualitative interviews we conducted with leading public health officials and practitioners. Finally, we examine ways to bridge the gap between economists and practitioners and discuss opportunities for the future.

METHODS

Health Economics Literature Review

Researchers have attempted to measure the rate of return on investments in public health programs in various ways, focusing on the costs and benefits, much as a financial analyst would calculate the rate of return for competing alternatives in a portfolio. Some researchers have used cost–benefit analysis, in which an analyst estimates in monetary terms the net social benefit of a program or intervention as the incremental benefit of a program minus the incremental cost. In cost–benefit analyses, analysts quantify health benefits with either a “human capital” approach (measuring the value of reduced health as the lost earnings of affected individuals) or a “willingness to pay” approach (assessing through market data or surveys what people are willing to pay for specific health benefits). Other researchers have used cost-effectiveness analysis, in which interventions are measured in net cost per unit of health gained (e.g., life-years gained). In cost—utility analysis, health gains are expressed in quality-adjusted life-years (QALYs) to incorporate both prolongation and quality of life. Not without their challenges, QALYs have been recommended and widely used by experts in the field in recent years, because they capture in a single measure gains from both reduced morbidity and reduced mortality, and they incorporate the value or preferences people have for different outcomes.¹

We searched the health economics literature broadly to explore how the value of investments in public health has been quantified with these metrics and in what contexts. First, we searched the PubMed database for public health interventions by searching for the phrases “public health” and “cost-effectiveness analysis.” Because that search yielded tens of thousands of articles, many of which mentioned formal value measurement only in passing, we performed a more targeted search on “public health” and “valuation.” That search yielded approximately 100 articles, including key overviews,^2–5 methodological papers, and applications to diverse areas of public health. To supplement this review, we searched the Tufts Medical Center Cost-Effectiveness Analysis Registry, a comprehensive database of original cost–utility analyses. Our search focused on US cost–utility analyses devoted to public health interventions.⁶

Qualitative Interviews With Public Health Practitioners

We complemented the literature reviews with an exploration of public health practitioners’ perspectives on defining and measuring the value of public health services. Using a semistructured interview protocol, we conducted a series of interviews with leaders of national public health organizations, state and local public health practitioners, academics, and elected officials (such as those on local boards of health). We asked respondents to define the component parts of public health service valuation and to identify what the metrics of valuation should be, what methodologies they use to measure value, and what data they collect. We interviewed 46 respondents: 24 from local health departments; 7 from state health agencies; 8 representing national public health organizations; 4 academics; and 3 members of local boards of health. Everyone we contacted agreed to be interviewed. We promised confidentiality to all respondents.

P. D. J. conducted all of the interviews and took detailed notes (the interviews were not audio or video recorded). Most of the interviews were in-person, one-on-one interviews. Two interviews were conducted in small focus groups (5 and 10 respondents); 3 interviews were with 2 respondents at a time; and 1 respondent was interviewed over the telephone. A research assistant coded the interview notes, which the researcher reviewed for accuracy. The researcher identified common themes and key differences across the interviews.

We have synthesized the interview data and documentary evidence to portray the ways in which public health practitioners and officials think about defining and measuring the value of public health services.

RESULTS

Health Economics Literature Review

Our literature review uncovered various methods to value public health interventions. Researchers have used human capital approaches to value the benefits of a diverse array of programs, from perinatal screening for group B streptococci⁷ to preventive intervention for HIV.⁸ They have employed willingness-to-pay studies to focus on global health programs such as mosquito netting⁹ and ivermectin distribution in Nigeria¹⁰; on prevention efforts of adverse drug events¹¹; on diabetes risk reduction¹²; and on environmental health in cardiorespiratory morbidity from air pollution.¹³

Researchers have also published numerous cost-effectiveness analyses in which benefits are measured in physical or natural units, such as cases of disease avoided or life-years gained. Examples include studies of the cost per additional influenza vaccination in the elderly,¹⁴ the cost per case prevented of Chlamydia trachomatis,¹⁵ and cost per death averted in newborn hemoglobinopathy screening within state health systems.¹⁶ Cost-effectiveness analysts have examined the cost per life-year gained for various screening programs (i.e., for colorectal cancer,¹⁷ cervical cancer, and sickle cell disease) as well as of immunization initiatives (i.e., for hepatitis B¹⁸ and pneumococcal pneumonia).¹⁹

Our search of the Cost-Effectiveness Analysis Registry revealed 45 cost–utility analyses applied to a range of public health programs, including those devoted to screening and surveillance, immunization, regulatory and education policy, care delivery, health behavior, and injury prevention (Table 1). In a few cases, cost–utility analyses have addressed regulatory or educational policy interventions, such as a US Food and Drug Administration regulation for folic acid supplementation to enriched grain products and a national policy for tobacco education. Other studies have assessed a policy of shifting nicotine replacement drugs to over-the-counter status and regulating cell phone use in cars. A few cost–utility analyses have assessed health behavior interventions, including community-based HIV prevention programs focusing on education, counseling, cognitive-behavioral treatment, and condom distribution. Many of the public health interventions analyzed saved on costs or provided relatively good value (i.e., had relatively low cost-effectiveness ratios), even if they increased societal costs, although the ratios revealed a wide range of values. For example, studies have found that, among men aged 60 to 64 years, one-time colonoscopic screening for colorectal cancer versus no screening saves on costs and increases QALYs²⁰; screening for diabetes mellitus versus no systematic diabetes mellitus screening (i.e., the usual practice) in all individuals 65 years and older has a cost-effectiveness ratio of $680 000 per QALY gained.²³

TABLE 1.

Types of Interventions Studied in US Public Health Cost-Utility Analyses, 1976–2003

Disease	Intervention	$/QALYab	Author
Screening
Cancer	One-time colonoscopic screening for colorectal cancer	Cost-savingc	Ness et al.20
	Continued Pap and HPV testing to screen for cervical cancer into very old age	80 000	Mandelblatt et al.21
Diabetes	Diabetic retinopathy screening in patients with type 2 diabetes mellitus	19 000	Vijan et al.22
	Screening for type 2 diabetes in individuals ≥ 25 years old	67 000	CDC Diabetes Cost-Effectiveness Study Group23
Infection	Genetic screening for prevention of rheumatic fever	8 500	King et al.24
Kidney	Screening for proteinuria	19 000	Boulware et al.25
Pre- and postnatal	Universal newborn screening by tandem mass spectrometry for medium-chain Acyl-CoA dehydrogenase deficiency	5 700	Venditti et al.26
	Universal newborn tandem mass spectrometry	6 100	Schoen et al.27
	Screening for cystic fibrosis carriers	9 500	Rowley et al.28
	Newborn tandem mass spectrometry for acidemia disorders	15 000	Insinga et al.29
Immunization
Blood-borne illnesses, STDs	Hepatitis A/B immunization	Cost-saving	Jacobs et al.30
	Hepatitis A/B vaccination vs. hepatitis B vaccination	13 000	Jacobs and Meyerhoff31
	Universal vaccination for HPV	23 000	Sanders and Taira32
	Hepatitis A vaccination	55 000	Arguedas et al.33
Respiratory and other infections	Haemophilus influenzae type b vaccination	Cost-saving	Zhou et al.34
	Hypothetical respiratory syncytial virus vaccination	6 100	Gessner35
	Increasing measles immunization rates	52 000	Zwanziger et al.36
	Pneumococcal vaccination	56 000	Pepper and Owens37
	Vaccination against invasive pneumococcal disease	4 100	Sisk et al.38
Regulatory and education policy
Multiple	Vitamin supplementation to lower plasma homocysteine levels	1 000	Tice et al.39
	Switching smoking cessation drugs to over-the-counter status	16 000	Keeler et al.40
	Intensive national school-based antitobacco education	20 000	Tengs et al.41
	Restrictions on the use of cell phones while driving	75 000	Cohen and Graham42
	Switching to use of emission-controlled urban transit buses	270 000	Cohen et al.43
	Regulations against using a cellular telephone while driving	350 000	Redelmeier and Weinstein44
Care delivery
	Resuscitation with publicly accessible automated external defibrillators	30 000	Cram et al.45
	Public access defibrillation by police	32 000	Nichol et al.46
	Training program for automated external defibrillators on aircrafts	36 000	Groeneveld et al.47
	Rapid defibrillation by targeted nontraditional responders	55 000	Nichol et al.48
HIV	State AIDS drug assistance programs in Oklahoma vs. Mississippi	18 000	Johri et al.49
	HIV postexposure prophylaxis according to US Public Health Service guidelines	91 000	Scheid et al.50
HIV and STD health behavior
	Condom distribution	Cost-saving	Bedimo et al.51
	HIV risk reduction counseling and education	7 500	Tao and Remafedi52
	Intervention on sexual behavior and condom use	37 000	Chesson et al.53
	HIV cognitive-behavioral risk reduction intervention	64 000	Pinkerton et al.54
Injury prevention
Injury	Safety-belt law	Cost-saving	Zaloshnja et al.55
	Safety-belt law	40d	Zaloshnja et al.56
	Air bags in cars	24 000	Graham et al.57
	Hip protectors for women	Cost-saving	Segui-Gomez58
	Drowning prevention program	Cost-saving	Zaloshnja et al.55
	Streetlight installation	Cost-saving	Zaloshnja et al.55,56
	Livestock control project	Cost-saving or fewer QALYs	Zaloshnja et al.55,56
	Suicide prevention program	460	Zaloshnja et al.55
Blood testing
Multiple	HIV antibody testing of donated blood	Cost-saving	AuBuchon et al.59
	Alanine aminotransferase testing of donated blood	3 600	Busch et al.60
	Solvent-detergent treatment of fresh-frozen plasma for transfusion	289 000	AuBuchon and Birkmeyer61
Surveillance
	Tuberculosis skin testing and treatment	Cost-saving	Khan et al.62
	Surveillance of cancer risk in Barrett's esophagus	120 000	Provenzale et al.63
Other health
	Donor heart transplantation	31 000	Mendeloff64
	Donor liver transplantation	35 000	Mendeloff64

Note. QALY = quality-adjusted life-year; HPV = human papillomavirus; STD = sexually transmitted disease.

Source. Center for the Evaluation of Value and Risk in Health.⁶

^aAll values presented in 2003 US dollars. “$/QALY” measures the cost-effectiveness of an intervention by comparing it with an alternative intervention via a ratio of incremental costs over incremental quality-adjusted life years gained because of an intervention.

^bThe cost-effectiveness ratios listed are point-estimate values from original articles dating 1976–2003 included in the Tufts Medical Center Cost-Effectiveness Analysis Registry. The cost-effectiveness ratios will vary according to the precise strategies, target populations, and perspective used. Additional data on the cost-effectiveness ratios associated with public health can be found at: http://www.cearegistry.org.

^cCost-saving means the intervention saves money and increases QALYs.

^dDecreases costs and decreases QALYs.

Interview Results

Potential metrics.

Respondents in our interviews were almost unanimous in agreeing that defining and measuring value is critical to generating public support for public health services. In defining value, respondents focused on what public health achieves for the community and on what it prevents or helps a community avoid. A typical response was that value avoids harm and is inherent to the public health mission. Many found it difficult to offer a concise definition, however, and relied on vague, almost tautological depictions.

Our interviews did not reveal a consensus on what value metrics to use for valuing public health systems or interventions. The leaders of national public health organizations in our sample stressed the need to develop data-driven methods of valuation. Although they did not disagree, the local health department respondents emphasized the difficulties in collecting and analyzing the data. About the only general agreement was that respondents were struggling to demonstrate the value of their services and believed that rigorous science is the key to understanding and measuring value. A small number of respondents also noted that developing multiple methods of valuation would be beneficial.

Cost–benefit analysis and cost-effectiveness analysis.

A majority of respondents identified cost–benefit analyses and cost-effectiveness analyses as potential instruments to measure the value that certain services bring to the community. Despite the attraction of these methods for demonstrating value, only 1 or 2 respondents felt that local health departments were currently able to undertake these analyses.

The most significant barriers to implementing cost–benefit analysis and cost-effectiveness analysis were the absence of both skilled staff and adequate data to conduct the analyses. Respondents consistently mentioned several other challenges: limited resources, communicating the results to the public and policymakers to help them understand the value of the services provided, and political willingness to base allocation decisions on the results of the analysis. To contend with concerns about staff capacity, 2 respondents recommended developing academic–practitioner partnerships; another suggested pooling resources across jurisdictions to conduct analyses.

Return on investment.

A majority of respondents noted the desire to demonstrate that public health services provide communities with a strong return on investment (ROI). A respondent from a national public health organization argued that ROI should be based on more than lives saved—for instance, the net present value of public health services. Those local health department respondents favoring ROI as a measure of value were quick to point out that they did not necessarily know how to translate it empirically.

Another concern the respondents expressed about ROI was that it could devalue certain important public health initiatives. Suppose, for instance, the ROI for vaccination is $10, but the ROI for the Special Supplemental Nutrition Program for Women, Infants, and Children is $3. Should resources be shifted to the higher ROI solely on the basis of cost efficiency?

Respondents worried that even if the conceptual challenges could be surmounted, there is a practical limit to ROI as a measure of value. In many prevention activities, the return on investment may be harm avoided. For instance, suppose a state invests $5 million in stockpiling antiviral drugs for a bioterrorism threat. If the attack does not occur, is there no return on the investment? Indeed, the actual ROI may not be determined for many years.

Even so, one proponent of ROI offered some ideas about use of the metric. This respondent noted that some public health services, such as epidemiology labs, could be marketed as profit centers, in which the measures of productivity could be turnaround time, workload units, and quality of disease surveillance testing. In this approach, the ROI could be fewer tests and improved accuracy rates.

Mortality and morbidity data.

No respondent disagreed with the proposition that collecting and analyzing morbidity and mortality data is an essential activity for assessing the value of public health services. But local health department respondents consistently argued that attributing changes in morbidity and mortality to public health interventions was very challenging. Surveys were too expensive to conduct. Willingness to pay and cost–benefit and cost-effectiveness analyses were beyond the local health departments’ analytic capacity. As a result, local health department respondents identified individual program outcomes as a more meaningful and tractable way of assessing morbidity and mortality trends.

Two types of solutions to this problem have arisen. According to interview respondents, the state of Wisconsin is taking the lead on developing morbidity and mortality measures and providing the information to local health departments. Wisconsin is developing community profiles to rank counties (relative to each other and to other states) on the basis of aggregate health outcome measures to understand changes over time and, perhaps, to identify and disseminate successful interventions.

A small number of respondents (particularly respondents from national public health organizations and academia) focused on showing the value of public health services in extending life (i.e., in years of life saved) and enhancing individuals’ productivity. For example, what are the productivity gains from avoiding chronic disease? What resources will be needed to save lives? How many lives did a public health intervention save? One national public health organization respondent argued that years of productive lives lost could be an interim measure that translates into tax income lost, hence justifying the need to intervene. Doing so would require translating information about visits into health status changes per year of life gained. In this approach, local health departments would need to collect data on health status indicators or conduct health impact assessments.

Cost-accounting models.

Interviews also revealed cost-accounting models, in which data are collected at the local level on various dimensions of public health to establish program priorities and allocate scarce resources. The Lake County General Health District in northeastern Ohio uses a particularly comprehensive model based on a combination of cost-accounting methods, community assessment, and priority ranking methods.⁶⁵

To measure the value of each public health program, the model assesses the service's public health importance along several dimensions, including community priorities, legal and regulatory requirements, financial impact, the number of people served (as a percentage of the population), whether the service would be available elsewhere in the community for the same number of people, and impact on mortality or morbidity if the program were not implemented. An important aspect of the model's priority setting as highlighted by local health department respondents is to conduct community needs assessments through surveys or town meetings. According to its developers, this model provides justification for making program trade-offs at the margin, including eliminating programs that are no longer needed. It also imparts transparency and accountability for political decisions.

Performance-based contracting.

Another model mentioned by a public health official was Wisconsin's experiment with performance-based contracting. In this approach, the state negotiated contracts with local health departments for public health services.⁶⁶ According to our respondents, the goal was to use contractual negotiations to set the value of services (i.e., to define exactly what the local health department would provide for the state's investment and what outcomes would result). Through the negotiations process, a local health department sets priorities for service delivery. The idea was to reward performance—and penalize failure—through financial incentives tied to outcomes. If the local health department could provide the expected results for less money, it could keep the difference. If the expected performance targets were not met, the local health department had to reimburse the state for a portion of the money.

Our interviews suggest that the model had considerable support at the state level but was less enthusiastically received at the local level. For this model to work, respondents noted, strong and continuous political support and a willingness to sanction failure to meet the contractual productivity goals would be required.

Key challenges.

Despite their desire to use metrics to measure value, respondents were cognizant of the challenges they faced. The heart of the difficulty in measuring value is the attribution problem—the difficulty of demonstrating that the investment in public health contributes to decreased morbidity and mortality (i.e., that the outcomes are related to the intervention). Respondents consistently said it was hard to demonstrate the “correlations between prevention and disease reduction.” Further, “people don't attribute value to public health and its impact on community health.” In a political environment that focuses on short-term benefits, the inability to show short-term population health gains only exacerbates the attribution problem. Just as important, it is difficult to measure the value of a local health department's intervention in isolation. These interventions often involve systems issues, which can only be measured through the combined inputs of various factors and collaborators.

Another challenge pertains to inadequate resources to conduct data analyses to measure value. Outcomes data are not readily available and may be very expensive to collect and analyze. Local health departments may not have the capital or staff capacity, in size or skill, to perform the more sophisticated quantitative methods recommended in our interviews.

A related problem our respondents recognized was staff resistance to the need to measure value, especially for making trade-offs at the margins between equally valued programs. Consistently, respondents noted that public health practitioners were reluctant to cut any program. A few went so far as to criticize practitioners as being “purists who won't compromise.” Although this was not a prevailing attitude among our respondents, the concern that public health practitioners have a “holier-than-thou” attitude can certainly be an impediment to local health departments making the fundamental trade-offs they now face.

Aside from the attribution problem, our respondents suggested that understanding public health's intangible values was the most difficult conceptual and measurement challenge they faced. Virtually every respondent maintained that a unique trait of public health was the intangible value that it has. Respondents characterized these intangibles in various ways, but centered on notions of social justice, prevention, the equitable distribution of health to the entire population, and the reassurance value of having public health programs in place (e.g., bioterrorism surveillance): “Improving the population's health thus depends on more than measurable economic constructs.” The reassurance value of public health, which one respondent depicted as “domestic tranquility,” was compared with the equally difficult to enumerate kind of intangible benefits of having police and firefighter protection. Respondents also emphasized difficult to quantify “sustained efforts” of public health investments—for example, the benefits of vaccination programs—that represent another intangible bonus. One respondent described these long-term efforts as “the hardest thing to measure.”

DISCUSSION

Explaining the Disconnect Between Economists and Public Health Officials

Taken together, our literature and interview results suggest a large gap between academic researchers (especially economists) and public health practitioners in measuring value. For the various metrics described above to be useful to practitioners, we need to bridge that gap.

Our review of the economic literature revealed numerous studies with varied conceptual approaches to value investments in public health interventions. These studies have not penetrated the practitioners’ toolbox, however. None of our interviews revealed any direct application of these metrics to the practical working needs of local health departments. Not one local health department respondent identified the existing research on valuation as a readily accessible source of information, even as they emphasized the need to better measure the value of investments in public health programs. Despite their struggles to measure the value of public health services, respondents did not appear to be familiar with any existing studies on the topic.

Several factors might explain these results. Part of the problem may lie with the quality and perceived relevance of the economic analyses. As Neumann et al. have explored elsewhere,⁶⁷ there remain variations in quality and notable gaps in the methods of published cost–utility analyses on the perspective of the analysis and in methods for costing and valuing health outcomes. Although cost-effectiveness analyses have covered a broad range of public health programs or services, they have generally not addressed the value of public health systems or infrastructure per se. Rather, they have focused on specific public health services, such as screening or surveillance programs. In many analyses, authors did not specify who the decisionmaker would be, nor did researchers typically specify what kinds of implementation costs or institutional hurdles might have to be overcome. As a result, these analyses’ utility to practitioners is limited, even though our respondents indicated a preference for measuring the value of individual programs.

Some other research has pointed to gaps in cost-effectiveness analyses of public health programs as opposed to clinical services. One study found that cost–utility analyses had largely overlooked Healthy People 2010⁶⁸ priority areas such as physical activity, environmental exposures, or tobacco use.⁶⁹ In general, the cost-effectiveness field has paid a great deal of attention to pharmaceuticals and surgical procedures and relatively little attention to public health strategies. Arguably, medical services receive much more funding than do public health activities, because their value is more comprehensively understood and measured.

A related problem is the lack of direct contact between academic researchers and public health practitioners. At present, there is no clear mechanism for disseminating the research results to practitioners. They attend different conferences, use different mechanisms to transfer information, and rarely collaborate on projects with researchers.

Even if practitioners were able to locate the research and trust its quality, the decisionmakers would likely have difficulty connecting the research to their own decisions. Economic analyses generally contain a “societal perspective,” which is appropriate in the sense that they incorporate all costs and benefits that accrue to society. The societal perspective analyses are difficult to use, however, because they ignore the working realities and lack of expertise in most local health departments. The studies may not reflect all of the implementation costs facing officials, for example. Nor do they take into account actual budget constraints facing local health departments. Aside from abstractions like QALYs, the studies typically do not account for the kind of intangibles (e.g., social justice) at the core of public health delivery.

Part of the problem is a disconnect in cultures. The conundrum is that the metrics developed must be feasible for overworked public health staff to use. If not feasible, the metrics will be dismissed out of hand. At the same time, the metrics must be robust enough to achieve results that would otherwise be unavailable. The scholarly research being conducted uses academic jargon and is usually published in somewhat obscure journals; local health departments lack training and resources to use the research. At best, therefore, the research remains elegant but inaccessible. Moreover, local health departments face political challenges in making resource allocation decisions on the basis of scholarly research. All of these challenges underscore the need to find ways to bridge the gaps between the cultures.

Opportunities for the Future

Our findings suggest several important directions for the future and the challenges, both conceptual and practical, that lie ahead.

Data collection.

Perhaps the most consistently recognized challenge to value measurement is the lack of core data sets. Compounding the lack of data sets is the lack of agreement on input and output measures, along with outcome measures. In our view, a high priority for the field of public health, at both the national and local levels, is to establish consensus on what data local health departments should routinely collect, which outcome measures should be examined, and how the data should be analyzed.

Such consensus would alleviate 2 problems. Agreement in the field could inform the development of standardized data collection approaches that are currently absent. It could also lead to solutions for more-meaningful data. Take morbidity and mortality data as an example. In theory, morbidity and mortality data should be a sine qua non of public health practice. In reality, such data have not been effective in generating ongoing political support for investing in public health, partially because of time lag issues (i.e., that the positive effects on the population's health are unknown until well beyond resource allocation decisions).

Our findings highlight a general need for ongoing dialogue on all of these issues, as well as resources for training local public health practitioners and for improvements of data systems. For their part, public officials would be well served by adopting formal evaluation components in programs, undergoing training in their use, and publishing results for the benefit of the entire public health community.

Our results also underscore a need to develop a framework and tools that consider the perspectives and requirements of the people making decisions. In particular, there is a need for academic researchers and national public health organizations and agencies to agree on a common research agenda. Consensus on a research agenda is essential to beginning to redress the gap between academics and practitioners.

Making cost–utility analyses more accessible.

Because they provide a means for comparing diverse programs in a consistent and defensible fashion, cost–utility analyses might be one part of a value measurement strategy in public health that also draws on other metrics and responds to the concerns raised in our interviews. Researchers conducting formal cost–utility analyses of public health strategies could help matters by adhering to recommended protocols for conducting studies (e.g., using cost per QALY framework and being clear about the perspective of analyses), focusing on public health systems and infrastructure rather than solely on clinical services, and specifying the kinds of implementation costs or institutional hurdles that decisionmakers might have to overcome in practice. Researchers might explore ways to communicate to public health officials the value of decision analytic models, which allow a comparison of alternative strategies while testing the strength of underlying assumptions, and the incorporation of difficult-to-quantify benefits in an explicit, quantitative, and systematic way.¹ Ideally, with these efforts made, formal cost-effectiveness analyses could occur alongside the kinds of qualitative assessments that are now in place.

As our interviews have demonstrated, the challenge of incorporating intangibles into value measurement is especially acute for public health, a field that traditionally places great emphasis on nonmonetary core values such as social justice and the social determinants of health.⁷⁰ In the future, it will be important both to step up efforts to capture these benefits in formal evaluations and to educate public health professionals about how the evaluation techniques, including the cost–utility analysis framework, allow some of these intangible benefits to be captured and measured. For example, decision analytic models with QALYs routinely attempt to capture the kinds of long-term effects of preventive interventions, such as vaccination programs. The models can also capture certain spillover effects, such as benefits to people not directly affected by programs. Further, the QALY framework could in theory capture intangibles, such as “reassurance” value, by placing different utility weights on well-described scenarios with and without a particular public health program.

Other intangible benefits, such as equity and social justice, may be more difficult, though not impossible, to measure. Analysts could weight QALYs gained to incorporate equity considerations.¹ At the very least, qualitative descriptions of equity and fairness might be presented to decisionmakers alongside quantitative estimates.

Defining and measuring the value of public health services is at a nascent stage, perhaps similar to where quality-of-care research for personal health services was about 20 years ago. To be successful, this effort must be viewed as a long-term endeavor. Our results suggest that investment is necessary and can be successful. We would go further and argue that, without a sustained effort to define and measure the value of public health services (and, over time, of the public health system itself), the public health system will have an increasingly difficult time competing for scarce public resources.