Rethinking How We Measure Costs in Implementation Research

Abstract

Background

Hospitals and other health care delivery organizations are sometimes resistant to implementing evidence‐based programs, citing unknown budgetary implications.

Objective

In this paper, I discuss challenges when estimating health care costs in implementation research.

Design

A case study with intensive care units highlights how including fixed costs can cloud a short‐term analysis.

Participants

None.

Interventions

None.

Main Measures

Health care costs, charges and payments.

Key Results

Cost data should accurately reflect the opportunity costs for the organization(s) providing care. Opportunity costs are defined as the benefits foregone because the resources were not used in the next best alternative. Because there is no database of opportunity costs, cost studies rely on accounting data, charges, or payments as proxies. Unfortunately, these proxies may not reflect the organization’s opportunity costs, especially if the goal is to understand the budgetary impact in the next few years.

Conclusions

Implementation researchers should exclude costs that are fixed in the time period of observation because these assets (e.g., space) cannot be used in the next best alternative. In addition, it is common to use costs from accounting databases where we implicitly assume health care providers are uniformly efficient. If providers are not operating efficiently, especially if there is variation in their efficiency, then this can create further problems. Implementation scientists should be judicious in their use of cost estimates from accounting data, otherwise research results can misguide decision makers.

INTRODUCTION

Health care delivery organizations must make difficult decisions on whether to implement evidence-based programs, and costs frequently play a role in this calculus due to conflicting budgetary priorities. As a result, many organizations seek to examine the benefits and costs of an intervention prior to implementation. One way to weigh the relative costs and benefits is by using a cost-effectiveness analysis (CEA), which many professional societies, such as the American College of Physicians, view as the gold standard for measuring health care value.¹ CEA computes an incremental cost-effectiveness ratio (ICER), which compares two or more interventions in their relative costs and relative quality-adjusted life years.

CEA is widely used in health care to compare the long-term costs and benefits of different treatment strategies. In 2016, the US Panel on Cost-Effectiveness recommended conducting additional analyses with shorter time frames or from a payer’s perspective.² These two issues are often interpreted as simple “tweaks”; however, these changes have material implications for measuring costs that, to date, have not been adequately appreciated in implementation research.

In a world where researchers use charges, costs, payments, and reimbursements, sometimes interchangeably, true north is the concept of opportunity costs. As stated by the World Health Organization, “Economists think of the costs of using resources for a particular activity as being the benefit foregone because the resources were not used in the next best alternative—the opportunity cost.”³ There is no database of opportunity costs, however, and it is easy for researchers to focus on accounting data and forget that the ultimate goal is to understand opportunity costs (see Table 1).⁵ Implementation researchers conducting economic evaluations will encounter scenarios where they will have to measure costs, and the best choice in each situation should be guided by the desire to measure the health care organization’s opportunity costs within a specific time frame.

Table 1.

Options for Measuring Costs in Administrative Data

Term	Definition	Challenge
Charges	The amount listed on the bill.	Without adjustment, bears little resemblance to payments or costs.4
Reimbursements/payments	The amount paid or reimbursed for services rendered	Payments are based on a pre-defined set of insured benefits. May not reflect the resources used for individual patients.
Activity-based cost	Cost estimates based on local prices and activities.	Often not available to researchers; embedded accounting rules are often not explicit
Opportunity cost	Benefits foregone because the resources were not used in the next best alternative	Not observed in any database

In this paper, I discuss methods for estimating health care costs in implementation research. I draw on the Panels on Cost-Effectiveness in Health and Medicine^{2, 6} in particular the chapters by Luce et al.⁷ and Basu.⁸ To highlight the challenges in estimating costs, I motivate the discussion using a hypothetical intervention designed to improve care in the intensive care unit (ICU). I then provide a brief background in microeconomic theory that underpins economic evaluations in health care, followed by recommendations for implementation researchers.

CASE STUDY: ICU CARE

There are an estimated 51,000 ICU beds in the US.^{9, 10} ICUs are designed to provide life-sustaining services, but up to half of ICU beds are not being used for patients who need life-sustaining care.^{11, 12} Many ICU beds are filled by lower acuity patients where the clinician wants additional monitoring or they are used by patients who are at the end of life and end up in the ICU for lack of a better place to go.

ICU care is expensive and each day in the ICU is approximately twice as expensive as a normal acute hospital bed because there are large fixed costs (i.e., specialized equipment) and more staff. The first day in the ICU is the most expensive because the patient receives more services on that day. Daily ICU costs decrease by almost half from day 1 to day 2 and then stabilize in the days after that. Using data from 2002, Dasta and colleagues¹³ reported daily ICU costs without mechanical ventilation of $6667, $3496, and $3184 on days 1, 2, and 3+, respectively.

Researchers have explored two strategies to improve care and reduce spending in the ICU.¹⁴ One involves diverting low acuity or end-of-life patients away from the ICU, and another involves transferring these patients out of the ICU faster. Both of these strategies seem valuable and one might hypothesize that implementing them would lower costs and improve care. However, it turns out that both approaches can lead to higher patient costs, depending on how those costs are computed.

We examined these two strategies using ICU data from the US Department of Veterans Affairs (VA), which has used an activity-based cost accounting system since 1998. Table 2 shows unadjusted VA ICU costs from 2018. The first day is the most expensive and then the cost per day declines, consistent with Dasta and colleagues.¹³ The breakdown of variable and fixed costs provides unique insights into these two strategies.

Table 2.

Unadjusted Costs for ICU Admissions by Day of Stay (n = 27,748)

Day of Stay	Cost per added day
	Total cost	Variable costs	Fixed costs
1	9605	5210	5915
2	6838	3692	2712
3	7250	3951	2829
4	7914	4303	3069
5	7332	4050	2889
6	7826	4362	2988
7	6458	3541	2645

Source: 2018 VA Treating Specialty Files

Note: Data were truncated on day 7

Implementing a diversion strategy will result in some unfilled beds. This yields savings in variable costs because fewer staff and supplies will be used (assuming the hospital can reduce staffing). But the fixed cost (ICU bed and specialized equipment) are still incurred, even if the bed is unfilled. The net result will depend on the savings in variable costs relative to the increases in fixed costs. In this case, an economic analysis using total costs to estimate savings would be misleading.

Implementing a program that transfers patients out of the ICU faster will result in more patients spending at least one day in the ICU. In this scenario, the fixed costs do not change, but because the first day incurs more variable costs than subsequent days, as shown in Table 2, this implementation strategy will result in greater costs than usual care.

The activity-based cost data provide insights into the hospital’s opportunity costs and how that changes over time. In the short term, the focus should be on the variable costs, and diverting patients is the preferred strategy because it saves more variable costs than usual care or a transfer strategy. In the long run, however, diversion does not save money unless the director converts the unused ICU beds into more productive assets. This option takes time and is usually not a short-term option, but it is a long-run option.

Market Failure and Health Economic Evaluation

If health care markets worked well, then ICUs would only be used for patients when the marginal benefit exceeded the marginal costs (i.e., patients needing life-sustaining care). Well-functioning markets efficiently govern transactions between producers and consumers. In these situations, markets create incentives for producers to innovate, improve quality, and reduce prices. Health care is an example where markets do not work well. Arrow¹⁵ described how information asymmetries and uncertainty lead to market failure. This has several broad implications, but most relevant for implementation scientists is the inability for purchasers or consumers to distinguish between high and low-value care.

Developed countries around the world vary in their approaches to value assessment in health care. The USA has focused much of its efforts on injecting competition into health care and encouraging purchasers, with the exception of the Centers for Medicare and Medicaid Services, to measure value. Other countries, such as the UK, have centralized some of these efforts. The National Institute for Health Center for Care Excellence works with developers to examine the cost-effectiveness of new drugs and devices.

In medicine, the gold standard for measuring value is cost-effectiveness analysis, sometimes referred to as cost-utility analysis. This compares the marginal costs and marginal benefits for two or more interventions,¹⁶ with results summarized in an incremental cost-effectiveness ratio (ICER). There is a long-standing, robust literature on CEA that goes back decades.^{2, 6, 17, 18} Methodologists often like its theoretical roots, while many practitioners prefer it over alternatives, such as cost-benefit analysis, which requires mortality or quality-of-life effects to be measured in dollars.

Although it is the dominant strategy for measuring value, CEA has limitations. CEA models are difficult to create, being both technically complex and time-consuming and expensive to build and update. CEAs are rarely done on existing treatments, so there is often a push to adopt new, more expensive treatments, rather than dropping low-value services.¹⁹ In addition, rarely is there any momentum to adopt disruptive innovations that may not be quite as good as existing treatments but much cheaper.²⁰ This happens frequently in other markets, such as the rollout of new smartphones that offer decent options at a fraction of the cost of top-of-the-line models.

Another challenge with CEA is tailoring the results to specific purchasers or providers. Consequently, critics sometimes say that the CEA models work well for the average patient, but they do not apply to “my patients,” or that they have a different perspective that is not reflected in the CEA model. The importance of perspective is highlighted by an example in substance use treatment where there was increasing evidence that substance use was cost-effective, while health care organizations were closing their substance use treatment programs. The paradox, it turns out, was explained by differences in perspective.²¹ In a societal CEA, the added costs to health care organizations for providing substance use treatment was offset by savings in criminal justice.²² Consequently, when viewed from the health care purchaser’s perspective, substance use treatment was net loss.

The US Panel on Cost-Effectiveness recognized the importance of perspective and shorter time frames in their 2016 update.² Unfortunately, changing the perspective or shortening the time frame will not automatically solve the problems facing implementation researchers wanting to conduct economic evaluations. This is because there are two challenges with measuring costs that are not frequently discussed but are particularly complex for implementation researchers. First, CEA models use cost estimates that reflect long-run costs; these estimates do not accurately reflect short-term opportunity costs. Second, CEA assumes that health care systems are uniformly efficient. As discussed next, these assumptions may not be appropriate for implementation research.

Variable and Fixed Costs

Economic evaluations should only include costs that can be varied in the time horizon. Health care organizations use inputs to produce health care services. Some of those inputs (e.g., supplies) are variable because they can be easily changed in the short term. Other inputs are fixed over a period of time. Building a new hospital requires a major capital investment, which cannot be easily withdrawn and invested elsewhere. The variability of some inputs including labor and leased space often depends on contracts.

A CEA analysis with a long-run perspective would include all costs because they can all be varied in the long run.²³ Analyses that take a short-term perspective should exclude costs that are fixed in that time period.⁷ This method is rarely adopted in practice, but it can be particularly important for implementation research, where “success” is often gauged in the next 1 to 3 years.

In the ICU example, an implementation study that uses a short-term time horizon should only include costs that vary in the short term. If we re-evaluate the two strategies considering only variable costs, diverting low-acuity patients away from the ICU saves variable costs and is the winning strategy. Conversely, cycling patients through the ICU faster would increase variable costs and this strategy should be avoided. Consequently, implementation researchers interested in a short-term horizon should only focus on costs that can plausibly vary in that time horizon. Fixed costs, by definition, do not have an opportunity cost because these assets cannot be repurposed (i.e., zero benefits foregone).

One challenge with this approach is that it requires detailed data that are only observable in activity-based cost accounting systems.²⁴ Large databases commonly used in research, such as Medicare fee for service claims or Optum data, lack information on the variable and fixed costs and will be insufficient in these situations. Similarly, accounting ledgers are too imprecise and rarely track costs related to specific programs or patient populations. In the best-case scenario, one could use existing activity-based cost accounting data, such as the VA Managerial Cost Accounting data, for the analyses. In the worst-case scenario, one would have to estimate the variable costs through direct measurement of activity-based costing, as has been advocated by Kaplan and Porter.²⁵

The importance of excluding fixed costs increases with the relative proportion of fixed to total costs. This issue is particularly salient in the ICU example because ICU fixed costs are large relative to total costs. If we examined other care delivery modalities, where fixed costs are relatively small or can be varied easily through short-term leases (e.g., telehealth), then excluding fixed costs may be less important.

Even if fixed costs are relatively small, including fixed costs can introduce statistical problems. The fixed costs can be thought of as a noise parameter, and in the best-case scenario (i.e., the noise is randomly distributed), this noise reduces statistical power. However, if the noise is not randomly distributed but is correlated in unobserved ways with other variables, then it reduces power and introduces bias.

Efficiency

In well-functioning markets, inefficient or poor-quality organizations do not survive unless they can find ways of improving their production. However, in health care, market forces do not provide the same governance; organizations can prosper even if they are inefficient. Efficiency, in this context, reflects two issues. First, is the issue of technical efficiency, or whether the organization is maximizing outputs while minimizing quantity of inputs. Second, there is the issue of whether health care organizations are producing the right mix of services for consumers, often referred to as allocative efficiency.

CEA is designed to address the question of allocative efficiency, and to do so, it often makes assumptions about technical efficiency. Luce et al.⁷ identify the importance of technical efficiency in CEAs and state, “If the economies of scale in producing (or returns to scale in providing) an intervention are not known, then it will be necessary to estimate a cost function and derive the correct marginal cost.” However, there is often insufficient data from which to estimate a cost function, so instead, CEAs often assume that health care organizations are technically efficient (at constant returns to scale). In 2008, Adang and Wensing²⁶ question whether the assumption that technical efficiency is largely ignorable. This assumption is a particular concern for implementation researchers where implementation strategies are being used to fix inefficiencies in the delivery of care.²⁷

There are a number of practical implications if we want to relax the efficiency assumption. First, researchers should prioritize obtaining cost data, and in particular activity-based cost data, from the health care organizations being studied. These data reflect not only the local prices but also the organizational issues (e.g., culture and management) that are often not easily observed but correlated with costs and quality.²⁸ Second, because many health care organizations do not have activity-based cost accounting systems, implementation scientists will need to extract resource utilization data and pair it with cost data. Here, the applicability and relevance of these cost data are critical. Pairing utilization data from a large academic medical center with average costs from medium-sized community hospital raises questions about whether those costs are applicable to the academic medical center. More research is needed to guide this choice because the alternative, micro-costing through direct measurement, would likely be too time-consuming and expensive.

DISCUSSION

Forty years ago, Weinstein and Stason published a landmark paper on conducting cost-effectiveness analysis (CEA) in health and medicine.¹⁷ The use of CEA grew quickly in the subsequent two decades, although methodological inconsistencies were common.²⁹ In 1996, the Panel on Cost-Effectiveness in Health and Medicine confirmed that CEAs should use a societal perspective and lifetime (long-run) perspective.⁶ In 2007, and later updated in 2014, the International Society for Pharmacoeconomic and Outcomes Research (ISPOR) issued guidelines for conducting budget impact analysis (BIA).^{30, 31} The motivation for the BIA reflected a growing feeling that the CEA standards—the societal perspective and the long-term time horizon—created challenges for decision-makers who faced a narrower perspective and a shorter time horizon. The BIA was designed to focus solely on costs, usually with a short time horizon and a payer’s perspective. However, the 2016 Panel on Cost-Effectiveness in Health and Medicine followed suit, revising its recommendations to include alternative perspectives and shorter time horizons when useful for decision-makers.²

While the discussions about CEA and BIA evolved, the field of implementation science emerged, with researchers eager to help health care organizations adopt evidence-based programs. Not surprisingly, implementation researchers struggled to identify which “standards” were most appropriate, especially when decision-makers were asking about “return on investment.” Recent reviews highlight the growing economic literature in implementation,^{32, 33} and Roberts and colleagues³² make strong recommendations for the field.

Implementation research poses interesting challenges for health economists. Too often, health economists use cost estimates from accounting databases that may be a poor proxy for a specific health care provider’s opportunity cost. Implementation researchers stress contextual factors, and health economists are sometimes too quick to identify unit costs that are nationally representative, but detached from the local context, and in doing so, they are failing to measure the health care organization’s opportunity cost. When a health care organization is trying to decide how to improve care, whether that involves appropriate prescribing or care in the ICU, understanding the organization’s context and opportunity cost is critical. One way to do this involves qualitative or mixed methods.³⁴

Implementation researchers may need to focus on the short-term perspective, but when decisions differ between the short-term and long-term perspectives, then decision-makers should be informed. In the ICU example, the appropriate way for an ICU to save money in the short run is to divert low-acuity patients away from the ICU and to leave those beds vacant. This strategy does not work in the long run, however. In the long run, vacant beds must be converted into productive resources. Highlighting these issues will go a long way in helping organizations make smarter short-term and long-term decisions. Hopefully, this will help improve care and reduce the $750 billion spent annually on low value and wasteful services.³⁵

Funding

Funding was provided by the VA Department of Veterans Affairs to Todd Wagner through a Research Career Scientist Award (RCS 17-154).

Compliance with ethical standards

Conflict of Interest

The author declares that he does not have a conflict of interest