Evaluating Frameworks that Provide Value Measures for Health Care Interventions

Abstract

The recent acceleration of scientific discovery has led to greater choices in health care. These new technologies, diagnostic tests, and pharmaceuticals have widely varying impact on patients and populations in terms of benefits, toxicities, and costs, stimulating a resurgence of interest in the creation of frameworks intended to measure value in health. Many of these are offered by providers and/or advocate organizations with expertise and interest in specific diseases (e.g., cancer, heart disease). To help assess the utility of and potential biases embedded in these frameworks, we created an evaluation taxonomy with seven basic components: (1) define the purpose; (2) detail the conceptual approach, including perspectives, methods for obtaining preferences of decision makers (e.g., patients), and ability to incorporate multiple dimensions of value; (3) discuss inclusions and exclusions of elements included in the framework, and whether the framework assumes clinical intervention or offers alternatives such as palliative care or watchful waiting; (4) evaluate data sources and their scientific validity; (5) assess the intervention’s effect on total costs of treating a defined population; (6) analyze how uncertainty is incorporated; and (7) illuminates possible conflicts of interest among those creating the framework. We apply the taxonomy to four representative value frameworks recently published by professional organizations focused on treatment of cancer and heart disease and vaccine use. We conclude that each of these efforts has strengths and weaknesses when evaluated using our taxonomy, and suggest pathways to enhance the utility of value-assessing frameworks for policy and clinical decision-making.

INTRODUCTION

The explosion of scientific knowledge, discovery, and technology began a new era of health care, wherein deployment of health care interventions depends upon personal, lifestyle, genetic, and molecular disease factors. While these advances have the potential to improve health, they increase the complexity of analyses to determine value at reasonable and affordable costs.(1–8)

Faced with new health care interventions and escalating costs of care, many professional organizations have developed frameworks to measure and/or rank the value of novel interventions. (3;7;9–22). Some of these new frameworks use components of cost-effectiveness analysis, while others create de novo value measures. It remains unclear how such frameworks will guide clinical practice policy, and whether they offer particular advantages over traditional approaches such as cost-effectiveness analysis.

We summarize a taxonomy of principles -- expanding upon cost-effectiveness theory -- that provide an approach to assess different value frameworks. Next, we apply this taxonomy to evaluate exemplar value frameworks to illustrate strengths, gaps, and potential concerns. We intend that this synthesis stimulate discussion about best practices and use of value metrics in policy and clinical decisions. We derived these principles in our synthesis of standard practice for the reporting of cost-effectiveness analyses, (23–25), recent ISPOR summaries of multiple criteria decision analysis (MCDA) reporting guidelines, (26) standard medical journal requirements for conflict-of-interest disclosures (27), standard Cochrane review methods (28) and our own analysis. While we draw many of these ideas from others, assembling them to evaluate models that estimate the value of health care is novel.

VALUE TAXONOMY

Value frameworks can assist in public health prioritization, practice guidelines, formulary or related resource expenditure decisions (e.g., purchase of equipment or coverage of a specific drug), and patient-physician decision-making about a specific care choice. Table 1 describes our taxonomy to evaluate these value frameworks. We intend this to apply to these different purposes, but specific elements may be more or less salient for different uses of the value framework. The following narrative highlights the implications of these varying purposes when using this taxonomy to evaluate value frameworks.

Table 1.

Proposed Taxonomy for Evaluation of Frameworks Assessing the Value of Health Care Interventions

Category	Component detail
Purpose
	Is the purpose defined and are the elements of the framework consistent with the purpose?
Conceptual approach, perspective, and preferences
	Is the value structure of the framework clear and transparent? If it requires assumptions, are do they have general and clinical face validity?
	What perspective does the framework use (e.g., societal, patient, provider, payer)? Is the perspective made explicit or embedded in the framework?
	Does the framework include standard methods of measuring value, such as cost-effectiveness analysis?
	Does the framework consider multiple attributes of medical interventions such as individual preferences, equity and the distribution of health care benefits and costs, and issues involving externalities (such as contagious diseases), and the value of scientific breakthrough? If the framework includes multiple dimensions of value, are the weights for each component population-based or expert-provided, and are the methods for eliciting such weights both clearly stated and methodologically sound?
	Does the framework allow individual patient preferences? If so, are they elicited with methods known to be free of bias and produce reliable results?
Intervention components and comparators
	Does the framework include all components and consequences of the intervention, or merely a portion of those (e.g., drug acquisition costs)?
	Does the framework aggregate or disaggregate such things as toxicity or other side effects of intervention?
	Does the framework assume as a baseline that some intervention will be provided, or does it allow for “watchful waiting” or “palliative care” as an option?
Data sources
	Are clinical and other data derived from expert opinion, population surveys, or other sources; are the sources made clear, and is the process replicable?
Economics/costs
	What is the effect of the intervention on the total cost of treating a defined population, including whether inclusion of the intervention will increase, decrease or leave unchanged the total costs of care for that population?
Uncertainty and identification of important gaps	Is uncertainty in data about costs or effects considered in conclusions about value? Are there gaps in knowledge about aspects of care that could change the value rating?
Conflicts of interest	Does the sponsoring organization have a financial stake in the process, and if so, is this declared? Does the framework bias the results in favor of the sponsor’s financial position?

Conceptual Approach, Perspective, and Preferences in Measuring Value

Value frameworks have evolved, in part, to provide greater standardization and transparency in health care decision-making. (29) Further, formal frameworks can minimize biases of intuitive decision making from issues of framing of the problem, risk analysis, inter-temporal tradeoffs, and others. (30)

We began our taxonomy in the formal framework of cost-effectiveness analysis (CEA). CEA analyses most commonly use the societal perspective, but this perspective usually does not correspond to that held by individual participants in health care decisions – patients, providers, payers, public policy decision makers, or owners of resources used to create health care interventions, nor does it consider community well-being or other important policy issues such as the distribution of benefits and costs across various subsets of the population (among other omissions).

No single perspective can represent the interests of all participants in value-based decisions. For example, physicians generally advise patients using a patient perspective, but may introduce the societal perspective in their role as steward of societal resources (31) or, in some cases, their own professional organization’s or personal financial situation. Thus, we believe that multiple perspectives should be considered, varying by individual or organization assessing the intervention, consistent with the most recent recommendations for reporting cost-effective analyses. (24;25)

CEA can also be useful in developing value frameworks since it provides data on intermediate events of importance to judging the value of an intervention, such as toxicity rates and overall survival, measured in life-years saved, quality-adjusted life years, or disability adjusted life years. (23–25; 32; 33) Compared with the existing care-standard, the ratio of added costs to added gains in health benefits-- the incremental cost-effectiveness ratio (ICER) – stands as one value criterion. However, no consensus exists on cut-points to determine “acceptability” of a given ICER, and both illness severity and the number of people affected can influence “cut point” choices.

CEA also does not typically capture less tangible factors such as fear associated with diseases like HIV, cancer, degenerative neurological conditions, or mental illness, since these do not represent distinct health outcome states for which utility measurements exist. In concept, CEA could incorporate such issues, but in many cases, the data requirements would be impossibly complex. Full CEA evaluation requires population-based utility level estimates for every state of nature in the model. Capturing such things as fear of the diseases, contagion effects, restrictions on freedom, and modification of religious rites (all arising in the response to Ebola virus) provide examples of issues that CEA cannot meaningfully capture.

While grounded in CEA, our taxonomy extends beyond this foundation. CEA does not generally consider many other attributes of decision-making that affect use of health care interventions. For example, CEA cannot address the distribution of health burdens that might fall particularly heavily on disadvantaged populations (e.g., sickle cell anemia in African-Americans), or whether some aspects of a medical intervention have important moral or ethical components, (e.g., abortion after detection of a fetal genetic mutation with profound consequences.

Applying CEA for decision making brings another issue to the foreground: a new technology may pass standard “cutoff” requirements but then overwhelm the available budget – the issue of “affordability.” CEA methods currently are not equipped to deal with this issue.

Multi-criteria decision analysis tools offer one approach to formally include these “other issues” into the value framework. These tools include such approaches as the Analytic Hierarchy Process (34) and the Multi-Attribute Utility Theory, (35–36), both used in recent health care decision processes. (37–40)

These models bring to the foreground an issue that does not arise in CEA: how does one balance among competing goals in a value model? Multi-criteria decision analysis methods use various methods to elicit weights attached to each (potentially) competing goal. With a single decision maker, these methods work well in general (34;36), but best methods to combine individual into group preferences remain elusive. If group estimates are not derived from relevant population samples, our framework suggests that analysts should provide good justification and clear logic for using alternative sources.

Our taxonomy proposes that value frameworks incorporate preferences, examine multiple perspectives, and are transparent. In addition, we suggest that value frameworks should consider distribution and ethical issues and other pertinent factors by using multi-criteria decision analysis methods.

Accounting for all Components of the Health Condition and Intervention

Value frameworks should consider many aspects of each disease process and all short-term and downstream benefits and harms of interventions. For example, in cancer, including only one cost component, such as drug acquisition costs for a new treatment, would bias the overall assessment by ignoring other costs, such as those of drug administration, supportive care, and other aspects of therapy. In treatment of HIV, cancer, and other conditions, drugs may have significant toxicity that can affect their value. Screening tests like mammography can save lives, but also carry with them meaningful rates of false positive results and over-diagnosis. These issues should enter a full value assessment, not only in health status outcomes and costs but also in emotional burdens created. (41;42)

Some value frameworks treat all harms as equal in consequence to patients or other decision-makers. For example, when the intervention is a drug treatment, rates of all side effects might be used in comparing the value of two therapies. This creates biases when the intensities and/or durations of the side effects differ, or when some effects have greater consequence than others to individual patients (e.g., nausea/vomiting vs. fatigue vs. increased risk of iatrogenic death). In best practice, our taxonomy recommends that individual preferences on these issues inform the overall assessments of alternative therapies, and that when infeasible, population-based estimates serve as an alternative.

Current value frameworks tend to focus on active therapy; e.g., drugs, surgery, and/or invasive tests. More recently, for certain diseases, such as early, non-aggressive prostate cancers, with invasive and potentially toxic standard-care therapies, alternatives now include no therapy (active surveillance). In the advanced disease setting, palliative interventions are becoming increasingly considered. If value frameworks do not explicitly consider these less aggressive approaches (i.e., only comparing novel therapies with standard therapy), the resulting decision may be biased towards therapeutic action, possibly conflicting with patient preferences. Therefore, we suggest that value frameworks compare all feasible options for patients, not just those using therapeutic intervention.

What are the sources of and process for use of data in the framework? Does this process conform to accepted methods for evidence evaluation?

Several accepted methods and hierarchies evaluate the quality of evidence about health care intervention effects. Randomized controlled trials generally provide stronger evidence than other methods (e.g., historical comparisons, case-control studies, and others), and sample sizes and research design in all studies affect precision and accuracy of parameter estimates. (28)

In cases where good quality, empirically based evidence on key issues in the value framework is not available, expert judgment may provide the only realistic option. Using expert-based estimates creates biases when the experts have a professional or financial stake in choices being evaluated (e.g., profit from sales of a device or drug, or enhanced reimbursement for delivery of a new agent -- see below for further discussion). When possible, valuation models should use experts not in conflict-of-interest situations and should clearly identify when any such conflicts exist.

Does the framework include the full economic consequences of all intervention components?

As in standard CEA, our taxonomy recommends that value frameworks include the full cost of all the components and consequences of an intervention. For example, in frameworks to evaluate drug treatments, economic impacts would include drug costs, provider time and delivery costs, staff time, facilities and equipment overhead and costs, costs of treatment-related side-effects and supportive care medications required, patient time and travel costs, and costs of all downstream events until death from the disease or other causes. To the extent that frameworks exclude portions of these costs, they result in biased recommendations.

Many value frameworks have different perspectives and include different costs. Patient-perspective frameworks, for example, might normally include only copayments made by patients. Payer-centric frameworks may focus only on the costs and savings to the health care provider. These approaches can lead to sub-optimal value-based decisions from a societal perspective.

Costs also need to be considered in the context of extant budgets. For example, hepatitis C medications have cost-effectiveness ratios within the range of other accepted interventions, but still have high absolute costs that may overwhelm pre-established budgets.

Projected budget impact has played a key role in some health care system decisions about these medications, mostly appearing as an independent discussion of “budget impact” in addition to standard cost-effectiveness analysis. (17) We believe that this approach misses a key point: budgets and acceptable levels for cost-effectiveness ratios cannot be considered independently. Introducing a new (and expensive) treatment adds budget pressure, which invariably will raise the value of incremental resources (what economists call the “shadow price”). The proper solution will likely involve reassessment of the existing budget and reexamination of interventions that cannot compete (in a cost-effectiveness paradigm) with the newly tightened budget situation.

How does the framework account for uncertainty in effect or cost data?

Many analytic methods incorporate effects of uncertainty on the conclusions of a value recommendation. A key aspect of characterizing uncertainty relates to identification of gaps in knowledge or evidence about aspects of care that could change the value conclusions. Our taxonomy does not specify a particular method to identify the impact of uncertainty, but rather that uncertainty and gaps in data be explicitly evaluated.

Conflicts of Interest

Professional specialty organizations often have the best knowledge about a particular set of health care interventions, and thus are well-placed to develop value frameworks, but may also have conflicts of interest in these issues. Value assessments may be subject to bias if financial incentives favor more (and costlier) interventions in certain delivery settings (e.g., fee for service vs. managed care or accountable care organizations). These issues extend also to the use of facilities owned in part or whole, directly or indirectly, by decision-making providers.

Beyond financial incentives, the purpose of specialist organizations is to advocate for professional authority and action, which may further bias these sponsors toward interventions based in their specialties (such as chemotherapy, which is rooted in oncology, compared with palliative care, with separate specialty affiliations). Thus, we should consider if value framework’s sponsoring organization has a professional stake in the process, and if so, whether the framework potentially biases the results in favor of the sponsor’s position.

Examples of Recent Frameworks

Value frameworks have recently emerged from different medical specialty groups representing oncology, (9;16–22) cancer screening, (10) radiation oncology, (11;12) cardiology,(7) surgery, (13) pediatrics, (3) and infectious disease and vaccines. (14) To demonstrate the application of our taxonomy, we selected four exemplar value frameworks spanning several specialties (Table 2).

Table 2.

Examples of Application of Proposed Taxonomy for Evaluation of Frameworks Assessing the Value of Health Care Interventions

Protocol Element	American Society of Clinical Oncology (19)	National Comprehensive Cancer Network (16)	American Heart Association-American College of Cardiology (7)	European Vaccine Working Group (33)
Purpose
Stated purpose	Patient-physician drug treatment decisions; separate framework for adjuvant vs. metastatic Rx	Patient-physician drug treatment decisions for surgery, radiation, and drugs	Inform practice guidelines and performance measures	Public health policy decisions
Framework consistent with purpose	Yes; will need to develop format for use in practice	Yes; included with online treatment guidelines	Yes	Yes
Approach
Clear and transparent conceptual framework	No	No	Yes	Yes
Do assumptions have empirically supported clinical basis?	Bonus points subjectively assigned	Scoring system subjective/expert opinion	N/A	Yes
Is the perspective made explicit	Implies clinical decision making perspective	Implies clinical decision making perspective	Societal, with separate analyses with patient perspectives for clinical decisions; consider subgroups	Societal with long term population outcomes
References existing methods of assessing value, allowing comparison across diseases	De novo method: net health benefit score and costs per drug treatment course. “Bonus points” and toxicity included in net benefits score for a new regimen vs. control therapy Points for benefit are assigned based on categories of median survival Does not allow comparisons across multiple available regimens	De novo method: evidence block score from 1–5 based on subjective assessment of benefits Does not allow comparisons across multiple available regimens	Uses costs per QALY to grade value low to high value using WHO-CHOICE framework; includes uncertain value category. Value categories can be based on societal cut-points or a percent of GDP	Uses dynamic agent based models and costs per QALY May consider single or multiple vaccines
Considers multi-decision attributes (preferences, equity, distribution, externalities)	Preferences included in metastatic framework only, but based on expert designation of “bonus points” for symptom palliation, and QOL, longer life with metastases	unknown	Considers preferences, distribution, and budget impacts	Explicitly considers use of multi-attribute theory to assist with decisions
Intervention components and comparators
All components and comparators considered	Only clinical benefit as gain in survival, major toxicity, and costs of drugs and supportive care for drug delivery; patient co-pay for drugs included. Other effects and costs excluded. Update to framework includes lower grade toxicities. Only examines one drug and one control drug	Effectiveness, safety (i.e., major toxicity), quality of evidence, consistency of evidence, and affordability No direct comparisons among treatment modalities	Implicit in CEA framework, but not specified; framework used as a broad template	Very broad horizon and effects considered in a CEA framework
Toxicity/side effects considered	Yes, but not in costs for hospitalization, etc.	For some regimens	Implicit but not specified	Yes
Considers no treatment as a comparator (e.g., “watchful waiting” or “palliative care” )	No	For some treatment options (surgery, radiation, drugs)	Unknown	N/A
Data sources
Trials or observational studies vs. expert opinion; quality of data	RCT and expert opinion	RCT and expert opinion	Specifies US data. Data rated from A–C based on a formal review of the evidence quality; includes randomized trials and observational data	Vaccine trials using intention to treat results
Economics/costs
All costs stemming from the intervention included	Only drug costs; drug delivery and supportive care for drug delivery Includes patient co-pays for drugs but not time costs	Drug cost, supportive care, and drug administration, and toxicity costs; costs ranked subjectively on an “affordability scale” from 1–5 (inexpensive to very expensive)	Includes all costs in CEA framework	Includes all costs in CEA framework
budget impact assessed	Budget impact not considered	Budget impact not considered	??	??
Uncertainty and information gaps considered	Not considered	Not considered	Not explicitly considered; does include an unknown category for value	Yes
Conflicts of interest	Not explicitly addressed. Disclosures included in journal article. Potential conflicts since value primarily determined by oncologists who deliver (and bill) care	Not explicitly addressed. Potential conflicts since value primarily determined by oncologists who deliver (and bill) care	Explicitly addressed; <50% of writing committee required to not have conflicts; relationships with drug or device companies allowed and disclosed online	Explicitly addressed; and disclosure included
Sponsoring organization has financial stake? Declared?	Yes, No	Yes, No	No, No	. Employees of drug or vaccine companies excluded; all conflicts of relationships declared

Oncology

The oncology frameworks largely focus on the value of new drugs, driven by the more rapid escalation of drug costs relative to other components of cancer care; and the impact of drug costs on patient co-payments. (31;43;44) In other disease specialties, new treatment paradigms, and expensive drugs and high priced technology have also stimulated the development of value frameworks.

The American Society of Clinical Oncology’s (ASCO) Task Force on Value in Cancer Care (18;19) and the National Comprehensive Cancer Center Network (16) both developed value assessments for systemic therapy drugs. Comments about this work recognize that the intended use centers on patient-based treatment decisions rather than a societal perspective. These frameworks followed many elements recommended in our taxonomy, but omit several key value components. First, they only focus on clinical outcomes related to survival and drug costs. Second, by limiting the framework to clinical trial evidence about episodes of drug treatment, they also do not permit assessment of the value of a given therapy in the general population. (45) Third, the framework excludes other important costs of care, such as future need for radiation, repeat surgery, or palliative care. Fourth, many relevant outcomes needed to determine value from a societal or patient perspective are excluded, including quality of life impact of therapy, (46) perhaps as a result of omission of such data in the original clinical trials of efficacy. (47) The revised ASCO framework (19) endeavors to capture all types of toxicity, but can only succeed to the extent lower-grade adverse events are reported.

Next, the method used to assign value points is both arbitrary and relies primarily on opinions of providers rather than samples of patients or their families. Finally, there is no accounting for uncertainty in the underlying data or the result, giving a false sense of precision when comparing among treatments.

Other oncology frameworks include Drug Abacus, (48) the European Society for Medical Oncology’s Magnitude of Clinical Benefit Scale, (22) and the pan-Canadian Oncology Drug Review (pCODR), (20;21). These frameworks seek to inform policy rather than clinical decision-making. They too focus primarily on drugs, with Drug Abacus evaluating cost of drug development and production vs. actual retail pricing. The pCODR is the only value framework in oncology that includes a societal perspective, patient stakeholders, components of a CEA framework including QALYs, considers evidence beyond trials, and determines how the short-term budget impact of new interventions affect the Canadian health care budget.

These oncology frameworks were developed by oncologists (with other researchers) who have professional and financial stakes in the drugs they prescribe, so that the possibility of conflict of interest arises. A recent review of oncology drug value frameworks was supported by a pharmaceutical company. The authors declared that they owned stock or consulted for this company. (2)

Cardiology

The American College of Cardiology and the American Heart Association (ACC-AHA) developed another recent value framework (7) to inform their practice guidelines. This framework rates both the quality of evidence and the value of care, with the latter based on a CEA framework. Based on CEA results, they developed a score for value from low to high using the World Health Organization WHO-CHOICE project, including the use of cutoff values of 1X to 3X per capita Gross Domestic Product. (49) Notably, the writing committee had fewer than 50% of members with industry relationships, and all of these relationships were published. Although the primary data resources for this evaluation come from independent entities, we note that when committee members are the providers of therapies, they may have inherent conflicts of interest that can cause deliberate or subconscious bias. In general, the ACC-AHA framework robustly addresses almost all of the key components of our taxonomy but could be improved by more explicit definitions of intervention components, and approaches to uncertainty.

Vaccines

Frameworks for evaluation of vaccines against infectious diseases present many complexities that do not arise with other medical interventions, including the need to capture spread of disease in the population, changes in population demographics due to aging and migration, the natural history of immunity, herd protection, community well-being, the need to model the population equilibrium, the lack of post-market surveillance for adverse events, and the ability of the vaccine to prevent disease or disease related effects (e.g., post-herpetic neuralgia). (6,50–51). These factors present unique challenges in defining the perspective, individual preferences versus effects of treatment of one person on unaffected persons (i.e., herd effects), and time horizon for a value framework. (39) Despite this complexity, the European guidelines for vaccine value assessment considers virtually all of the components in our taxonomy. (52) These guidelines explicitly address conflicts of interest: the developers excluded employees of drug or vaccine companies and required members to disclose potential conflicts.

CONCLUSION

Although the movement to examine the value of health care is rapidly gaining prominence, no consensus exists on value’s structure and components. Wider implementation of value frameworks may be possible by further use of data from clinical trials and information made available through electronic records.(46;53)) It will also be useful to include education about value frameworks in medical and public health training.(454;55) Our taxonomy could support these efforts and extends prior overviews (2;25) by including a comprehensive set of metrics grounded in cost-effectiveness analysis.

Summary.

Summarizes principles that could be used to evaluate value frameworks and applies these to analyze strengths, gaps, and potential concerns about some recently published frameworks.