Evaluation of the ASCO Value Framework for Anticancer Drugs at an Academic Medical Center

Abstract

BACKGROUND:

Anticancer drug prices have increased by an average of 12% each year from 1996 to 2014. A major concern is that the increasing cost and responsibility of evaluating treatment options are being shifted to patients. This research compared 2 value-based pricing models that were being considered for use at the University of California, San Francisco (UCSF) Medical Center to address the growing burden of high-cost cancer drugs while improving patient-centered care.

PROGRAM DESCRIPTION:

The Medication Outcomes Center (MOC) in the Department of Clinical Pharmacy, University of California, San Francisco (UCSF), School of Pharmacy focuses on assessing the value of medication-related health care interventions and disseminating findings to the UCSF Medical Center. The High Cost Oncology Drug Initiative at the MOC aims to assess and adopt tools for the critical assessment and amelioration of high-cost cancer drugs. The American Society of Clinical Oncology (ASCO) Value Framework (2016 update) and a cost-effectiveness analysis (CEA) framework were identified as potential tools for adoption. To assess 1 prominent value framework, the study investigators (a) asked 8 clinicians to complete the ASCO Value Framework for 11 anticancer medications selected by the MOC; (b) reviewed CEAs assessing the drugs; (c) generated descriptive statistics; and (d) analyzed inter-rater reliability, convergence validity, and ranking consistency.

OBSERVATIONS:

On the scale of -20 to 180, the mean ASCO net health benefit (NHB) total score across 11 drugs ranged from 7.6 (SD = 7.8) to 53 (SD = 9.8). The Kappa coefficient (κ) for NHB scores across raters was 0.11, which is categorized as “slightly reliable.” The combined κ score was 0.22, which is interpreted as low to fair inter-rater reliability. Convergent validity indicates that the correlation between NHB scores and CEA-based incremental cost-effectiveness ratios (ICERs) was low (-0.215). Ranking of ICERs, ASCO scores, and wholesale acquisition costs indicated different results between frameworks.

IMPLICATIONS:

The ASCO Value Framework requires further specificity before use in a clinical setting, since it currently results in low inter-rater reliability and validity. Furthermore, ASCO scores were unable to discriminate between drugs providing the most and least value.

RECOMMENDATIONS:

The evaluation provides specific areas of weakness that can be addressed in future updates of the ASCO framework to improve usability. Meanwhile, the UCSF Medical Center should rely on CEAs, which are highly accessible for the highlighted cancer drugs. The MOC role can include summarizing and disseminating available CEA studies for interpretation by clinicians and financial counselors around drug value.

What is already known about this subject

• Cancer drug prices are increasing at a faster rate than most other classes of medications and can pose a significant financial burden to patients, hospitals, and managed care settings.

• There are several value frameworks available to assess the cost of cancer drugs relative to the benefits that they provide.

• The ASCO Value Framework assesses the clinical benefits, toxicity, and other values to create a numerical score for a cancer drug, called the net health benefit (NHB).

What this study adds

• The ASCO Value Framework is not yet ready to be implemented in a hospital setting because of low validity and reliability among scorers of the framework.

• Several possible changes could improve the usability of the ASCO Value Framework, such as clarifying the meaning of the NHB score in terms of value decisions and creating a more reliable way to measure the toxicity of a drug.

• Reasons for caution are offered regarding the widespread adoption of the ASCO Value Framework if evaluated by practicing clinicians and used in value-based decision making.

The high price of new anticancer drugs raises questions about whether their value warrants the high prices. When analyzing drug price per survival benefit over time (from 2007 to 2011), the annual drug price increase was 14% for each month of life gained.¹ The price for anticancer drugs has increased by an average of 12% each year from 1996 to 2014.¹ Two million people may forgo needed care because of cost, since the out-of-pocket cost burden for cancer drugs is particularly high.² Payers need to determine how to manage benefits, and health systems are grappling with how to provide quality care while not bankrupting care delivery. The burden and shifting of responsibilities are especially detrimental for high-cost anticancer drugs. Each entity involved with the choice of medications for formularies as they flow from manufacturer to patient, and each entity involved with payment decisions starting with the patient and ending with the manufacturer, are responding to their own incentives to maximize economic benefits while providing health care needs. It can be appealing for health system managers to choose a simple value scoring mechanism, without checking to see if it is reliable and valid. The best approaches to discourage ineffective treatments and violations of pricing norms remain unclear, but each player in the health system needs to be ready to face this problem.

Program Description

The goal of this research is to determine how an academic medical center can begin to address the challenge of high-cost anticancer drugs by employing a new framework to assess drug value. The Medication Outcomes Center (MOC) at the University of California, San Francisco (UCSF) Medical Center focuses on assessing the value of medication-related health care interventions and disseminating findings to the UCSF Medical Center. The High Cost Oncology Drug Initiative at the MOC aims to assess and adopt tools for the critical assessment and amelioration of high-cost cancer drugs. This assessment includes evaluation of the tools available for value-based pricing decisions as a first step to choose a value framework to assist in making value-based cancer drug decisions. The tools evaluated here were the American Society of Clinical Oncology (ASCO) Net Health Benefit (NHB) Value Framework and the “gold standard” cost-effectiveness framework. These tools were analyzed by (a) testing the inter-rater reliability of the 2016 ASCO NHB scores using 11 anticancer drugs and 8 clinician raters; (b) reviewing cost-effectiveness analyses (CEAs) that assessed these drugs; and (c) evaluating the convergent validity of the ASCO NHB scores in comparison with the incremental cost-effectiveness ratio (ICER), which is the “gold standard” value assessment.³

Availability

First, the scientific literature was reviewed to identify the tools available to assess anticancer drug value. Broad value frameworks that involve multiple value evaluations, such as the Institute for Clinical and Economic Review—which includes comparative-effectiveness analysis, CEA, budget impact analysis, and public discussion—were not included, limiting our evaluation to single value framework elements. Four major frameworks were identified: the National Comprehensive Cancer Network Evidence Blocks, Memorial Sloan Kettering Cancer Center’s Drug Abacus, the ASCO Value Framework, and the CEA framework used by the National Institute for Health and Care Excellence (NICE) for drug value assessment in the United Kingdom. The first 2 frameworks are still in the developmental stages with limited guidelines on their use in real-world settings.

Each framework has a unique set of strengths and weaknesses,⁴ and each was initially assessed for the presence of the following attributes: readiness to use now, transparency, target audience, scoring system, method of measuring efficacy and safety, and inclusion of patient-centric metrics (i.e., quality of life). These attributes were chosen based on their use to the medical center for providing guidance to providers on how to facilitate patient-provider conversations about weighing benefits and costs in cancer drug treatment decisions. Based on these criteria, the Drug Abacus and the Evidence Blocks frameworks were considered not transparent about how to implement and score cancer medications and were not ready to use. The ASCO Value Framework and the CEA framework were determined to be the most suitable frameworks, of the 4 identified, for further evaluation, primarily because of readiness, transparency, and patient-centeredness.

To evaluate the frameworks, the study investigators focused on 11 new anticancer drugs that the medical center identified as either added to the UCSF Medical Center’s formulary in the last 2 years or under consideration for addition (Table 1).

TABLE 1.

Descriptive Statistics for ASCO and ICER Value Frameworks: Comparative Ranks and Annualized Cost per Month for 11 Anticancer Drugs

Drug Versus Comparator	ICER Value,a $	ASCO NHB Median	ASCO NHB Range	ASCO NHB Mean ± SD	Relative Rank: ICER/ASCO NHB	Annualized $/Monthb
Ramucirumabx9	12,060	31.1	13.0-39.2	29.2 ± 10.04	1/7	5,123
Plerixafor10	14,693	12.0	-3.4-13.7	7.6 ± 7.8	2/11	2,404
Trabectedin11-13	42,633	29.9	9.7-36.1	26.8 ± 10.4	3/8	3,375
Pembrolizumab14	42,923	52.5	43.1-66	53.1 ± 9.8	4/1	5,188
Carfilzomib15	107,520	29.1	19.5-55.0	32.0 ± 11.6	5/6	15,328
Trastuzumab emtansine16,17	124,247	47.3	33.1-49.1	45.0 ± 6.2	6/2	9,786
Niivolumab18-20	136,215	43.6	34.9-66	44.9 ± 9.9	7/3	5,894
Ipilimumab14,21	152,139	40.3	12.8-58.2	37.9 ± 20.4	8/5	11,265
Elotuzumab22	289,607	22.6	17.6-38.6	26.2 ± 8.5	9/9	15,787
Pertuzumab23	472,668	46.2	30.2-52.2	43.8 ± 8.9	10/4	7,052
Romidepsin	–	26.6	5.5-43.4	25.2 ± 19.0	-/10	6,586

Note: ICER amounts over $100,000-$150,000/QALYS are not considered cost-effective.

^aICER = [(COST rx1-COST rx2)/(QALYS rx1-QALYS rx2)].

^bAnnualized WAC cost per month for the total median treatment episode.

ASCO = American Society of Clinical Oncology Value Framework; ICER = incremental cost-effectiveness ratio; NHB = net health benefit; QALYS = quality-adjusted life-years saved; SD = standard deviation; WAC = wholesale acquisition cost.

Next, a PubMed literature search was performed using the search terms cost and cost-effectiveness and the drug names for any cost-effectiveness studies available that assessed the 11 anticancer drugs to determine the availability of cost-effectiveness information for addressing value questions in cancer therapy decisions. We did not formally evaluate the quality of each published CEA, but each analysis generally followed the International Society for Pharmacoeconomics and Outcomes Research CEA guidelines. CEAs generally compare cost per quality-adjusted life-year (QALY) between 2 alternative treatments. The resulting ICER value is the only value framework calculation (along with net benefit and cost benefit analyses) that integrates differences in cost and differences in effectiveness between 2 or more alternatives into 1 numeric metric. It represents the maximum amount of additional costs necessary for each additional QALY gained when adopting one treatment over another.¹

Usability

The ASCO framework tool uses dimensions of clinical benefit, toxicity, and a bonus area (e.g., palliation benefits, quality of life, and treatment-free intervals) to calculate a final NHB score. The investigators asked 8 pharmacist clinician experts in oncology to complete the 2016 ASCO framework tool for advanced disease for the 11 anticancer medications, using the instructions provided by ASCO and Schnipper et al. (2016) about the updated framework.⁵ Training involved a single meeting where the ASCO framework tool was provided, and the scoring approach described. The clinicians were provided with all phase II and phase III approval studies available for the 11 drugs. The clinical raters were instructed to choose at least 1 of the phase III studies available for each drug when completing the tool; a phase II study was used if phase III studies were not available. Clinical raters were also allowed to access other sources if desired. Scoring was completed independently across raters. The annualized 2016 Cardinal Health wholesale acquisition cost (WAC) was calculated based on the median drug treatment episode per month for an individual weighing 70 kg in order to estimate cost for the ASCO framework. WAC is the list price from a manufacturer to a wholesaler or a direct purchaser without discounts. WAC price was used because it avoids discounts, which can be variable across purchasers; it could be provided by the UCSF Medical Center; and it is a consistent standard that can be compared across institutions.

Reliability and Validity Analysis

Descriptive statistics were used to describe the minimum and maximum rating, median, mean, and standard deviation (SD) across raters for each drug ASCO framework score and subscore.

Cohen’s Kappa coefficient (κ), which measures inter-rater agreement, was used to determine the inter-rater reliability of the ASCO framework total score and each subscore across all raters, as well as the combined score (including clinical benefit, toxicity, and bonus points). Since κ takes into account the agreement occurring by chance, it is a more robust measure than simple percentage agreement. Kappa scores range from zero (no agreement) to 1 (perfect agreement). Statistical significance for κ does not indicate high or low agreement, so interpretation followed published guidelines: slight (0.0-0.20), fair (0.21-0.40), moderate (0.41-0.60), substantial (0.61-0.90), and almost perfect (0.81-1.00).⁶ We used Stata data analysis and statistical software (release 14) for our analysis (StataCorp, College Station, TX).

The convergent validity of the ASCO framework score and the ICER value results from the CEA tool was also determined using Stata software. Convergent validity is a subtype of construct validity referring to the degree to which 2 measures of constructs that should be related (such as value-based decisions) are in fact related. The ASCO framework score for each cancer drug was compared with the published ICER in U.S. dollars using 3 methods. First, the Pearson product-moment correlation was calculated between our ASCO ratings and the ICER value. Second, the correlation between our ASCO ratings and the willingness-to-pay (WTP) threshold was determined. The WTP threshold is the benchmark ICER value used for decisions regarding the value of care when using CEAs. In the United States, this value has ranged from $50,000 per QALY gained to as much as $200,000 per QALY gained, or 3 times a country’s gross domestic product per capita. The more commonly accepted ($100,000) WTP was used for analysis by coding the variable as 1 if the ICER value is less than $100,000 and 0 if the ICER value is greater than $100,000. ICER values less than or equal to $100,000 would be considered by decision makers as cost-effective and would be selected as an acceptable value, while ICER values greater than $100,000 would not be considered cost-effective. Third, we analyzed the rank of the ICER value from the most to the least cost-effective value (lowest ICER value is the most cost-effective) and the mean of clinical benefit, toxicity, and bonus points together with the NHB score from the ASCO framework.

Observations

Flow Diagram

A flow diagram was constructed for the movement of a drug’s use from manufacturer to patients with cancer and then the flow of payment from patient back to the manufacturer. The purpose was to evaluate each pressure point in the system beginning with defining how the UCSF Medical Center could use value frameworks to increase communication regarding value between the physician and payer (Figure 1). The evaluation of the usability, reliability, and validity of the ASCO framework and the ICER score of the CEA informed the approach to facilitating this communication.

FIGURE 1.

Incorporation of Patient-Centered Value-Based Frameworks into Hospital Systems

ASCO Tool Assessment

The mean total advanced ASCO NHB framework score across the 11 drugs ranged from 7.6 (SD = 7.8) to 53 (SD = 9.8). Since the range of possible scores is theoretically -20 to 180, there is concern that only the bottom third or less of the scale was being used; more variability in the scores across different treatments should be present (Table 1).

The Kappa coefficient κ for testing reliability of the NHB total score across the clinical raters is only 0.11, which is considered slightly reliable. The clinical benefit score had the highest κ (0.24), a fair agreement. The toxicity score had the lowest inter-rater reliability, with a κ of 0.06 and was described by the clinician raters as the most difficult area to score. The combined κ score (combining clinical benefit, toxicity, and bonus points) was slightly better at 0.22.

Convergent validity was then analyzed, and the resulting correlation between the mean ASCO NHB scores and WTP based on ICER (being above or below a threshold of $100,000) was also very low (-0.215). Treatments above this WTP threshold are generally considered not to be efficient enough over the alternative treatment to be selected for use. The low correlation between the NHB score and WTP suggests that NHB scores do not provide us with information similar to cost-effectiveness.

Finally, the ICER score was ranked from most cost-effective (highest ICER value) to least cost-effective (lowest ICER value), and the ASCO score was ranked from most to least effective, with 1 being the most cost-effective/highest NHB. The annualized monthly WAC per person was then calculated for each drug regimen using the Cardinal prices (Table 2).⁷ The rank and WAC allowed determination of knowing if the ASCO tool was separating drugs into those with and without value in the same manner as the ICER value. The results show that the ASCO NHB framework score was not consistent with ICER ranks, even when considering the WAC costs of each (Table 2). Therefore, the analysis showed that the ASCO Value Framework is not yet able to provide an accurate mechanism to help evaluate drugs for use to meet the needs of a medical center such as the UCSF Medical Center.

TABLE 2.

ASCO Inter-Rater Reliability and Validity Scores

	ASCO Net Health Benefit	ASCO Clinical Benefit	ASCO Toxicity	ASCO Bonus Points
Inter-rater reliability Kappa score (interpretation)	0.11 (slight)	0.24 (fair)	0.06 (slight)	0.11 (slight)
Convergent validity Pearson’s correlation score (WTP)	-0.21	-0.34	0.01	0.03

ASCO = American Society of Clinical Oncology Value Framework; WTP = willingness to pay.

Implications

The Cancer Care Task Force that created the ASCO Value Framework stated that the 2016 update would not be the final version of the tool.⁵ As such, this evaluation of the currently available ASCO Value Framework identified important lessons to be learned and potentially incorporated into future updates.

Our results show that the ASCO Value Framework is not ready to be used to self-rate cancer drugs using the current ASCO directions in a clinical setting because of low inter-rater reliability (IRR), which means that the framework cannot be consistently scored across raters. Reasons for a low IRR can include characteristics and training of the raters, what is being compared when making the ratings, which clinical trial data is used to inform the ratings, and characteristics of the value framework. Our results highlight the need for value frameworks to specify all of these possible variations in order to obtain high IRRs. Our discussions with raters after their analyses suggest that a primary reason for the low inter-rater reliability found here is because the framework does not specify a comparator drug for the ratings; thus, the scores would naturally differ when raters selected clinical trials that compared survival and toxicities of the particular anticancer drug against different comparators. By simply specifying the comparator(s) for each cancer drug, the tool developers could further standardize the framework, while incorporating multiple scores (when applicable) for some drugs based on the number of comparators that are clinically relevant.

The Kappa coefficient for toxicity, which was 0.06, was the lowest across all categories. This may be a result of the complicated and sometimes confusing directions presented in the framework.⁵ For example, there was confusion among clinicians regarding how to calculate the percentage difference between the toxicity score of the drug being evaluated and the score of the comparator. Future iterations of the framework should clarify this point. Additionally, the toxicity score is biased against clinical trials that are more thorough in reporting adverse events, since all adverse events are scored in increments of 0.5 between 0.5 through 2.0. For example, a drug that has grade 1 or 2 toxicities for an adverse event in 0.3% of patients will have the same score as a drug that has grade 1 or 2 toxicities in 9.9% of its patients for the same adverse event. One way to decrease this bias is to assign weighted scores based on a gradient that directly corresponds to the percentage of patients that have an adverse event.

Ascertaining a score for bonus points—which assigns additional points for the tail of the curve, palliation, quality of life, and treatment-free intervals—within the framework was also problematic. The majority of the clinical trials for the 11 cancer drugs did not report palliation, quality of life, or treatment-free intervals. This pattern may be because these outcomes are often reported in phase IV clinical trials, which are not readily available for new drugs. The issue further emphasizes the need for the ASCO Value Framework task force to set criteria for which clinical trials should be used for evaluating a specific cancer drug.

There was concern that the ASCO scores were not able to discriminate between those providing the most and least value, when compared with the ICER score. Of course, the comparison is not entirely equal considering that the ICER value includes costs and effects and that ICER values include all costs (e.g., hospitalizations, physician visits, and treatment of side effects), not just drug costs. However, it is of concern that there was little variation in ASCO NHB scores across different drugs and also so little correlation in the decisions that one would make if using the ICER value or ASCO frameworks to recommend treatments. Work is needed to more fully evaluate value frameworks and how they will accurately distinguish value.

It is important to note the high accessibility of cost-effectiveness studies for the 11 anticancer drugs used in our evaluation. The benefits of a CEA are that it has a comparator, includes more than just drug costs, usually assesses survival or progression-free survival (in cancer drugs), considers the quality of that survival, and combines these into 1 number for comparison with other CEAs. CEAs are also generally made more transparent by including sensitivity analyses that help determine their accuracy. Given that CEAs are most often readily available and that the ASCO framework has unexplained weaknesses, we concluded that the MOC should rely on the CEA framework at present to communicate value of anticancer drugs to clinicians.

Limitations

The strength of this study is that it is the first to look critically at the ability of the ASCO framework to meet basic reliability and validity criteria. However, this study has several limitations. First, the developers of the ASCO framework indicate that it is not yet ready for use, so it may have been premature to evaluate the 2016 version. In addition, our sample was small, and it may be that other clinicians will do better across raters. These results are the experience of just 1 setting so should be repeated in more settings with different raters. The intent here is to highlight the importance of reliability and validity for future framework updates. We also acknowledge that CEAs, as the gold standard used here, may not be completely fair comparators and have their own flaws in usefulness for decision makers. However, CEAs has been used as valid measures of drug value in decision making for many years and are the best comparators available.

Recommendations

It is important to have a method to define the value of cancer treatments, and the design of the ASCO framework facilitates physician-patient discussion around value, which is an important goal. This study identifies weaknesses in the ASCO framework as it now stands, which affects how useful it can be in cancer drug decision making by medical centers. Since the ASCO framework is meant to assist clinicians in clinical decision making, it is important that this tool perform consistently across users.⁵ The especially low reliability ratings for toxicity and bonus scores suggest that improvements are needed. Specification of a comparator for each rating and recognition that drugs can have multiple ratings depending on this comparator would also improve clarity in the use of the ASCO framework. The ASCO and CEA frameworks are understandably complex, and a software application that is planned for the ASCO framework could help ease the burden of its interpretation. Software tools have also been provided with CEAs to aid in tailoring their results to each unique site of care. Schnipper et al. indicates that others have expressed concerns about the practicality of the ASCO tool.⁵ Given its complexity and that only clinical trials that make head-to-head comparisons should be used by the tool, its utility in a busy physician’s office is a significant concern. In addition, the ASCO inclusion of only drug costs limited our ability to fully compare value with the CEA, which includes other direct costs that might make the total patient cost burden lower, even when the drug cost is higher than an alternative.

The intent here is to provide an assessment of the reliability and validity of the ASCO framework as it might be used in a single medical center and, in so doing, to remind developers, as well as potential users such as payers, managed care representatives, and medical centers, of the importance of reliability and validity to the evaluation of any measurement tool. Future work should evaluate the reliability and validity of the ASCO framework across more drugs, use of more raters, and across more centers. CEAs are available for most drug comparisons and, despite their limitations, are the strongest value framework now available for cancer drugs. The Institute for Clinical and Economic Review currently uses CEA studies as part of its evidence reports on effectiveness and value of drugs.⁸ Critical review of the reliability and validity of its decision making process will be essential as it expands into cancer treatments.

This assessment allowed us to conclude that the ASCO tool could not be easily implemented in our medical center. To make it more useful for clinicians to discuss with their patients, we wanted the MOC to make the ASCO ratings generically and supply them along with the clinical trials upon which the ratings were based in the electronic medical record for easy access by clinicians. The clinicians would then act to personalize them for their patients and discuss value with them. But without adequate reliability across raters, this method of application would not be valid. Alternatively, we may decide to post CEA studies on the electronic medical record instead, with an easily understood interpretation written by the MOC, until the ASCO framework is further validated.