The Accuracy and Usefulness of the National Comprehensive Cancer Network Evidence Blocks Affordability Rating

Abstract

Background:

The National Comprehensive Cancer Network (NCCN) Guidelines’ Evidence Blocks is the most comprehensive value framework in oncology. The Evidence Blocks includes the Affordability criterion, which reflects the financial cost of each treatment on a 1-5 scale. The accuracy of Affordability is unknown.

Methods:

We calculated Medicare costs for all first-line and maintenance treatments for the 30 cancers with the highest incidence in the US that had published NCCN Evidence Blocks as of December 31, 2018. We assessed the accuracy and consistency of Affordability across different treatments and cancer types. Among different treatments for the same indication, we determined the frequency with which Affordability correctly identified the more-expensive treatment.

Results:

There were a total of 1,386 treatments in our sample. Lower Affordability scores were associated with higher costs. There was significant variation in cost at each level of Affordability; for treatments with Affordability = 1 (very expensive) costs ranged from $4,551-$43,794 per month for treatments administered over an undefined time period and ranged from $2,865-$500,982 per course of therapy for treatments administered over a defined time period. Among treatments for the same indication, Affordability incorrectly identified the more-expensive treatment in 7.9% of pairwise comparisons. Inaccuracies were reduced when we reassigned Affordability scores based on calculated treatment costs.

Conclusions:

Evidence Blocks Affordability generally correlated with treatment costs but contained inaccuracies which may limit its usefulness to clinicians in identifying high-value treatments. This study suggests that the accuracy of Affordability may be improved by indexing more directly to specified dollar value thresholds.

1. Introduction

The high and rising cost of cancer drugs has drawn intense scrutiny from payers and policymakers and prompted calls for reform [1-4]. For patients, research suggests that higher out-of-pocket costs reduce adherence to potentially life-saving therapies [5].

In response to this issue, there has been increased emphasis on “high value” cancer care – that which achieves improved outcomes at a reasonable cost. To support clinicians in identifying high-value treatment options, several groups have developed oncology-specific decision tools often referred to as “value frameworks” [6, 7]. Memorial Sloan Kettering Cancer Center’s Drug Abacus tool allows users to assign prices to cancer drugs based on the user’s valuation of drug characteristics such as effectiveness, toxicity, and unmet need [8]. The ASCO Value Framework uses clinical trial data to assign a “net health benefit” score for each treatment, which may be adjusted based on patient preferences [9]. The Institute for Clinical and Economic Review (ICER) has conducted detailed reviews of the relative cost-effectiveness of treatment options for several major cancer types [10].

The most comprehensive value framework is the Evidence Blocks, developed and published by the National Comprehensive Cancer Network (NCCN) [11]. The Evidence Blocks provide a graphical representation of the NCCN Guidelines committee’s evaluation of each treatment recommended in the Guidelines, on the basis of five criteria: efficacy, safety, quality of evidence, consistency of evidence, and affordability. The Evidence Blocks is the most widely-recognized value framework among oncologists, who also believe it to be the easiest to use in a clinical setting [12].

The Evidence Blocks’ affordability assessment (henceforth, “Affordability”) aims to provide information to users regarding the relative costs of treatments. It is intended to be comprehensive of direct treatment costs, including the purchase and administration of drugs, growth factors, antiemetics, and hospitalizations [13], and summarizes them on the following scale: 1 = Very expensive, 2 = Expensive, 3 = Moderately expensive, 4 = Inexpensive, and 5 = Very inexpensive. In conjunction with the other four Evidence Blocks criteria, Affordability may help users gauge the relative value of different treatments, identifying which treatments achieve good outcomes while maintaining lower cost. Because clinicians may not have an accurate understanding of treatment costs [12, 14], Affordability helps to address this important information gap.

Affordability is determined by expert opinion of NCCN Guidelines committee members and is not based directly on drug or treatment prices. Committee members vote on the Affordability of each treatment on the 1-5 scale, and the average value is then reflected in the Evidence Blocks. Therefore, if committee members do not have full information regarding treatments costs, this “price opacity” may be passed on to Affordability. Additionally, the levels of the Affordability score (“very expensive,” “moderately expensive”) do not correspond to specific ranges of treatment costs in dollar terms. These factors may introduce inaccuracy and variation into Affordability. Therefore, we assessed the accuracy of the Affordability score and its usefulness as a tool to help clinicians gauge costs.

2. Methods

2.1. Sample Identification and Affordability Ratings

We included the 30 cancer types with the highest incidence in the US that had published Evidence Blocks as of December 31st, 2018 (Figure 1). For each of these cancer types, we extracted the Affordability score for all first-line and maintenance treatments.

Figure 1: Selection of National Comprehensive Cancer Network (NCCN) Guidelines, treatments, and treatment groups.

Time-unlimited treatments include treatments intended to be given for an undefined time period, and time-limited treatments include those intended to be given for a predetermined time period and/or number of cycles. AML, acute myeloid leukemia.

For each treatment, we identified the dose and the number of doses of each drug, based on the availability of sources in the following hierarchy: (1) dosing schedules specified within the NCCN guidelines, (2) standard dosing levels specified on the drug label, (3) a dosing schedule from the reference or references cited by the NCCN guidelines for each treatment, (4) a manual literature search in PubMed for a dosing schedule corresponding to the relevant indication. If an applicable dosing schedule was not attainable through any of these sources, the treatment was excluded.

If multiple dosing schedules for the same drug were included, we used the schedule with the least-frequent dosing interval and the lowest dose. If multiple combinations of different drugs and/or durations of therapy were provided (e.g. “4 to 6 cycles”), we selected the treatment and duration felt to best represent the current standard of care, as identified through discussion with clinical experts in the relevant cancer type if necessary.

2.2. Cost Calculations

To avoid inconsistencies in the prices paid by different insurers and because Medicare is the single largest payer for US cancer patients [15], we used Medicare prices in our analysis. For physician-administered drugs reimbursed under Medicare Part B, we used the January 2019 Medicare ASP file and identified the payment limit for the Healthcare Common Procedure Coding System (HCPCS) code corresponding to each drug [16]. For oral drugs reimbursed under Medicare Part D, we used the Medicare Plan Finder website to identify the unit retail price, as previously described [1, 17]. We used drug doses corresponding to a 70 kilogram body weight, a 1.7 m² body surface area, and normal renal function, where applicable.

In cases where the NCCN guidelines named a class of drugs (e.g., “aromatase inhibitor”) rather than a specific drug, we used the cheapest option among those currently in common use in the US for use in price calculations. A full description of dosing assumptions can be found in the Supplementary Appendix.

In addition to drug costs, we also included administration fees and supportive care in determining overall treatment costs. Administration fees were determined by using HCPCS codes corresponding to the route of administration and typical durations of infusion for each drug. For supportive care medications, we included both growth factors and physician-administered anti-emetic drugs. We determined which treatments warranted supportive care medications by using the NCCN Myeloid Growth Factor and Antiemesis guidelines. For growth factors, we assumed that treatments with high (>20%) risk of neutropenic fever would receive a single dose of peg-filgrastim per cycle.

2.3. Analysis

Due to the difficulty in comparing costs for treatments administered for a limited duration of time (often, adjuvant or neo-adjuvant treatments) versus those administered for an unlimited duration of time (often, treatments for advanced/metastatic disease), we categorized each treatment as either “time-limited” or “time-unlimited.” For time-limited treatments, we determined the number of doses for the full treatment course, and calculated costs across the full course of therapy. For time-unlimited treatments, we calculated the average monthly cost of treatment.

To assess the accuracy of Affordability, we conducted pairwise comparisons of treatments with respect to Affordability and calculated treatment costs. In each comparison, we characterized Affordability as being “correct” if the less-costly of the two treatments received a higher Affordability score, “same” if the two treatments received the same score, or “incorrect” if the more-costly treatment received a higher Affordability score.

We hypothesized that NCCN Guideline committee members may judge Affordability differently across various cancer types and sub-types, because as different cancers vary greatly in the costliness of treatment. For example, physicians may view a $10,000-per-month targeted drug for kidney cancer to be affordable, in the prior absence of effective treatment alternatives, but also view $10,000-per-month for a new drug treating hormone receptor-positive breast cancer to be very expensive, as effective and inexpensive hormonal therapies already exist. Therefore, we conducted pair-wise comparisons within treatment groups, which we determined to be a group of treatments recommended for the same NCCN-defined indication (for a full description of treatment group definitions, see Supplementary methods). After defining treatment groups, we conducted all possible within-treatment group pairwise comparisons.

We assessed the usefulness of Affordability by comparing its performance to that of other possible methods of assigning the Affordability score. We constructed several hypothetical versions of Affordability, each defined using specific dollar value thresholds. Because physicians interpret NCCN Affordability as conveying cost information in dollar terms [12], we hypothesized that Affordability scores based on denoted price ranges might be useful. For comparative purposes, we created several “alternate” versions of Affordability, each using different, plausible treatment cost ranges for each level of Affordability (Table 1, Table S1). We then compared the actual NCCN Affordability scores to these alternate versions based on the accuracy of correctly identifying cost differences within pairwise comparisons of treatments.

Table 1:

Cost thresholds used to create the alternate version of Affordability ($).

	Time-unlimited treatments					Time-limited treatments
Affordability score	5	4	3	2	1	5	4	3	2	1
Cost range	0 - 1,000	1,001 - 3,000	3,001 - 9,000	9,001 - 27,000	>27,000	0 - 3,000	3,001 - 9,000	9,001 - 27,000	27,001 - 81,000	>81,000

To assess the association between Affordability and treatment cost, we used generalized linear models estimating the calculated treatment cost as a function of Affordability, clustered at the level of the treatment group. We hypothesized that Affordability may not be well modeled as a continuous variable and may be better modeled as an ordinal scale with variable spacing between levels. Therefore, in our model we used indicator variables for each level of Affordability (1-5), choosing one level as referent.

3. Results

As of December 31, 2018, NCCN Evidence Blocks were available for 27 of the 30 highest-incidence cancers. After excluding treatments in which dosing was not available, there were 1,386 treatments in our sample (Figure 1). Of these, 541 were time-unlimited treatments, and 845 were time-limited. We conducted pair-wise comparisons within treatment groups with more than one treatment; of these, there were 519 time-unlimited treatments and 798 time-limited treatments.

Across all cancer types, Affordability scores of 3 were most common (550 treatments), and 5 was the least common (20 treatments) (Table 2). There was significant cost variation within each level of Affordability. For example, monthly costs for time-unlimited treatments with Affordability of 3 ranged from $240 to $12,318. Across different cancer types, there was variation in the mean treatment costs at each level of Affordability (Table S2). For example, the mean cost among time-limited treatments with Affordability of 3 ranged from $2,353 for prostate cancer to $38,012 for chronic lymphocytic leukemia (CLL).

Table 2:

Price ranges and averages at each level of the Affordability rating ($). σ, standard deviation.

	Time-unlimited					Time-limited
Affordability	N	Mean	σ	Median	Range	N	Mean	σ	Median	Range
5	18	23	16	18	4 - 69	2	6,086	0	6,086	6,086 - 6,086
4	98	723	994	629	4 - 7,515	131	4,417	3,016	4,173	498 - 22,130
3	162	2,001	2,544	1,124	240 - 12,318	388	13,706	19,256	6,185	0 - 144,239
2	227	12,548	7,211	11,904	417 - 64,630	290	55,651	91,865	25,788	678 – 775,559
1	36	16,232	8,374	13,771	4,551 - 43,794	34	122,025	118,739	63,094	2,865 - 500,982

There were a total of 4,860 within-treatment group pairwise comparisons. The largest difference in cost between two treatments in the same treatment group that received the same Affordability score was that of high-dose ipilimumab ($775,559) and biochemotherapy ($31,965) for the treatment of melanoma (both treatments had an Affordability of 2) (Table S3). We identified a total of 384 cases (7.9%) where Affordability was incorrect. For example, for the treatment of non-clear cell renal cell carcinoma, the combination of everolimus plus bevacizumab ($30,538 per month) had Affordability of 2 whereas nivolumab ($14,454 per month) had Affordability of 1 (Table S4).

In regression analyses, lower Affordability was generally associated with higher treatment costs, as expected (Figures S1, Tables S5, S6). However, the change in cost associated with a 1-unit change in Affordability was not uniform. For example, among time-unlimited treatments, a change in Affordability from 4 to 3 was associated with a $1,302 cost increase, and a change from 3 to 2 was associated with a $10,287 increase.

Among the alternate versions of Affordability we created, we found that the version which performed the best was that which placed the cut points between levels of Affordability at thresholds increasing by a factor of 3 (Table 1). This alternate version had less variation than the existing Affordability score (Figure 2). When using the alternate version of Affordability to make pairwise comparisons of treatments, accuracy was improved (Figure 3); the alternate version was incorrect in zero cases, a greater proportion of treatment pairs with similar costs (e.g., cost differences <$1,000) received the same Affordability score, and a lesser proportion of treatment pairs with large cost differences received the same Affordability score. Two additional alternate versions (Table S1) resulted in similar improvements in accuracy (not shown). Results were similar whether cost differences between treatments were expressed as differences or ratios (Figures S2, S3).

Figure 2: Distribution of treatment costs and Affordability scores, NCCN Affordability vs alternate version.

Time-unlimited treatments include treatments intended to be given for an undefined time period, and time-limited treatments include those intended to be given for a predetermined time period and/or number of cycles. N = 541 for time-unlimited treatments, N = 845 for time-limited treatments. For the two cases with drug costs of $0, we assigned a cost of $1 to allow for inclusion on a logarithmic scale.

Figure 3: Accuracy of Affordability in comparing treatment costs.

Treatments were divided into treatment groups as described. We then made all pairwise comparisons among treatments within the same treatment group. The X-axis reflects the cost difference, in USD, between the two treatments in each pair. The Y-axis reflects the percentage of treatment pairs with that cost difference for which the Affordability score was correct, incorrect, or the same; Affordability “correct” was defined as the more expensive of the two treatments receiving a lower Affordability rating and “incorrect” as the more expensive treatment receiving a higher Affordability rating. Time-unlimited treatments include treatments intended to be given for an undefined time period, and time-limited treatments include those intended to be given for a predetermined time period and/or number of cycles.

4. Discussion

The NCCN’s Affordability scores were correlated with calculated treatment costs. Treatments assessed as being less affordable were generally more expensive, in line with expectations. However, NCCN Affordability scores also contained inaccuracies which may limit their usefulness for clinicians.

Our results are consistent with a recent study which evaluated the correlation between Affordability and real-world costs in advanced non-small cell lung cancer and identified significant variation in cost within strata of the Affordability score [18]. This study used claims data to obtain total treatment costs for patients receiving different NCCN-recommended treatments; in contrast, the current study used Medicare prices to calculate anticipated costs. Both approaches are valid, answering slightly different questions. As NCCN Affordability aims to be comprehensive of all care costs while receiving treatment [13], claims data may provide the best assessment of what costs are actually incurred with each different treatment. However, claims may also capture costs unrelated to cancer treatment, as well as costs which would occur regardless of which treatment is chosen. If the aim were to assess the costs resulting directly from treatment, then real-world costs may introduce some degree of noise. The current study focused on the directly controllable costs – drugs, administration costs, supportive care – that would be most salient to the provider in making treatment decisions.

The challenge of measuring value in cancer care is significant, and the NCCN Evidence Blocks are an important step forward. Because each NCCN Guidelines committee comprises leading clinical experts in the field, the Evidence Blocks may represent the most accurate appraisal available regarding clinical metrics: effectiveness, safety, and level of evidence. However, treatment costs may be further from committee members’ areas of expertise, and Affordability may be limited by same problem of price opacity that makes accurate cost assessment difficult in clinical practice. Our study demonstrates that the NCCN could improve the accuracy of Affordability by indexing it to treatment costs more directly. This could be done by establishing specific cost thresholds corresponding to each level of Affordability, and then either assigning the Affordability score on the basis of calculated cost alone (e.g., without the input of committee members), or providing treatment costs to committee members at the time they make their assessments. By thus tying Affordability more closely to dollar thresholds, the NCCN could not only improve the accuracy of this measure, but also better meet the needs of Evidence Blocks users, who interpret Affordability scores as corresponding to specific ranges of cost [12].

However, even with such changes, there remain inherent limitations in condensing a wide range of treatment costs onto a 5-point scale. The levels on the scale will inevitably contain treatments with large cost differences between them. For example, even in the proof-of-principle alternative version of Affordability created in this study, time-unlimited treatments costing $9,000 and $27,000 per month of therapy would both receive an Affordability score of 2. The 5-point scale is limited in the amount of information it can contain and fails to convey to the user important information about the magnitude of cost differences between treatments.

An ideal alternative would be simply to inform providers and patients of anticipated treatment costs. This would provide greater granularity than a scale measure, allowing for easier comprehension of treatment costs in financial terms. Providing this information accurately would inevitably present challenges. For example, the complexity of the private insurance market may make it difficult to estimate average treatment costs across all providers and payers. However, payment rates for the single largest payer for US cancer care – Medicare – are publicly available. The current study shows the feasibility of calculating Medicare costs across a large number of treatments. These treatment costs are directly relevant to the Medicare patient population, which constitutes half of US cancer patients, and may still be helpful for privately-insured patients because private and Medicare costs are generally proportional [19]. Medicare costs for standard treatment dosing schedules and patient measurements could be provided within the NCCN guidelines, either instead of or in addition to the 5-point Affordability scale.

Additionally, it may be possible to tailor treatment costs to individual patients. An online tool, in the model of the FRAX or Adjuvant! Online calculators could generate individual treatment costs with minimal additional input from providers, such as patient weight, body surface area, and planned duration of treatment. Such a tool would provide additional specificity in guiding value-based treatment decisions.

Our study was limited by its reliance on Medicare payment data. Costs would be different for patients with commercial insurance, Medicare Advantage, Medicaid, and other insurance types, and for the uninsured. Our study did not include costs associated with radiation therapy and stem cell transplantation, though we addressed this in our comparisons by clustering within groups of treatments that were similar with respect to the use of radiation and/or transplantation. We include only drug and administration costs, and not ancillary costs such as inpatient hospital stays. For time-limited regimens, we calculated costs for the standard duration of therapy, and did not factor in the proportion of patients whose treatment may be truncated due to toxicity or progression. We did not attempt to capture patient out-of-pocket costs, which is a limitation shared with the NCCN Evidence Blocks themselves.

Our work suggests that the NCCN Affordability score could potentially become a more useful and accurate tool for real world treatment decisions by incorporating cost data into the existing expert-opinion driven framework. Alternately, it would be feasible and informative to provide anticipated treatment costs. Increasing cost transparency in this manner would further empower patients and physicians to more effectively pursue high-value treatment decisions and minimize financial toxicity.

Key Points.

The NCCN Evidence Blocks Affordability measure is generally correlated with treatment cost, but also contains many inaccuracies. More expensive treatments are frequently scored as being more affordable than other treatments for the same cancer type. The usefulness of the Affordability measure could be improved by linking it more closely to specific cost thresholds.