The Value of Comparative Effectiveness Research: Projected Return on Investment of the RxPONDER Trial (SWOG S1007)

Abstract

Objective

The objective of this study was to assess the value of research of the RxPONDER study, an ongoing comparative effectiveness RCT designed to evaluate a 21-gene profile in early stage, node-positive breast cancer.

Methods

We developed a disease-based decision-analytic model to compare use of the 21-gene profile versus standard care. Key clinical data were derived from SWOG-8814, an RCT of chemotherapy in lymph node-positive breast cancer. Other model parameters were obtained from published sources. Probabilistic simulations and value of information calculations were used to assess the expected value of sample information (EVSI) and the expected value of sample parameter information (EVSPI).

Results

The cost of the RxPONDER trial is expected to be at least $27 million. The expected value of research of the RxPONDER trial ranged from $450 million to $1. billion, representing a return of 17 to 39 times the projected cost of the trial. The primary objective of RxPONDER, to assess survival, had the largest estimated value relative to other model inputs. The value of RxPONDER increased by $50 million to $100 million after stakeholder input on additional data collection.

Conclusion

The RxPONDER study appears to represent a good investment of public research funds. Stakeholder engagement and assessment of the return on investment should be considered to optimize and quantify the value of comparative effectiveness studies.

Introduction

For women with lymph node-positive hormone-receptor positive (HR(+)) breast cancer, National Comprehensive Cancer Network Guidelines (NCCN) recommend endocrine therapy and adjuvant chemotherapy.[1] Although adjuvant chemotherapy in this patient population has been shown to increase disease free-survival (DFS)[2], treatment is associated with significant toxicities.

A 21-gene expression profile (Oncotype DX^®) has been developed that provides a recurrence score (RS) on a continuous scale from 1 to 100 to reflect the risk of disease recurrence and probability of response to chemotherapy. The 21-gene profile has been shown to be prognostic (risk of recurrence regardless of treatment) and predictive (identifying patients who would benefit from adjuvant chemotherapy) in women with lymph node-negative breast cancer.[3, 4] The benefit of the 21-gene profile in lymph node-positive HR(+) breast cancer is not well defined. There is limited evidence from a recent retrospective analysis of a subset of patients (367 of 927) in a phase 3 chemotherapy trial (SWOG-8814) suggesting that the 21-gene profile may be useful in improving patient outcomes in this population.[5] Despite the promising preliminary evidence, the benefit of chemotherapy for patients with a low risk of recurrence is uncertain and therefore the comparative effectiveness of management using OncotypeDX® vs. current practice (adjuvant chemotherapy for all patients) is an important question for this population.

The RxPONDER (Rx for Positive Node, Endocrine Responsive Breast Cancer) randomized controlled trial (RCT) (SWOG S1007; clinical trials registry NCT01272037) was designed to identify an optimal cut-point for the recurrence score and to provide definitive evidence of the prognostic and predictive value of the 21-gene profile among women with lymph node (LN(+)), hormone receptor positive (HR(+)), and HER2-negative breast cancer. RxPONDER was designed in consultation with an external stakeholder group representing patient advocates, health insurers, expert clinicians, and manufacturers through the Center for Comparative Effectiveness Research in Cancer Genomics (CANCERGEN) study, and includes assessment of patient-reported outcomes and healthcare costs.[6]Comparative effectiveness research (CER) has been defined by the Institute of Medicine (IOM) as “the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat and monitor a clinical condition, or to improve the delivery of care. The purpose of CER is to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population levels.”[7] An important feature of CER is the development of a prioritized research portfolio through input from multiple stakeholders across different backgrounds representing the societal perspective. While CER studies provide potentially large benefits to society, CER may be provide little individual return for specific groups of stakeholders, such as individual payers or providers, and thus the public role of CER (government agency funding such, ie. NIH) is of particular importance.[8, 9] Public and private research funding agencies have limited resources and are faced with difficult investment decisions that will impact evidence generation from CER. Therefore, assessing the value of research in CER is crucial in allowing the prioritization and funding of the CER which will have the greatest value to society. We sought to clarify the value of investing in a large-scale comparative clinical trial, RxPONDER.

Methods

Approach

We developed a mathematical model to project the long-term clinical and economic outcomes of use of the 21-gene profile in guiding chemotherapy decisions for women with lymph node-positive HR(+) breast cancer. Data from SWOG-8814 was used to estimate the patient outcomes (survival), while literature sources were used to identify cost and utility inputs. Our goal for the model was to estimate the clinical and economic value of the information that would be provided from the RxPONDER trial. We used a value of research approach (also known as ‘value of information’, VOI) to estimate these values. The value of information analysis is based on the idea that research is valuable when its results lead to a change in patient management decisions that either improve patient outcomes or are more cost-effective; thus it accounts for both the magnitude of the additional benefit as well as the probability that a decision change will be made. When there is very little uncertainty surrounding the outcomes of a current decision, additional research has little chance of changing that decision and thus the value of research is low. On the other hand, if there is significant uncertainty surrounding the benefits and harms of competing approaches to manage a disease or condition, then additional research has the potential to influence that decision greatly, and the value of research is larger. In this case, the key decision is whether use of the 21-gene assay in this population results in better outcomes compared to current management of women with breast cancer involving lymph nodes. Specifically, we assessed the expected value of sample information (EVSI) which provides the (monetized) value of a trial given the design and sample size of the trial.

Study design and model structure

We constructed a decision-analytic model to consider two interventions: 1) women are screened with the 21-gene assay, then the recommendation of adjuvant chemotherapy depends on the RS, and 2) current standard of care (SOC), in which all women are recommended adjuvant chemotherapy based on NCCN guidelines.[1] We used a decision tree to define patient RS groups, then used partitioned survival methods to estimate the effectiveness of the interventions by calculating mean patient time spent in the disease-free, recurrence, and death health states (among each RS group) (Figure 1). Time spent in the disease-free health state was determined by estimating mean area under the curve of DFS curves for below and above a cut-point in RS at which chemotherapy would be recommended (as to be determined by RxPONDER). Based on an exploratory re-analysis of DFS in the SWOG-8814 trial, we estimated that the cut-point above which chemotherapy would be recommended occurs at an RS of 19, which was used in our base-case scenario. We evaluated healthcare costs from the perspective of a U.S. payer. A patient lifetime horizon (40 years) was used in this analysis. Costs and outcomes were discounted at a rate of 3% based on recommendations for conducting economic evaluations in healthcare in the US.[10]

Figure 1. Decision Tree.

Patients enter the model and receive the 21-gene assay or no testing (standard of care). Those who are tested with the 21-gene assay are recommended adjuvant chemotherapy as determined by the recurrence score. All of those who are not tested are recommended adjuvant chemotherapy consistent with current NCCN guidelines. In both strategies there are patients who do not follow the recommendations.

Patient population

The patient population consists of women with early stage breast cancer, HR(+), HER2(-), and 1-3 positive nodes. The patient population in the model (and in RxPONDER) differs somewhat from the previous SWOG-8814 trial, as that earlier trial included women with HER2(+) disease, women with greater than 3 positive nodes, and only postmenopausal women. The trial population of RxPONDER includes only women with RS≤25; however, our model included women with RS>25 since they also would be tested in clinical practice and incur the cost of the test. We evaluated both costs and outcomes for those with RS≤25; however, we only included testing costs for those with RS>25 since all women in this group would be recommended both chemotherapy and endocrine therapy so there is no clinical choice to be made. The proportion of the population in each RS group was derived from SWOG-8814 trial data.

Disease-free and overall survival inputs

Table 1 shows the input parameters used in the analysis. We calculated Kaplan-Meir (KM) survival curves for disease-free survival based on a re-analysis of SWOG-8814 data for women with a RS≤25 and 1-3 positive nodes. These survival curves were then further stratified by below and above a RS cut-point of 19. Hazard ratios (HR) calculated from the DFS curves were used to estimate treatment effect of adjuvant chemotherapy (HR in RS≤19: 1.13 (95%CI 0.50-2.57), HR in RS>19: 0.80 (95%CI 0.25-2.52)), and the prognostic effect of a RS below the cut-point (RS≤19) vs. above the cut-point (RS>19) (HR 1.84, 95%CI 0.68-5.02). In the base-case scenario, we used data from the DFS KM curves for the first 10 years, and then extrapolated using a Weibull model for disease-free survival. Because limited information on overall survival (OS) was available, we made the assumption based on clinical studies that survival after recurrence was an average of 2 years.[11, 12] Patients who recurred earlier in time (and who were presumably younger) had a slightly higher longer survival after recurrence while those recurring later in time had a slightly shorter survival after recurrence. We used U.S. life tables[13] and a long-term study (30 year follow-up, n= ∼200 per arm) on OS in node-positive women[14] as a guide to extrapolate overall survival beyond 10 years.

Table 1. Model Inputs.

Model Inputs	Base Case	Low	High	Distribution	Reference
Survival
Predictive HR: RS≤19	1.13	0.50	2.57	LogNormal	SWOG-8814
Predictive HR: RS>19	0.80	0.25	2.52	LogNormal	SWOG-8814
Prognostic HR: RS≤19 vs. RS>19	1.84	0.68	5.02	LogNormal	SWOG-8814
Time from Progression to Death (RS≤19) (years)	2	1	3	LogNormal	[11, 12]
Time from Progression to Death (RS>19) (years)	2	1	3	LogNormal	[11, 12]
Predictive HR: RS≤19 (10 years+)	1	0.75	1.25	LogNormal	Assumption
Predictive HR: RS>19 (10 years +)	1	0.75	1.25	LogNormal	Assumption
Utilities (Quality of Life)
Progression Free, no chemotherapy	0.8	0.7	0.9	Beta	[12, 15, 16]
Progression Free, chemotherapy	0.5	0.4	0.6	Beta	[12, 15, 16]
Recurrence	0.3	0.2	0.4	Beta	[12, 15, 16]
Proportion of patients in risk categories and receiving chemotherapy
in each risk group:
RS≤19	51%	38%	64%	Dirichlet	SWOG-8814
RS:19-25	16%	12%	20%	Dirichlet	SWOG-8814
RS>25	33%	25%	41%	Dirichlet	SWOG-8814
recieving chemo:
% Receiving Chemotherapy: 21-gene assay Arm, RS≤19	45%	10%	80%	Uniform	[20-23]
% Receiving Chemotherapy: 21-gene assay Arrm, RS>19	95%	80%	100%	Uniform	[20-23]
% Receiving Chemotherapy: SOC Arm	95%	80%	100%	Uniform	[20-23]
Costs
Cost of Adjuvant Chemotherapy	$28,770	$24,492	$33,047	Normal	[18]
Cost of 21-gene Assay	$3,800	$2,850	$4,750	Normal
Cost of Primary Prophylaxis with GCSF	$15,526	$11,645	$19,408	Normal	[32]
Cost of Secondary Prophylaxis with GCSF	$5,702	$4,277	$7,128	Normal	[32]
Cost of Recurrence	$51,137	$38,353	$63,921	Normal	[11]

Quality-of-life

Utility values based on multiple literature sources[12, 15-17] were applied to the modeled life expectancies to obtain quality-adjusted life-years (QALYs).

Cost inputs

The cost of adjuvant chemotherapy were based on a retrospective claims analysis.[18] The attributable cost of a distant recurrence was estimated based on an average length of the recurrence of 2 years,[11] and the cost of future recurrences were discounted back to present value. All costs were updated to 2010 prices using the medical consumer price index.[19]

Patient preferences regarding use of chemotherapy

Although chemotherapy is recommended for women with lymph node-positive breast cancer, in clinical practice not all patients will follow that recommendation.[20-23] In patient behavior studies in lymph node-negative women, patients tended to follow the RS recommendation, with only 2-9%[20, 22, 23] choosing to receive chemotherapy despite having a low RS. In node-positive women, however, the risk of recurrence of a low RS may be similar to that of a woman with node-negative disease but a high RS (∼10-25%).[5, 24] Given that there is a demonstrated benefit of adjuvant chemotherapy in LN(+) women and based on patient behavior studies in node-negative women,[20-23] we assume that a significant proportion of women (i.e., about 45%) would perceive the risk of forgoing adjuvant chemotherapy as too large and, therefore, would want to receive chemotherapy despite having a low RS. However, due to the lack of information specifically in node-positive women, we used a wide range (10-80%) in our sensitivity analysis, described below.

Study outcomes

The main outcome of this study was the value of the RxPONDER trial measured in U.S. dollars. The estimated health outcomes (QALYs) gained from the model, were converted into currency by using a incremental net benefit (INB) approach,[25] where QALYs are monetized by multiplying by the willingness to pay threshold (WTP), to give a monetized health benefit value. The cost is then subtracted from the monetized health benefit to give the incremental net benefit.

$$\text{INB}=\text{WTP}\times \text{QALY}-\text{Cost}$$

The INB estimates were then used in the value of research calculations described in the next section.

Value of research calculations

To estimate the expected value of sample information (EVSI), we followed the methodology outlined by Ades et al.[26] This approach is based on a Bayesian framework and involves simulating the inputs in the model across their distributions. Survival parameters and hazard ratios were assigned log-normal distributions. Utilities and probabilities (with the exception of behavioral inputs (e.g., percent receiving chemotherapy), see below) were assigned to beta distributions, while costs were varied under normal distribution assumptions. For patient behavioral inputs, we assigned a uniform distribution, as there currently is a lack of information (as discussed above) regarding their likely mean values in node-positive patients.

The EVSI calculation takes into account gathering new information for all the parameters in the model and estimates the value of a trial of a specific sample size. We additionally assessed the expected value of sample parameter information (EVSPI), which estimates the value of a trial of a specific sample size which will only collect information on particular parameters in the model.

The original protocol for RxPONDER aimed to collect information on survival parameters, however after feedback from external stakeholders, the protocol was modified to include collection of data on other parameters including utilities, costs and patient preferences. Therefore, because the stakeholder informed RxPONDER trial will collect information on all parameters in the model, the EVSI calculation represents the value of the RxPONDER trial. On the other hand, the EVSPI calculations represent specific components of the RxPONDER trial, including the original protocol to collect data only on survival parameters, as well as parameters collected as a result of stakeholder feedback including utilities, costs and patient preferences.

The value-of-research analysis results were projected based on an individual level estimate extrapolated to the U.S. population over a 10-year time frame using the Surveillance, Epidemiology and End Results (SEER) database[27] and then applying to U.S. census data.[28] We estimated results for three different potential societal willingness-to-pay (for a QALY) thresholds ($100,000, $150,000 and $200,000). The thresholds represent how much society might be willing to pay for a gain in QALYs, with the $100,000 threshold being the most commonly cited for most disease states. Some have argued that a higher threshold such as $150,000 and $200,000 may not be unreasonable in oncology.[29, 30] At least 1 million simulations were run in the EVSI analyses to ensure convergence of results.

Results

Value of the RxPONDER trial

Given the study design of RxPONDER (in which information on survival,utilities, costs and patient preferences are being collected, and which aims to recruit 2000 patients per treatment arm), the EVSI for an individual patient at various willingness to pay thresholds was $2,800 ($100K/QALY threshold), $4,700 ($150K/QALY threshold) and $6,500 ($200K/QALY threshold). The potential affected population (accounting for HR status, HER2 status and lymph nodes) was estimated to be 20,600 per annum over 10 years. Thus, by taking into account all current and future breast cancer patients who could benefit from the information provided by the RxPONDER trial, the estimated societal value of this trial ranged from $450 million to $1.05 billion (Figure 2). The estimated cost to the NIH of the study is $27 million, which will includes investigator costs to track patients and the testing costs that are not covered by patients' insurance.

Figure 2. Value of the RxPONDER Trial.

The “Survival” parameter represents the expected societal value of the trial design before stakeholder input. The expected value of the trial after incorporating stakeholder input is represented by the “RxPONDER” parameter. The values of individual components of the RxPONDER trial are also shown under “Patient Preferences”, “Costs” and “Utilities.”

The estimated societal value of RxPONDER based on the original protocol (which aimed to primarily collect information on survival parameters only) ranged from $400 million to $960 million (Figure 2). After stakeholder input, the additional parameters collected increased the value of the trial by $50 million to $100 million. The value of the additional parameters individually had little to no value, with the exception of patient preferences, which had an estimated societal value of $2.1 million to $5.5 million.

Discussion

We evaluated the potential value of research directed at understanding the benefits and harms of the 21-gene profile in women with 1-3 positive lymph nodes compared to current therapy, and the societal value of research of the RxPONDER trial. Our results indicate that research on the use of the 21-gene profile in this patient population is likely to provide high value to society. Specifically, the value of the RxPONDER trial – including clinical, patient, and economic impacts—ranged from $450 million to $1.05 billion. This represents a projected return on the investment (ROI) of 17 to 39 times the NIH trial cost, suggesting the study is a good investment in research resources. These findings were driven by 1) the high level of uncertainty in outcomes based on current evidence, 2) the high incidence of breast cancer, and 3) the severity of clinical and economic outcomes in lymph node-positive disease. Altogether, these results confirm the aims and objectives of the RxPONDER study which include collecting information on clinical, patient and economic outcomes associated with the 21-gene assay.

The main objective of RxPONDER - to collect information on clinical (survival) outcomes - is supported by our analysis, which indicates collecting information on survival parameters has the largest value relative to other parameters in the model. Furthermore, our analysis indicates that modifications to the study design based on external stakeholder input added $50 million to $100 million to the value of the study. Interestingly, the added research objectives of the trial based on stakeholder input don't provide much value individually (relative to the overall trial), indicating that these parameters alone are not sufficient to change patient care decisions; however, when combined with information on survival outcomes they can provide additional information to influence patient care decisions. Additionally, the value of research increases as the assumed societal willingness to pay for a QALY is increased. This is due to the driver of the value of research, survival, influencing patient outcomes (QALYs) greater than costs, and by definition patient outcomes are more relevant at higher willingness to pay thresholds. This finding implies that CER in areas of greater importance to stakeholders – where there may be a higher willingness to pay for benefits – may be more valuable.

The role of VOI in shaping research prioritization is a developing area of research. In this case, we performed the VOI of RxPONDER after the design of RxPONDER was determined through the involvement of multiple stakeholders. This was done to not only educate stakeholders on the concept of VOI to impact decision making, but also to evaluate the potential place of VOI in the research prioritization process. In the future, it would be ideal to use VOI to assist stakeholders throughout their decision making process, with the potential steps as follows: 1) seek initial feedback on study design from stakeholders 2) perform VOI analysis based on stakeholder feedback, and 3) present VOI analysis to stakeholders to determine final study design. Additionally, there is a potential role for VOI analyses at large public funding agencies to assess the return on investment for larger studies. Whether such analyses would be conducted by study investigators or independently is one of the important issues that need to be addressed, as discussed at a recent NHLBI workshop on the topic. [http://www.nhlbi.nih.gov/meetings/workshops/info-modeling.htm]

We did not conduct an analysis of the investment across multiple disease areas and technologies; given limited funds for federally supported clinical research, there could be other trials that would provide a greater ROI. Additionally, because VOI analyses rarely have been used to date in the United States to prioritize research, it is difficult to predict how these results would influence decision making (i.e., are the values large enough to prioritize this research?). However, in a stakeholder-driven prioritization exercise, we found differences in EVSI/EVSPI estimates of 1 to 2 orders of magnitude across research areas influenced investment priorities in cancer genomics (Carlson J, unpublished results). Thus, we expect that VOI analyses may be useful for identifying studies that provide either particularly high or low value, but will be less useful for differentiating between studies with relatively similar VOI. In this analysis, the absolute value of the RxPONDER trial was large, thus it from a societal perspective, there is a strong case that this is a good investment.

Our findings on the relative importance of further research on clinical outcomes is in agreement with a recent study by Hall et al.[31] In this study, they performed an expected value of perfect information (EVPI) analysis, and found that research on recurrence rates was at least 10 times more important compared with other types of evidence. However, Hall and colleagues calculated a hypothetical upper bound estimate (EVPI),[31] and did not account for the specific study design and sample size of RxPONDER.

Our analysis has a number of limitations. First, our study was informed by limited information on overall survival and a lack of information on the decisions that women with lymph-node positive disease will make regarding adjuvant chemotherapy, even if they are in a low risk group. In regards to overall survival, the benefit of chemotherapy in the low RS group may be overestimated due to the small sample sizes. In regards to patient decisions, we attempted to address this issue by using a wide range of estimates based on the choices that have been observed for women with lymph node-negative breast cancer.[20-23] The specific chemotherapy regimen used in SWOG-8814 (cyclophosphamide/doxorubicin/fluorouracil (CAF)) is no longer a preferred treatment,[2, 5] and thus our analysis may underestimate the health benefits and costs of current chemotherapy regimens that will be used in the RxPONDER trial. Another potential limitation was the difference in the overall trial population in SWOG-8814 (the data source) differed slightly from RxPONDER and our model. Despite this, we limited the influence this had on the results by analyzing a specific subgroup of SWOG-8814 trial data to populate our model. We also did not use VOI analyses to evaluate the return on investment of different sample sizes for the trial, as the sample size was driven by survival effect size estimates and established before the VOI analyses were performed. Lastly, another limitation is that we did not include indirect costs such as the cost of lost productivity or patient time. By excluding such costs our estimates may be conservative and therefore including the indirect costs may increase the value of research if uncertainty in these estimates were resolved by additional research.

In conclusion, our findings indicate comparative effectiveness research on the use of the 21-gene profile in women with lymph node-positive breast cancer is of relative large importance. Because of the significant uncertainty and subsequent value surrounding further research on clinical outcomes, the RxPONDER trial represents a valuable research investment. Comparative effectiveness research studies by necessity often will be relatively large and expensive; stakeholder engagement and assessment of the return on investment should be considered to optimize and quantify their value.