Development and Evaluation of an Approach to Using Value of Information Analyses for Real-Time Prioritization Decisions Within SWOG, a Large Cancer Clinical Trials Cooperative Group

Abstract

Objective

Value of information (VOI) analyses can align research with areas with the greatest potential impact on patient outcome, but questions remain concerning the feasibility and acceptability of these approaches to inform prioritization decisions. Our objective was to develop a process for calculating VOI in “real time” to inform trial funding decisions within SWOG, a large cancer clinical trials group.

Methods

We developed an efficient and scalable VOI modeling approach using a selected sample of 9 randomized phase II/III trial proposals from the Breast, Gastrointestinal, and Genitourinary Disease Committees reviewed by SWOG’s leadership between 2008 and 2013. There was bidirectional communication between SWOG investigators and the research team throughout the modeling development. Partial expected value of sample information for the treatment effect evaluated by the proposed trial’s primary endpoint was calculated using Monte Carlo simulation.

Results

We derived prior uncertainty in the treatment effect estimate from the sample size calculations. Our process was feasible for 8 of 9 trial proposals and efficient: the time required of 1 researcher was <1 week per proposal. We accommodated stakeholder input primarily by deconstructing VOI metrics into expected health benefits and incremental healthcare costs and assuming treatment decisions within our simulations were based on health benefits. Following customization, feedback from over 200 SWOG members was positive regarding the overall VOI framework, specific retrospective results, and potential for VOI analyses to inform future trial proposal evaluations.

Conclusions

We developed an efficient and customized process to calculate the expected VOI of cancer clinical trials that is feasible for use in decision making and acceptable to investigators. Prospective use and evaluation of this approach is currently underway within SWOG.

Setting priorities for future medical research is a crucial but often complex process. Value of information (VOI) analysis has received increasing attention as a framework to inform research prioritization decisions.^1-5 VOI analyses quantify the benefit of collecting additional information to reduce or eliminate uncertainty in a specific decision-making context. The first published example using VOI methods to inform research prioritization in the United States showed that the results influenced the priority rankings of research topics across a range of cancer genomic applications.¹ Following the promise of this work and in response to national calls for improved and more systematic research prioritization within cancer clinical trials cooperative groups under increasingly strained budgets,⁶ we sought to integrate VOI analyses into the trial proposal evaluations of SWOG, a large clinical trials cooperative group.

Although VOI results have been shown to affect research prioritization decisions, it can be time and resource intensive to generate de novo decision analytic models and conduct literature searches for model inputs, making traditional, stand-alone modeling efforts unsuitable for integration into SWOG’s evaluation processes.¹ The objective of this study was therefore to refine a minimal modeling approach to VOI and develop an efficient process to estimate the potential VOI of proposed clinical trials that is both acceptable to SWOG decision makers and feasible for integration within SWOG’s proposal evaluations.

Herein we describe the development of our process to calculate VOI using a sample of SWOG’s recently reviewed trial proposals and we provide a preliminary assessment of its feasibility, strengths, and weaknesses. Our findings contribute to the growing literature on developing and implementing quantitative approaches to inform prioritization decisions within US-based publicly funded research organizations. An evaluation of whether and how the VOI results generated from this process affect prospective trial proposal evaluations is currently underway within SWOG.

METHODS

Setting

This work was conducted as part of a Patient- Centered Outcomes Research Institute (PCORI)–funded project evaluating a structured approach to prioritizing cancer research using stakeholders and VOI within SWOG. SWOG maintains a diverse portfolio of clinical trials across the cancer care spectrum. ⁷ Within SWOG, research study ideas are proposed and developed by members from organ-based committees (e.g., lung, breast) and, if reviewed favorably by the committee, are submitted to SWOG’s Executive Review Committee (ERC) for a final internal review. If SWOG approves of the study, it is then forwarded to the Cancer Treatment Evaluation Program at the National Cancer Institute.

Overall Approach

Our initial approach was primarily informed by the literature on applications and methodological approaches to VOI^1,2,5,8 and the conceptualization of minimal modeling by Meltzer and others.^4,9 Minimal modeling VOI calculations can be performed without full disease and/or decision analytic modeling if a prior study characterizes uncertainty in comprehensive measures of health outcomes (e.g., quality-adjusted life years [QALYs] and costs) that are sufficient to conclude whether one outcome of a decision is better than another.⁹ We refined our process using several rounds of feedback from SWOG members regarding the type of information and modes of presentation they would find most compelling. In so doing, we moved away from VOI approaches and presentation styles typical in academic journals and toward a pragmatic framework that is efficient to calculate and customized to SWOG decision makers’ needs and preferences.

Study Sample

Our analyses included proposals from the Breast, Genitourinary, or Gastrointestinal Disease Committees, 3 of the largest and most active committees within SWOG. We obtained all (n = 34) randomized phase II or phase III trial proposals from these disease committees that were reviewed by SWOG’s ERC between 2008 and 2013. A member of our study team who was not involved in the VOI analyses selected 10 proposals to be used for our modeling development. One “randomized” phase II study (not approved by the SWOG ERC) did not include an alternative hypothesis in favor of one treatment; rather, it was in favor of both treatments relative to a historical control and was excluded from our VOI process development.

VOI Calculations

We estimated the expected value of sample information for each of the proposed trials in our sample portfolio using Bayesian decision theoretic methods. ¹⁰ The expected value of sample information is the societal benefit of acquiring additional evidence from a sample to inform a decision. VOI analyses put a value on reducing uncertainty by calculating how the outcomes of decisions made with future evidence might differ from those made today and can thereby help align investments with areas in which research would have the greatest potential effect on patient outcomes. The VOI of a clinical trial is a function of 4 key elements: 1) the current level of decision uncertainty (i.e., the probability of making suboptimal treatment decisions based on current knowledge); 2) the consequences of making a suboptimal treatment decision in terms of a patient’s life expectancy, quality of life, and/or healthcare costs; 3) how much new information would be collected in the trial; and 4) the number of future patients likely to face the decision.

To quantify each of the elements above, we first created decision models for the treatment intervention(s) evaluated in each trial proposal and characterized the uncertainty around model inputs with probability distributions. We then simulated the range of expected trial results according to these probability distributions under the trial’s planned sample size and length of follow-up. We synthesized the simulated trial results for the primary endpoint with the existing (prior) evidence by characterizing uncertainty in existing evidence using distributions that were conjugate to the likelihood of the simulated trial data and in line with the assumptions used in the trial’s sample size calculations.¹⁰ We then compared the QALYs gained and incremental healthcare costs of decisions made with the additional evidence from the proposed trial to those made with only existing evidence. Finally, we estimated the size of the relevant patient population expected to face the treatment decision being investigated by the trial over the expected lifetime of the information. The size of the population was derived the Surveillance, Epidemiology, and End Results database and published literature. In our base case, we assumed a bounded 10-year time horizon for the information being generated by each trial^11,12 and we explored alternative values in sensitivity analyses; all analyses used a 3% discount rate and incorporated a delay in the acquisition of information corresponding to the accrual and follow up time of the trial.¹³ All analyses were performed in R software.¹⁴

Stakeholder Engagement and Analysis

Stakeholders were members of SWOG, including clinical trialists, clinicians, statisticians, and patient advocates, whose acceptance and cooperation with the VOI analyses are essential for the prospective implementation and/or members who have a vested interest in the outcomes of this work.^15,16 We engaged SWOG members from the ERC and the Breast, Genitourinary, and Gastrointestinal Disease Committees (approximately 200 total) in an iterative and multifaceted manner to actively solicit their preferences and needs (Figure 1). Our goals were to create shared understanding of VOI analyses and make transparent and effective decisions regarding its future use in SWOG’s proposal evaluation process.¹⁶ We conducted an in-person VOI training session that was open to all interested SWOG members and solicited feedback during the Spring 2014 SWOG meeting, participated in 3 Web-enabled conference calls with SWOG ERC members, validated key inputs of specific decision models with the chairs of each disease committee, and presented the final VOI process and results of the retrospective analyses to each of the disease committee members during the Fall 2014 SWOG meeting. Throughout the process, we also had ongoing informal discussions with the chairs of each disease committee and the current and previous chairs of SWOG.

Figure 1.

Overview of VOI modeling process and engagement with SWOG members. VOI, value of information.

To increase familiarity with VOI concepts, we distributed written educational materials and provided access to online training that covered VOI techniques, interpretation, and applications (see Appendix) prior to each in-person meeting or Web-enabled conference call. During engagement sessions, all stakeholders were asked to comment on key assumptions or perceived barriers to generating or communicating VOI results identified a priori by the modeling team, but they were encouraged to ask questions or discuss any other concerns or priorities. Field notes from each meeting were analyzed using a targeted thematic analysis.¹⁷

RESULTS

Characteristics of the 9 proposals used to develop and evaluate our VOI modeling approach are summarized in Table 1. Three trial proposals used time to death as the primary endpoint, 4 used time to progression, and 2 used time to recurrence. The median sample size was 680 (range, 92–3400).

Table 1.

Summary of 9 Trial Proposals Reviewed by SWOG’s Executive Review Committee between 2009 and 2013 That Were Used to Develop Our VOI Analyses Processes

Proposal ID	Trial Proposal Title	Phase	Sample Size	Committee	Endpoint	Year Reviewed
A	Prospective Evaluation of the Benefit of a Standard Versus An Extended Pelvic Lymphadenectomy Performed at Time of Radical Cystectomy for Bladder Cancer With Adjuvant Chemotherapy Administration for Node-Positive Disease	III	630	Genitourinary	PFS	2010
B	A Phase III Randomized Trial Comparing LHRHa + TAK-700 With LHRHa + Bicalutamide in Patients With Newly Diagnosed D2 Prostate Cancer	III	1486	Genitourinary	OS	2011
C	A Randomized Phase II Pilot Study Prospectively Assigning Treatment for Patients Based on ERCC1 for Advanced/Metastatic Gastric Cancer or Gastroesophageal (GE) Junction Cancer	II	200	Gastrointestinal	PFS	2010
D	Randomized Phase II Clinical Trial of AZD-6244 and MK-2206 v. mFOLFOX in Patients with Metastatic Pancreatic Cancer after Prior Chemotherapy	II	120	Gastrointestinal	OS	2011
E	Randomized Phase II Study Comparing the Novel MEK Inhibitor, Trametinib, to Standard of Care Chemotherapy in Patients With KRAS Mutant Metastatic Colorectal Cancer	II	92	Gastrointestinal	PFS	2013
F	Exemestane v. a Combination of Exemestane and the Monoclonal Antibody IGF-1R Inhibitor IMC-A12 in Patients With Metastatic ER/PgR Positive Breast Cancer	III	690	Breast	PFS	2009
G	Capecitabine and Dasatinib as Adjuvant Therapy in Patients with HER-2/neu Negative Breast Cancer	III	720	Breast	RFS	2008
H	Adjuvant Endocrine Therapy +/− Everolimus in Patients With High-Risk, Node-Positive, Hormone Receptor Positive and HER2-neu Normal Breast Cancer	III	3400	Breast	RFS	2011
I	Intensive v. Less Intensive Dosing of Zoledronic Acid v. Denosumab as Adjuvant Therapy for Early Stage Breast Cancer	III	680	Breast	OS	2010

OS, overall survival; PFS, progression-free survival; RFS, recurrence-free survival.

Implementing a Minimal Modeling Framework

Our initial intention was to sample directly from distributions representing uncertainty in comprehensive health outcomes for each intervention evaluated; however, 6 of the 9 trial proposals used an intermediate endpoint, such as progression-free survival or disease-free survival, and did not report a mapping function to comprehensive outcomes. Furthermore, for 5 trial proposals, the endpoints were expected to occur over a timeframe of ≥5 years, necessitating a framework that could account for age-specific competing causes of death.

We therefore developed a Markov model framework that focuses on the trial’s primary endpoint and consists of up to 3 health states (Figure 2): 1) alive, preprimary endpoint, 2) alive, postprimary endpoint, and 3) death. We derived the transition probabilities of experiencing the trial’s primary endpoint for the control arm from the relevant survival parameters included in the trial proposal by assuming a constant failure rate (i.e., an exponential distribution) following the assumptions used in the sample size calculations. In sensitivity analyses, we also examined alternative plausible survival distributions (see Appendix for details). Determining the transition probabilities for the new or experimental treatment, particularly the attendant uncertainty, was more complex and is described in the next section.

Figure 2.

Depiction of the Markov modeling framework used in our value of information calculations. Individuals enter the model as “alive, preprimary endpoint” state in the same way they enter the proposed clinical trial. They can remain in this health state, experience the primary endpoint of the trial (e.g., recurrence or progression) and thereafter be in the “alive, postprimary endpoint” state, or die. Patients who experience the primary endpoint of the trial remain in the “alive, postprimary endpoint” health state until death. When the primary endpoint of the trial is overall survival, the health states for “alive, postprimary endpoint” and “dead” are collapsed into a single state.

We extrapolated from intermediate endpoints to death for 6 trial proposals using data derived from meta-analyses that summarized the empirical relationship, and uncertainty therein, between surrogate endpoints and death in the respective disease setting (see Appendix for details and examples). Finally, we focused on cost and utility estimates that were appropriate for broadly defined health states, such as the annual net costs of ongoing cancer treatment for a specific disease setting.¹⁸

Characterizing the Prior Distribution of Treatment Effect

We originally hoped that proposals would contain sufficient data from earlier studies to derive an empirical prior distribution of the treatment effect; however, no proposals included directly translatable prior evidence (e.g., an earlier phase study of the same treatment[s] in a similar clinical setting) or used meta-analytic techniques to comprehensively summarize the current evidence and its attendant uncertainty. We therefore devised an alternate approach that leverages the evidence synthesis and implicit expert opinion used in the trial’s sample size calculations and the historical outcomes of past cooperative group cancer trials. Specifically, just over one-half of all late phase cooperative group clinical trials, most of which evaluated overall or event-free survival as the primary endpoint, found new treatments to be at least marginally better than the control and approximately one-quarter found a statistically significant result in favor of the new treatment defined according to the primary outcome specified in the trial protocol.^19-21 We therefore constructed a prior distribution of the treatment effect estimate for the trial’s primary endpoint by fitting a distribution in which the assumed values under the null and alternative hypothesis align with the appropriate percentiles (i.e., 60th and 25th in our retrospective analysis). Furthermore, SWOG members confirmed that these estimates were reasonable benchmarks for the trials in our sample and agreed to provide individual-level estimates of these probabilities elicited from the entire disease committee for future trial proposals in the prospective evaluation.

Feasibility of Applying a Modified VOI Modeling Framework

Based on information in the trial proposal, the trial’s design and stated purpose, and informal discussions with clinical experts, we determined that our modeling framework could be applied to 8 of the remaining 9 proposals. Our feasibility criteria were that the model captured the key expected differences in comprehensive outcomes either directly via the treatments’ impact on the primary endpoint (n = 2) with adjustment for quality-of-life effects or indirectly if a mapping function existed to link the surrogate endpoint to a comprehensive outcome (n = 6). Our modeling framework was not appropriate for 1 trial proposal to evaluate interventions expected to have important differences in quality of life that would affect treatment decisions, but which only included overall survival as a primary endpoint and because there was no appropriate mapping function from overall survival to quality-adjusted life expectancy (proposal I).

The VOI analyses took 1 researcher 1 to 2 days per trial proposal once the final process was developed. The majority of this time was spent reviewing the literature for appropriate values and validating these with clinical experts and other stakeholders. After reviewing the models, the disease committee chairs confirmed that our models were acceptable, were informative, and sufficiently captured the key expected differences in outcomes between the treatments under study in these 8 proposals.

Aligning VOI Results With Stakeholder Preferences

The feedback we received from SWOG members led us to deconstruct the clinical and economic components of traditional VOI calculations. A slight majority of SWOG members expressed the sentiment that SWOG’s mission was to conduct trials that had the greatest potential to improve health regardless of costs (“You can’t say an improvement in survival is a negative because of costs. It’s matter of philosophical differences between clinicians and health economists.”). Others felt that drug prices were ultimately out of SWOG’s control and were often difficult or impossible to know before a trial started, particularly for new investigational agents.

A slightly smaller group of SWOG members argued that ignoring costs would be “naïve and shortsighted,” given the extremely high cost often associated with new cancer treatments. They felt that costs of many cancer therapies had reached a tipping point and that it was critical for SWOG to start considering how their investments could alleviate or aggravate this problem. Other members pointed out that some trials, particularly those in early-stage cancers, were evaluating interventions to prevent progression and that a key secondary benefit would often be reduced downstream medical costs (“What if you were saving costs? Some trials in early-stage bladder cancer would do just that…it would reduce downstream medical costs”).

To accommodate these divergent preferences regarding treatment costs, as well as mirror usual treatment adoption decisions in the United States, the decision to adopt a treatment within our simulations is made on the basis of health benefits (i.e., QALYs) rather than net monetary benefits. We report the expected incremental health benefits and healthcare costs associated with acquiring additional information from a trial; how ever, the incremental costs are those incurred following decisions made according to health benefits alone. In other words, we did not explicitly incorporate a willingness-to-pay threshold in our VOI calculations.

We also assumed that the control arm of each trial was the current standard of care, following the design of the trials, and established a minimum probability the new intervention was more effective that was required to change clinical practice. This was 80%, corresponding to the upper bound of clinical equipoise and a point beyond which SWOG members felt a clinical trial would no longer be necessary to inform clinical practice. Other studies have found a similar threshold for clinical equipoise among different populations, such as institutional review board members²² and the public.²³

Finally, several SWOG members expressed concerns that population-level VOI estimates would undervalue trials in rare cancers (“If you compare all the trials in pancreatic cancer versus all the trials in breast cancer, you’ll of course find higher VOI in breast cancer”). To accommodate these concerns, we report both patient-level and population-level VOI measures for each trial proposal (Table 2) so that SWOG members can explicitly consider the relative value to individuals and society, and the tradeoffs therein, when making investment decisions.

Table 2.

Patient Level and Population Level VOI Results for 9 Retrospective Trial Proposals Used to Develop Our Modeling Process^a

Proposal ID	Patient Level		Population Level
	Incremental QALYs	Incremental Healthcare Cost, $	Incremental QALYs	Incremental Healthcare Cost, $
A	0.438	1,800	33,000	130 million
B	0.147	92,000b	10,100	6.32 billionb
C	0.092	32,200	4,900	1.70 billion
D	0.160	30,500b	21,300	4.06 billionb
E	0.094	15,800	10,500	1.77 billion
F	0.481	54,000b	65,500	7.35 billionb
G	0.258	23,200	42,300	3.81 billion
H	0.302	24,800	20,800	1.71 billion
I	–	–	–	–

QALY, quality-adjusted life year

^aExpected incremental QALYs gained and expected incremental healthcare costs are shown separately; see Methods section for more details.

^bCalculated using benchmark prices for interventions without a market price (i.e., prior to US Food and Drug Administration approval); see Appendix B for details.

Final VOI Modeling Process

Our final process for calculating VOI for SWOG’s trial proposals is outlined in Figure 1. In brief, we build an initial decision model based on the information included in the proposal and characterize uncertainty using the above-described methods. We then verify the model with clinical experts from the respective disease committee and create a final model. An overarching component of the process is the training of SWOG members in VOI concepts and applications. If our prospective evaluation is successful, VOI calculations will be transitioned to SWOG’s statistical coordinating center so they become an integral part of trial development.

Decision Makers’ Acceptance of the VOI Process and Retrospective Results

We received generally positive feedback from SWOG members when we presented the final modeling process and results of our retrospective VOI analysis. In particular, members expressed enthusiasm and support for the idea of using VOI analyses to inform trial prioritization decisions (“What they’re doing is incredibly important… You will see trials with negatives. We need it to make strategic decisions.”). SWOG members also stated that requiring members to be explicit about the likelihood of a trial reaching its endpoint (“We don’t do this often enough. We’re not critical enough to ask directly, ‘What is the likelihood of reaching the trial’s endpoint?”’) or the assumptions used in sample size calculations (“This method [VOI] ties us more closely to the assumptions [the statistician] is going to make in the sample size calculations”) would be an informative exercise independent of the VOI analyses. SWOG ERC members also felt that the results of these formal elicitations would be particularly useful for their triage decisions.

DISCUSSION

We worked with key stakeholders to develop an efficient and customized VOI modeling framework that is feasible to conduct in “real time” within SWOG’s current trial proposal evaluation process and is acceptable to stakeholders. Our VOI process focuses on the primary endpoint of the proposed trial and leverages the prior evidence summarized in the proposal. We also modified the calculation and presentation of VOI results from those commonly found in academic journals to be more closely aligned with SWOG stakeholder needs and preferences.

Implications and Lessons Learned

We found that with some modifications, the modified VOI analyses were feasible for most (8 of 9) trial proposals in our sample. Furthermore, the reason why calculating VOI was not feasible for the remaining proposal was nevertheless informative of the trial’s expected value: a trial in which the primary endpoint does not capture key expected differences in patient outcomes is unlikely to generate valuable evidence to inform treatment decisions. It is therefore instructive that this proposal was not approved by SWOG in part because the primary endpoint of the trial was not considered clinically significant. This finding also highlights a larger theme that emerged from presenting our VOI modeling process and results to stakeholders: in many cases, the explicit process of creating decision models and characterizing uncertainty was as informative as the VOI results in understanding a trial’s expected value.

Our modeling approach was also efficient: it took 1 researcher less than 1 week to calculate the VOI for each trial proposal. Such efficiency was critical so as not to impede SWOG’s trial evaluation process. Moreover, SWOG investigators are providing additional data needed for our VOI analyses in future proposals, further reducing the turnaround time to produce VOI results during the prospective phase. It was efficient to validate key inputs and assumptions with stakeholders given their familiarity with the trial proposals and the limited number of modeling inputs used; however, this step occurred several months after the initial models were developed. Establishing a consistent and ongoing interface between the modeling team and key clinical stakeholders will be critical to ensure that these models are built and validated quickly when VOI analyses are implemented prospectively into SWOG’s proposal evaluation processes.

We worked with SWOG stakeholders to customize the VOI analyses to improve their acceptance and usefulness in informing prioritization decisions. For example, a goal of SWOG is to pursue trials less likely to be pursued by industry or single institutions, and many SWOG members view trials in rare cancers as aligning with that mission. SWOG members may therefore place a higher value on some trials in rare cancers, which would not be directly reflected in the population-level VOI estimates. By providing both individual-level and population results, the decision makers’ preferences for trials in rare cancers remain implicit, but the implications to society of such preferences are made explicit. Our approach also leaves the willingness to pay for health benefits implicit, which was important given the range of stakeholder opinions about what the appropriate willingness-to-pay threshold should be. Although this work was driven exclusively by feedback from stakeholders, the approach may offer a framework for using VOI methods to prioritize research investments in the portfolios of other US-based funding agencies.

Finally, our findings are also instructive for future efforts to engage decision makers at other organizations in the implementation of VOI analyses in research prioritization efforts. Based on our experience working with SWOG members, we recommend 1) providing repeated exposure to VOI concepts in different formats to accommodate a range of preferences for learning and engagement (e.g., printed materials, in-person presentations, Web-based training), 2) presenting tangible examples of VOI analyses for research studies with which decision makers are already familiar, 3) using an open forum for stakeholders to ask questions and express their concerns before the analytical approach is finalized or implemented, and 4) providing multiple options and metrics to communicate the results.

Comparison With Previous VOI Applications

The modeling framework we developed can likely be extended to other settings within and outside of cancer clinical trials with some customization; however, its efficiency is tied directly to having a relatively well-developed research study proposal that includes relevant evidence and expert opinion to empirically characterize the relationship between the trial’s primary endpoint and a comprehensive measure of health outcomes. Thus, our final modeling approach is most accurately viewed as a hybrid between full decision analytic models and the conceptualization of “minimal modeling” by Meltzer and others.

Our approach focused exclusively on using VOI analyses to inform trial selection and prioritization at the level of SWOG’s ERC. Prior research has shown how VOI analyses can inform trial design by determining the optimal sample size (i.e., one that maximizes the differences between the cost of doing a trial and the VOI gained from the results), identifying additional study outcomes that would be valuable to collect during the trial, and/or illustrating how the treatment decision might benefit from future evidence available outside of a clinical trial setting.^24,25 The focus on research prioritization was a tractable first step with the dual goal of improving prioritization decisions and increasingSWOGmembers’ familiarity with the methods and concepts of VOI analyses. We did not consider the potential value of nonclinical trial study designs because such studies are largely outside the scope of SWOG’s mission and research portfolio.

In a white paper commissioned by PCORI, Claxton and others examined the potential use of VOI techniques to inform research prioritization decisions in the United States and, similarly to our approach, assumed treatment decisions were made according to health, rather than net, benefits to patients.²⁶ How VOI estimates based on health benefits differ from those based on net benefits is related to the impact that costs have on the current and future decision uncertainty and consequences. In the trial proposal of pelvic lymph node dissection (proposal A), decision uncertainty is driven largely by uncertainty in effectiveness and differences in healthcare costs are not expected to have a major effect on outcomes; therefore, the expected value of sample information would be similar if calculated according to health or net benefits. In all other trial proposals, however, costs are expected to impact decision uncertainty and consequences and the VOI results would not be comparable. Unlike in the white paper, we also provide decision makers with the expected incremental healthcare costs associated with future treatment decisions made based on health benefits. Thus, although some SWOG decision makers may choose to prioritize trial investments based on net (rather than health) benefit, they must do so within a setting where treatment adoption decisions are driven by health benefits.

Our analyses also differed importantly from a Bayesian framework in determining the “current” treatment within our simulations. An assumption in Bayesian decision theoretic VOI analyses is that the treatment considered optimal is implemented into clinical practice, regardless of the magnitude of the benefit or amount of uncertainty. Under this framework, the experimental treatment would be considered optimal based on limited prior evidence or expert opinion for all trial proposals in our sample; however, by design, the control arm represents current standard of care. Because the VOI to alter clinical practice depends on what treatment patients currently receive, we assumed that the current treatment was the control arm and established a minimum probability the experimental treatment was sufficiently superior to change practice within our simulation framework. We believe that this pragmatic approach more accurately reflects real-world treatment decisions in the United States, particularly the risk -aversion to adopting promising but highly uncertain treatments.

Claxton and Sculpher conducted a pilot study using VOI analyses to inform research prioritization decisions within the United Kingdom.²⁷ Their primary challenges were not technical or methodological, but rather the reluctance of decision makers to adopt explicit criteria for research prioritization. We therefore anticipated that SWOG members would be similarly reluctant to consider VOI analyses in setting research priorities; however, we found that they all understood the implications of the organization’s limited research budget and were generally enthusiastic and interested in quantitative tools that could inform their investment decisions. The key point of disagreement was the extent to which SWOG was responsible for considering the downstream costs of cancer treatment when developing or prioritizing clinical trials. Although our final VOI modeling process will not provide an answer to this question, it does provide a rigorous and systematic framework for prioritizing research investments by their expected clinical value; in many cases, it may make the opportunity costs of not considering downstream healthcare costs explicit.

Limitations

There are several key limitations to this work. First, we did not conduct a formal survey of SWOG stakeholders’ opinions of the VOI methods or results. Our objective was to identify key barriers to using VOI analyses to inform SWOG’s investment decisions and refine our modeling approach accordingly. To do so, we engaged directly with SWOG members through various formats and at various times, thereby providing multiple opportunities for different voices to be heard and provide feedback. We received positive feedback from the ERC and the Breast, Gastrointestinal, and Genitourinary Disease Committees and are proceeding with the prospective evaluation; we are therefore confident that the majority of stakeholders accept our final VOI modeling process.

Second, it is difficult to assess whether the VOI results could have improved SWOG’s decision making in our retrospective evaluation. Each trial has not only different VOI estimates but also different expected costs and resource requirements. Thus, decision makers must consider both the absolute and relative return on investment as well as other factors, such as feasibility or strategic fit, when making decisions. Although we cannot meaningfully assess whether VOI estimates could have informed and ultimately influenced trial funding decisions in this retrospective analysis, our ongoing prospective evaluation of this process will provide these important results.

Our modeling VOI process also has several potential drawbacks. First, our decision models may be overly simplistic representations of complex clinical processes. Although the chairs of each disease committee confirmed that our VOI models sufficiently captured the key outcomes of the treatment decision in the sample of historical trial proposals, it will be important to determine whether additional modifications or refinements to our process are necessary when implemented prospectively and across a larger sample of trial proposals.

Second, our parameterization of the prior distribution for the treatment effect mirrored the assumptions included in the sample size calculations of the trial proposals. Importantly these calculations assumed an exponential survival function. Sensitivity analyses indicated that the VOI results were robust to such assumptions; however, it remains to be seen whether such assumptions are reasonable for future trial proposals. Therefore, we plan to continue evaluating the impact of alternative survival distributions on VOI estimates for future trial proposals as part of our prospective evaluation.

Third, our modeling framework relied on 2 key assumptions: 1) that the comparator arm represents current standard of care, and 2) that the time horizon for the information being generated by each trial was 10 years. Feedback from SWOG stakeholders indicated that the comparator arm assumption was appropriate for our retrospective sample, but it may not be generalizable to trials that compare 2 or more interventions already widely used. If this situation is encountered in the prospective evaluation, we will carefully distinguish between VOI potentially generated from the trial and the value of implementing currently available evidence into clinical practice.^28,29 We anticipated that SWOG members would provide input on the expected time horizon for each proposal; however, our assumption of a 10-year time horizon was not questioned when we presented our models for feedback. In later conversations, it emerged that SWOG members felt a time horizon was a nebulous concept, particularly given the rapidly and oftentimes unexpectedly changing clinical landscape in oncology, and they were therefore reluctant or unable to provide specific estimates for this input. Thus, when soliciting input from clinical experts on this key assumption in future work, we plan to provide them with historical benchmarks as context. In the interim, we provide SWOG decision makers with VOI estimates at the per-patient level and population level, the former of which is not affected by the time horizon, and we are upfront and transparent with decision makers about the inputs, assumptions, and limitations of our models.

CONCLUSIONS

We developed an efficient and reproducible process to rapidly generate VOI estimates for cancer clinical trial proposals that is transparent and acceptable to decision makers within SWOG. Our findings indicate that implementing VOI analyses within a clinical trials research organization’s real-time proposal evaluation processes is generally feasible and acceptable to stakeholders with appropriate customization. Future work will assess whether this approach to calculating VOI can meaningfully inform and ultimately influence prospective clinical trial research investment decisions within SWOG and thereby help align their research portfolios to have the greatest public health impact.