Value of Information Analysis within a Stakeholder-Driven Research Prioritization Process in a US Setting: An Application in Cancer Genomics

Abstract

Objective

The objective of this study was to evaluate the feasibility and outcomes of incorporating value of information (VOI) analysis into a stakeholder-driven research prioritization process in a US-based setting.

Methods

Within a program to prioritize comparative effectiveness research areas in cancer genomics, over a period of 7 months, we developed decision-analytic models and calculated upper-bound VOI estimates for three previously selected genomic tests. Thirteen stakeholders representing patient advocates, payers, test developers, regulators, policy-makers, and community-based oncologists ranked the tests before and after receiving VOI results. The stakeholders were surveyed about the usefulness and impact of the VOI findings.

Results

The estimated upper-bound VOI ranged from $33M to $2.8 billion for the three research areas. Seven stakeholders indicated the results modified their rankings, nine stated VOI data was useful, and all indicated they would support its use in future prioritization processes. Some stakeholders indicated expected value of sampled information might be the preferred choice when evaluating specific study designs.

Limitations

Our study was limited by the size and the potential for selection bias in the composition of the external stakeholder group, lack of a randomized design to assess effect of VOI data on rankings, and the use of expected value of perfect information versus expected value of sample information methods.

Conclusions

Value of information analyses may have a meaningful role in research topic prioritization for comparative effectiveness research in the US, particularly when large differences in VOI across topic areas are identified. Additional research is needed to facilitate the use of more complex value of information analyses in this setting.

Introduction

In a healthcare system with limited resources for research, it is vital to identify research areas with the greatest likelihood of influencing clinical practice and improving patient outcomes. A quantitative approach to research prioritization that has received increased attention, particularly within the context of comparative effectiveness research, is value of information (VOI) analysis. This approach involves the application of methods from economic theory and decision analysis to estimate the humanistic and economic value of performing additional research to better understand the safety, efficacy, and cost of technologies and medical interventions.(1, 2)

The VOI approach, though conceptually compelling, is complex and can be nontransparent to decision makers. A multitude of stand-alone VOI analyses have been published evaluating a diverse range of research topics, but VOI has rarely been used to inform research funding decisions. In the UK, two pilot VOI research prioritization projects have been performed with positive results. In the US, a group from Duke university performed a pilot study evaluating the potential use of VOI for research prioritization, but no applications of VOI directly linked to research decision making processes have been published in the US.(3–5)

The objective of this pilot study was to assess the feasibility, strengths, and weaknesses of a pragmatic approach for incorporating formal VOI analysis into a stakeholder-driven research prioritization process. This study was conducted within the context of the Center for Comparative Effectiveness Research in Cancer Genomics (CANCERGEN), a collaboration between four institutions: Fred Hutchinson Cancer Research Center, the SWOG (Southwest Oncology Group), one of the largest cancer clinical trials groups in the US, the University of Washington, and the Center for Medical Technology and Policy.(6) The nature of our endeavor was exploratory, and was performed with the aim of informing future efforts to integrate VOI into research prioritization.

Methods Overview

Setting

The VOI analyses were conducted to provide external input to SWOG leadership regarding priority comparative effectiveness research opportunities in cancer genomics. The approach described herein was thus not an evaluation of specific studies or study designs (e.g., RCTs), but the first step in quantitatively identifying promising research areas. Integral to the process was an external stakeholder advisory group (ESAG) with 13 representatives from a diverse range of constituencies: patient advocates (2), payers (3), test developers (2), regulators (1), policy-makers (2) and practicing oncologists (3), as previously described.(7) These members were chosen based on their knowledge, experience, and willingness to commit for a two-year term on the ESAG. The goals were to 1) identify and transmit priority research areas to SWOG leadership and investigators for their consideration within existing prioritization processes, and 2) identify topic areas for development of specific comparative effectiveness research studies in collaboration with SWOG investigators.

Qualitative Prioritization Process

The qualitative research prioritization process employed within CANCERGEN used a structured landscape analysis and ranking by stakeholders using specific criteria to cull the many genomic tests in development into a manageable set of three high priority cancer genomic tests for further detailed evaluation using VOI.(8) The qualitative criteria used were: population impact; adequacy of current standard of care; test performance (i.e. analytic and clinical validity); benefits; harms; economic impacts; evidence of need for an RCT; clinical trial implementation and feasibility; and market factors. The investigators and external advisory group jointly developed these criteria.

The three high priority genomic test topics selected using the qualitative criteria were: 1) ERCC1 expression testing for platinum-based adjuvant therapy in fully-resected early-stage non-small cell lung cancer (NSCLC), 2) epidermal growth factor receptor (EGFR) mutation testing for erlotinib maintenance therapy after 1st line chemotherapy in advanced NSCLC, and 3) breast cancer tumor markers (carcinoembryonic antigen and cancer antigens 15-3 and 27.29) for detection of recurrence after primary breast cancer therapy.

VOI-informed prioritization process

The quantitative VOI prioritization and evaluation process included multiple phases (see Figure 1): 1) development of decision-analytic models in collaboration with stakeholders 2) calculation of VOI results, 3) education of stakeholders about VOI, 4) presenting and discussing the VOI results followed by a re-ranking of tests, and 5) evaluating the stakeholders’ perceptions of the VOI analyses on the research prioritization process.

Figure 1. VOI Research Prioritization Process.

Figure 1 illustrates the research prioritization process including the development of decision-analytic models, education of stakeholders about VOI, presentation and discussion the VOI results followed by a re-ranking of tests, and the evaluation of the impact and stakeholders’ perceptions of the VOI analyses on the research prioritization process.

1. Development of models

We used a simulation model approach to estimate the value of future research. The decision-analytic models, built in Microsoft excel (Microsoft Corporation. Redmond, WA), were designed to reflect key events in the clinical pathway between the use of tests and health outcomes and were structured similar to previous models we have developed in this area.(9, 10) As part of model development, the ESAG and internal modeling experts were provided an opportunity to review model structures, key assumptions, and key inputs during a web-enabled conference call. We evaluated internal validity by subjecting the model to thorough debugging using null and extreme input values and a detailed review of all mathematical formulas and coding. Model schematics, key model inputs, and assumptions used to generate VOI results are provided in a technical appendix (see Appendix A). Data to populate the models were derived from the literature based on systematic searches, expert clinical input, and clinical trial proposals. We performed two separate VOI analyses for ERCC1 testing because standard care varies between stage I and stage II patients.

2. VOI calculations

We chose a metric called the expected value of perfect information (EVPI), which estimates the upper level dollar value of reducing all uncertainty about which strategy is the optimal intervention. We chose this measure because it has been used in previous research prioritization exercises outside the U.S. and is not heavily dependent on study design considerations, which can vary widely in terms of their structure (e.g. prospective vs. retrospective), comparators, and choice of outcomes. The limitations of using EVPI as a metric are addressed in the Discussion section. We used the models to calculate the net-benefit for a given treatment strategy based on an estimation of the societal willingness-to-pay for health gains (e.g., a quality-adjusted life-year, QALY), which allows for the conversion of the estimated clinical impact of a given strategy into a monetary value. The net-benefit of a given strategy was calculated by subtracting the cost of the strategy from the monetized health gains to provide the net-benefit (QALY x willingness to pay – cost). We examined a range of willingness to pay thresholds, but for presentation of the results to the ESAG, we chose a value of $150,000/QALY as it approximates the revealed willingness-to-pay threshold in the U.S.(11)

We performed probabilistic sensitivity analyses using standard methods to calculate the probability and consequences (i.e. impact on expected net benefit) of making a wrong decision about the treatment decision—which allow the estimation of the upper level per patient VOI.(12) Without available data on the correlation of the uncertainty distributions we assumed that they were uncorrelated. To generate the upper level population VOI, we multiplied the per-patient VOI by the affected population, i.e. the number of patients expected to encounter the treatment decision over the lifetime of the technology. The size of the affected population was calculated using SEER incidence data, a discount rate of 3%, and an assumed 10-year lifetime for the technologies chosen to reflect an approximate middle point to the length of time a new technology takes to become fully diffused.(13)

3. Stakeholder education

We prepared and provided the ESAG brief educational materials on VOI theory and methodologies as well as technical briefs on the decision models that enable the VOI calculations (see Appendices A and B). The education process included presentation of the VOI background and model-specific briefs via a moderated web-enabled teleconference with the ESAG. The ESAG was provided ample opportunity to discuss and ask clarifying questions about VOI and the specific models. In addition, electronic versions of the briefing documents were provided for review prior to the teleconference.

4. Presentation of VOI results and re-ranking of tests

We presented the results of the VOI analyses to the ESAG during a moderated web-enabled teleconference. Opportunities were provided for questions and discussion of the findings amongst the ESAG members and the CANCERGEN investigators. ESAG members were then asked to re-rank the top three tests. The scoring process involved sequential ranking of tests with the top choice removed after each round.

5. Stakeholder evaluation of VOI

ESAG members received an online survey to evaluate the VOI process consisting of 15 multiple choice, yes/no, and Likert-scale questions (see Table 1). Topics included the usefulness of VOI, whether the VOI results impacted stakeholders’ ranking, and whether they would like to use VOI in future research prioritization exercises. The survey was sent to ESAG members immediately following the presentation and discussion of the VOI results in the initial teleconference and a subsequent follow-up call with two stakeholders unable to attend the first call.

Table 1.

Survey Results

QUESTIONS	RESPONSES
How useful was the VOI approach in helping you prioritize which genetic tests would most strongly benefit from a prospective CER study?	Not useful		Neutral		Somewhat useful		Mostly useful		Very useful
	0	(0%)	0	(0%)	4	(31%)	5	(38%)	4	(31%)
Please rate your agreement with the following statements:	Strongly Disagree		Disagree		Neutral		Agree		Strongly Agree
- The VOI added new information to the priority setting process.	0	(0%)	0	(0%)	1	8%	9	(69%)	3	(23%)
- The VOI information was understandable.	0	(0%)	0	(0%)	2	15%	9	(69%)	2	(15%)
- The VOI information was presented in clear, non-jargon language.	0	(0%)	0	(0%)	1	8%	11	(85%)	1	(8%)
- The VOI information was applicable to the priority setting process.	0	(0%)	0	(0%)	1	8%	7	(54%)	5	(38%)
- Adequate information was provided regarding model assumptions.	0	(0%)	1	(8%)	3	23%	9	(69%)	0	(0%)
- I am confident in the model results for ERCC1 testing.	0	(0%)	2	(15%)	2	15%	8	(62%)	1	(8%)
- I am confident in the model results for EGFR mutation testing.	0	(0%)	1	(8%)	2	15%	9	(69%)	1	(8%)
- I am confident in the model results for breast cancer tumor markers.	0	(0%)	2	(15%)	1	8%	9	(69%)	1	(8%)
Did the VOI results change your overall ranking?	YES		NO
	7	(54%)	6	(46%)
- If YES, what aspect of VOI was most influential to your change in ranking?	The results (i.e. the value of perfect information)		The way in which the question was framed		The data that was used to address the question		Other
	4	(31%)	0	(0%)	3	(23%)	0	(0%)
Given what you have learned about VOI, would you use the results of a VOI analysis in a priority setting process again?	YES		NO
	13	(100%)	0	(0%)
Do you have any questions about VOI or the individual models that remain unanswered?	YES		NO
	3	(23%)	10	(77%)
Would additional information have helped you better understand and use the VOI results?	YES		NO
	5	(38%)	8	(62%)

Abbreviations: VOI, value of information; CER, comparative effectiveness research; EGFR, epidermal growth factor receptor; ERCC1, excision repair cross-complementation group 1

Results

After identifying the three top candidate genomic tests it took seven researchers (averaging 40% full-time equivalent) approximately 7 months to develop the VOI models, educate the ESAG about VOI, present and discuss the VOI results, and have the ESAG revote as to their top three tests. The estimated cost for this effort was approximately $250,000. The bulk of this time was spent developing the decision analytic models and conducting the VOI analyses.

VOR Results

The upper-bound VOI for ERCC1 was $2.2 and $2.8 billion in Stage I and Stage II disease, respectively; $2.1 billion for BC (breast cancer) markers; and $33 million for EGFR (Table 2). The affected populations over a ten-year time horizon were estimated to be 234,051 for ERCC1 for both stage I and stage II, 416,746 for BC markers, and 170,253 for EGFR. These large numbers reflect the high incidence of breast and lung cancers. The probabilities of making a wrong decision about testing using existing information for ERCC1 were 26% (Stage I lung cancer) and 42% (Stage II lung cancer), 43% for breast cancer tumor marker tests, but only 11% for EGFR testing in lung cancer. The high values for ERCC1 Stage II and BC markers indicate a high level of uncertainty surrounding the key model inputs (e.g. the treatment effect hazard ratio). The consequences of making a wrong decision were largest for ERCC1 (Stage I: $47,300; Stage II: $22,500) followed by BC markers ($11,700) and EGFR ($1,600). These values reflect the average impact on patients in terms of morbidity, mortality, and cost when a non-optimal decision is made because of the existing uncertainty.

Table 2.

Value of Information Results

Cancer genomic application	Affected Population*	Probability of making the wrong decision about testing	Consequences of making the wrong decision about testing	Value of Information** (millions)
EGFR mutation testing in maintenance treatment in advance NSCLC	170,253	12%	$1,600	$33
ERCC1 Testing in early stage NSCLC: Stage I	234,051	26%	$47,300	$2,800
ERCC1 Testing in early stage NSCLC: stage II	234,051	42%	$22,500	$2,200
Breast Cancer Tumor Marker Testing	416,746	43%	$11,700	$2,100

Over 10 years, discounted at 3%;

^**Calculated at $150,000 per QALY willingness to pay

Abbreviations: EGFR, epidermal growth factor receptor; ERCC1, excision repair cross-complementation group 1; BC, breast cancer, NSCLC, non-small cell lung cancer

Perceptions of Decision-Makers

Discussions during presentation of the VOI results were primarily focused on clarifying technical aspects about the simulation models, the calculation of the affected populations, the choice and application of the willingness to pay threshold chosen, and the presentation of the VOI results including uncertainty in the VOI estimates.

The ranking of the tests changed from 1) ERCC1, 2) EGFR, and 3) BC markers to 1) ERCC1, 2) BC markers, and 3) EGFR. Thus, BC markers were ranked higher relative to EGFR compared to the initial prioritization vote. Stakeholders stated this was based on the several orders of magnitude difference in VOI between EGFR and BC tumor markers topic areas. Specifically, some stakeholders acknowledged that there appeared to be substantially more value in studying what they had considered an older, inappropriately used technology than they had realized before being presented the VOI analyses. There was recognition that if the tumor markers do not provide clinical value, healthcare resources are currently being wasted for those who receive testing, and if the markers do provide clinical value, there is a significant population of women that could receive a meaningful benefit. Stakeholders also commented that the level of uncertainty for EGFR testing was much lower (11% probability of error) compared to BC tumor markers (43% probability of error).

The results of the survey evaluating the VOI process are provided in (Table 1). The majority (9/13, 69%) of ESAG members found the VOI analysis to be useful, and approximately half (7/13, 53%) stated they changed their priority ratings specifically based on the VOI results. All stakeholders reported that they would use results of a VOI analysis in a priority-setting process again. In a subsequent follow-up call with stakeholders in which EVSI concepts were discussed, several stakeholders noted that this metric would be more appealing to them when discussing specific study designs, rather than general research areas.

Discussion

We evaluated the feasibility and perceptions of value of information analysis in a process where a diverse group of external stakeholders provided recommendations to SWOG regarding priority research areas in cancer genomics. Over the course of approximately seven months, we synthesized relevant data, developed decision-analytic models, generated VOI results, presented them to the stakeholders, and collected their rankings and opinions.

Implications and Lessons Learned

The results of this pilot study suggest that it is feasible to implement VOI into the research prioritization process for a large clinical trials organization; however, VOI is labor-intensive and requires staff with specialized training. In addition, we chose a relatively simple application of VOI—focusing only on EVPI—in part to make the exercise tractable and acceptable to decision makers, but also due to resource and time constraints. The decision makers involved in our study appeared to be amenable to this technique, in part, we believe, because it was presented simply and across a broad range of topics. In addition, the EVPI presents a single dollar value “bottom line” that provides high-level benchmarking of the maximum potential benefit of further research on divergent technologies and diseases.

Ultimately, the VOI results appeared to impact some decision makers’ priority ranking of the candidate tests. We hypothesize that the large VOI difference (over an order of magnitude) between EGFR and BrCA tumor markers was the primary cause of the change in prioritization. Although we did not employ a randomized study design to more definitely assess the effect of VOI, our findings suggest that VOI did play some role in influencing decision makers’ rankings.

Use of Expected Value of Sample Information techniques to inform research prioritization, estimate trial size requirements, and incorporate specific research costs may be feasible with certain stakeholder groups or types provided that substantial time and resources are applied. Approaches to simplify the analysis of the VOI, such as those proposed recently by Meltzer et al., will be essential before application of VOI concepts and methods can become widespread.(14)

Based on the investigators’ experiences and follow-up interviews with decision makers we believe there are two key lessons learned from our VOI research prioritization exercise. First, it is important to involve the stakeholders throughout the entire process from research topic selection to model development and analysis such that they are engaged and adequately prepared to understand and incorporate the VOI results into the larger research prioritization activity. We also believe that it is essential to provide a thorough educational component prior to showing the VOI results. Because VOI is a multidimensional concept, training allowed decision makers a chance to compare, quantitatively, the factors that most influenced the overall result.

The use of VOI analyses to inform research prioritization may be a good investment for research organizations and funding agencies given the relatively small cost of such analyses compared to the cost of research studies. However, implementing the processes to effectively use the information is challenging, and most importantly, time consuming. Future efforts likely will need to focus on improving the time efficiency of the process to make use of VOI in real-world settings feasible.

Comparison to previous VOI applications

VOI, and EVPI in specific, has been put into practice previously in the U.K. through two pilot projects, one for the UK National Coordinating Centre for Health Technology Assessment (NCCHTA) and another for the National Institute for Health and Clinical Excellence (NICE).(2, 3) The processes and experiences in the UK examples are similar to ours with the exception that we used a group of stakeholders with the intent of providing recommendations on research priorities in cancer genomics whereas the UK examples interacted directly with the research funding organizations that functioned across disease areas. The UK funders found the VOI analyses to be interesting, potentially useful, and thought that value of information analysis should be considered and developed for future prioritization decisions.10

Recently, an AHRQ-funded methods project was undertaken to evaluate model-based EVPI calculations for use in research prioritization in the context of a stakeholder process. In contrast to our study, the investigators generated per person EVPI estimates as opposed to population EVPI, selected cases using convenience sampling as opposed to a landscape analysis, and compared stakeholder rankings of research questions (e.g. impact of adherence) within comparative effectiveness research areas as opposed to our ranking across comparative effectiveness research areas. In accordance with our findings, the investigators indicated that the stakeholders were receptive to VOI and that the feasibility of conducting VOI within research prioritization processes is contingent upon sufficient time, resources, and expertise.(15)

Implications of VOI Results for Cancer Genomic Testing

The VOI analyses indicated that there would be substantial value from additional research on ERCC1 testing in early stage NSCLC and BC tumor marker testing in early stage breast cancer, but considerable less for EGFR testing to select maintenance treatment in advanced NSCLC. These differences highlighted the fact that the existing data on EGFR testing provided greater certainty in the association between variants and clinical outcomes (e.g., a narrower 95% confidence interval for the treatment effect hazard ratio) relative to the other two interventions. In addition, EGFR testing had the smallest affected population due to its more limited indication.(16, 17) Explaining how these factors influenced EVPI made the VOI process more transparent to decision makers, and we believe heighted their acceptance of the method.

Limitations

This study had a number of limitations that warrant mention. There is the potential for selection bias in the composition of the ESAG if stakeholders who agree to participate in a research prioritization project in cancer genomics are not representative of stakeholders in general. Therefore, in assembling the ESAG, the goal was to ensure active engagement and balanced representation among the stakeholder groups. We also used a previous ranking against which to compare the post-VOR test rankings, rather than a randomized, controlled design, which limits the strength of our conclusions regarding the impact of VOI.

Our stakeholder group was intended to represent the diverse stakeholders involved in cancer genomics, and created in light of a guiding principal of CER that such groups should be engaged in prioritizing, designing, implementing, and disseminating CER. However, this group is clearly different from a funding agency such as NCI or NIH, and further studies, particularly in the US, are needed to assess the feasibility of VOI in such settings.

Although we provided the stakeholders with educational materials regarding VOI methods and technical aspects underlying the decision-models and VOI calculations, some still expressed difficulty understanding the concepts. We attempted to strike a balance between adequately preparing the ESAG to understand the VOI models and providing a manageable amount of information. Ideally, engaging stakeholders in discussions over VOI should occur in a face-to-face setting, with more time for questions and answers about the underlying assumptions of the model. In future work, we believe that VOI results or concepts should be presented even earlier in the prioritization process.

Finally, several value of information metrics have been introduced and discussed in the context of prioritizing comparative effectiveness research including the expected value of perfect information, expected value of perfect parameter information, and the expected value of sample information.(18–22) We used EVPI, which estimates the upper level dollar value of reducing all uncertainty about which strategy is the optimal intervention. It is a pragmatic yet limited approach to evaluated VOI. It can provide a useful metric to gauge the general state of knowledge as well as the potential benefit of research to patients. If the EVPI is low, for example, decision makers can rule out less compelling research opportunities and focus resources on areas with greater potential value. However, EVPI has important limitations. A very low EVPI may indicate that it is not worth doing a study, but a very high (or even moderately high) EVPI does not indicate the opposite. EVSI methods, in contrast, estimate the total value of imperfect or limited information from a specific research study, where the imperfection arises due to resource constraints and being unable to resolve all forms of uncertainty about comparative effectiveness in a pragmatic setting. Ultimately, since prioritization of research depends on comparing the expected value of information from a pragmatic research study to the cost of performing such a study, EVSI is a more appropriate metric to use for the final selection of research projects and also to inform optimal research designs, although significantly greater resources and time are required to conduct EVSI calculations. However, there exist two challenges in using such metrics for research prioritization in practical settings: 1) the computational burden of EVSI is more complex and time consuming than EVPI calculations; and 2) for decision makers, EVSI may be more challenging conceptually than EVPI. Assessing the success in communicating EVPI information to stakeholders was considered a useful first step before providing EVSI information. Moreover, ranking and ruling out research areas (rather than specific studies) based on the overall potential value within the research area is the natural step preceding the use of EVSI to rank specific study designs. The successful and appropriate implementation of VOI in research prioritization processes will require that more transparent and time efficient approaches to EVSI be developed.

Conclusion

Prioritizing research topics is becoming increasingly important as research budgets decrease and the cost of conducting research continues to rise. This is especially true in comparative effectiveness research of cancer genomics, which can require large patient numbers and commensurate expenditures. New methods for prioritizing research are needed to ensure that our finite research dollars are spent wisely. Applied in concert with professional judgments, formal VOI analysis accompanied by sufficient support may help decision makers maximize the impact of research portfolios on medical care and human health. Further research on VOI approaches and metrics, and explicit integration with existing prioritization processes will be essential to achieve the goals of comparative effectiveness research – to provide optimal information to stakeholders within existing resource constraints.