Getting our priorities straight: a novel framework for stakeholder-informed prioritization of cancer genomics research

Abstract

Purpose

Prioritization of translational research on genomic tests is critically important given the rapid pace of innovation in genomics. The goal of this study was to evaluate a stakeholder-informed priority-setting framework in cancer genomics.

Methods

An external stakeholder advisory group including patients/consumers, payers, clinicians, and test developers used a modified Delphi approach to prioritize six candidate cancer genomic technologies during a 1-day meeting. Nine qualitative priority-setting criteria were considered. We used a directed, qualitative content-analysis approach to investigate the themes of the meeting discussion.

Results

Stakeholders primarily discussed six of the original nine criteria: clinical benefits, population health impacts, economic impacts, analytical and clinical validity, clinical trial implementation and feasibility, and market factors. Several new priority-setting criteria were identified from the workshop transcript, including “patient-reported outcomes,” “clinical trial ethics,” and “trial recruitment.” The new criteria were incorporated with prespecified criteria to develop a novel priority-setting framework.

Conclusion

This study highlights key criteria that stakeholders can consider when prioritizing comparative effectiveness research for cancer genomic applications. Applying an explicit priority-setting framework to inform investment in comparative effectiveness research can help to ensure that critical factors are weighed when deciding between many potential research questions and trial designs.

INTRODUCTION

The rapid development of genomic technology in medicine is enhancing understanding of how genomic variation introduces heterogeneity in response to clinical and public health interventions. This knowledge presents the opportunity to incorporate genomic factors into clinical decision making; however, genomic technology translation has been limited by numerous factors, among which include a lack of prospective trial evidence comparing the health outcomes of genomic testing strategies with standard clinical care.^1–3

Comparative effectiveness research (CER), generally defined as “systematic research comparing different interventions and strategies to...inform patients, providers, and decision makers...about which interventions are most effective for which patients under specific circumstances,” has the potential to address these evidence gaps.³ Prospective CER trials are offen conducted to compare the average effects of interventions or strategies in “real-world” settings.^4,5 Prospective CER trial designs can be structured to generate comparative evidence about the outcomes of genomic versus standard clinical care strategies.³ Few CER studies of this type have been conducted to date. However, given stakeholder requirements for more rigorous evidence of the relative benefits and harms of genomic technologies, prospective trials will likely play an important role in future evaluation of clinical utility (defined as “improvement in life expectancy or quality of life for patients and their families”⁶) and decisions about appropriate translation of candidate technologies.³

Ideally, researchers would conduct prospective CER trials to evaluate all promising genomic technologies, and those with demonstrated clinical utility could move forward toward clinical application. However, this scenario is not realistic because of the vast number of existing and developing technologies, the prohibitive costs of such studies, and the limited number of patients with targeted diseases who are willing and able to participate in trials.^7–9 For example, a recent horizon scan identified 250 new genomic tests during the period from May 2009 to May 2010.⁸ An example of this reality is the SWOG RxPONDER (Rx for Positive Node Endocrine Responsive Breast Cancer) trial to investigate chemotherapy benefit in patients with node-positive breast cancer and low to intermediate Oncotype Dx recurrence scores.¹⁰ This prospective CER trial plans to screen 9,000 patients with breast cancer to recruit 4,000 participants.¹⁰ The trial is estimated to cost ~$27 million due to the substantial screening effort required to reach target accrual, high cost of the Oncotype Dx test, and considerable infrastructure requirements. Clearly, trials of this size and cost cannot be conducted to evaluate all new technologies. Consequently, stakeholders must make decisions about how to best invest genomic research resources to improve public health. This necessitates robust priority-setting frameworks that engage the producers and end users of genomic technologies and facilitate decisions about which technologies should move forward on the translational pathway.^11,12

Clinical trial prioritization and decision making has traditionally been carried out by a limited subset of stakeholders, namely researchers and physicians. However, the recent momentum of CER has increased efforts to involve a more comprehensive array of stakeholder constituencies in research prioritization and decision making.^13–15 These constituencies include clinicians, researchers, patient/consumer advocates, policy makers, funders, payers, and industry.¹⁶ An important challenge to stakeholder engagement is the involvement of individuals with diverse expertise, including those with potentially little research experience.^14,15 Successful engagement requires thoughtful selection of stakeholders in relation to the project's needs, education, and expert facilitation. This type of approach has the potential to yield more robust research evidence, but little formal understanding exists regarding what factors these diverse stakeholder groups should consider, how their distinct perspectives meld into shared understanding and/or consensus, and how these factors relate to final priority setting.⁹

In this study, we performed a qualitative analysis of discussions from a stakeholder workshop in which six candidate genomic technologies were priority ranked for evaluation in a prospective CER trial. The study objective was to identify criteria that a diverse group of stakeholders identified as important for the prioritization of potential cancer genomics projects and to report major themes of the stakeholder discussion. Although an important end point of a prioritization process is choosing which studies actually receive funding, these criteria need to be discussed, evaluated, and used before funding decisions can ultimately be made. The findings provide a basic framework for systematically prioritizing genomic technologies for additional evaluation and illustrate the potential impact of engaging a diverse group of stakeholders in research prioritization, which is increasingly important given the stakeholders’ central role in CER.

MATERIALS AND METHODS

Setting

This study was conducted in the setting of the Center for Comparative Effectiveness Research in Cancer Genomics (CANCERGEN), a multidisciplinary consortium including the Fred Hutchinson Cancer Research Center, University of Washington, the Center for Medical Technology Policy, and Southwest Oncology Group, one of the largest of the National Cancer Institute–supported cancer clinical trial cooperative groups in the United States. Under the leadership of the Fred Hutchinson Cancer Research Center, CANCERGEN brings expertise in CER, clinical trial design, and cancer genomics to facilitate the rapid design and implementation of prospective comparative effectiveness studies of genomic technologies. The overall mission of CANCERGEN is to generate high-quality evidence regarding the clinical utility and economic value of genomic technology applications as compared with standard care. As a part of this project, we held a 1-day meeting for an external stake-holder advisory group (ESAG) to produce a priority ranking of six candidate genomic technologies in cancer for evaluation in a prospective CER trial to be carried out by SWOG.

Population

This study involved CANCERGEN's ESAG, which consisted of 13 individuals selected to represent the perspectives of a diverse range of constituencies related to cancer genomic technologies: professional patient/consumer advocates (two; one of whom is a cancer survivor), payers (three), clinicians (three), policy-makers/regulators (three), and the life sciences and diagnostic industry (two).¹⁷ One of the primary aims of CANCERGEN was to implement a stakeholder engagement process to inform the prioritization and design of CER studies in the area of cancer genomics for the evaluation in prospective CER trials to be carried out by SWOG. The project was approved by the institutional review board of the Fred Hutchinson Cancer Research Center.

Prioritization process

The prioritization activities involved ranking six candidate genomic technologies that were identified through a landscape analysis during an earlier phase of the CANCERGEN project.¹⁷ Landscape analysis is a process that involves evaluation of the literature coupled with domain-specific expert consultation.¹⁷ The study investigators selected the six candidate technologies because they believed them to have the greatest potential for clinical impact if prospective CER evidence was generated. These technologies are shown in Table 1.

Table 1.

Overview of candidate genomic technologies under consideration

Candidate technology	Purpose
EGFR mutation testing for erlotinib maintenance therapy after first-line chemotherapy in advanced NSCLC	Disease prognosis and identification of the patients with mutations that are most likely to benefit with erlotinib maintenance therapy
ERCC1 expression testing for platinum-based adjuvant therapy in resected early-stage NSCLC	Disease prognosis and identification of ERCC1-negative patients who are most likely to benefit from platinum-based adjuvant chemotherapy
BRAF mutation testing in colorectal cancer to inform use of cetuximab and panitumumab	Disease prognosis and identification of patients with mutations that are good candidates for therapy with cetuximab and panitumumab
EGFR gene copy number (FISH) testing to inform cetuximab therapy in advanced NSCLC	Disease prognosis and identification of patients who are FISH positive and most likely to respond to chemotherapy + cetuximab
Gene expression profiling in multiple myeloma to identify high-risk patients for more aggressive therapy	Disease prognosis and identification of patients with high-risk profiles for treatment with bortezomib
CEA, CA 15-3, and CA27.29 marker testing to detect recurrence after primary breast cancer therapy	Prognostic for recurrence of breast cancer and can lead to earlier treatment of recurrence

EGFR, epidermal growth factor receptor; ERCC1, excision repair cross complementation 1; FISH, fluorescence in situ hybridization; NSCLC, non-small-cell lung cancer.

To facilitate the ranking of the genomic tests by the ESAG, we developed an initial list of priority-setting criteria on the basis of previous CER prioritization efforts by the Agency for Healthcare Research and Quality.¹⁸ These criteria were also informed by feedback from the ESAG. The final priority-setting criteria are shown in Table 2.¹⁷ The rationale and process for selecting these criteria have been described previously.^17,18

Table 2.

Description of prespecified priority-setting criteria

Criterion	Description
Population health impact	The potential for use of the test to reduce population-level disease burden (disability, morbidity, or mortality) on patients and their families, caretakers, and communities, and contribute to reducing health disparities
Adequacy of current standard of care	Adequacy of current approaches for obtaining prognostic and predictive information about patients
Analytical and clinical validity	Test results are reproducible and have the potential to be accurate in predicting whether or not an individual will develop cancer and/or is likely to experience a recurrence of cancer
Benefits	Clinical decision making based on the test has the potential to provide clinically significant improvement in health outcomes and/or patient care compared with standard of care
Harms	Clinical decision making based on the test result will lead to clinical harms in patients who are false “positive” or lack of benefit in patients who are false “negative”
Economic impacts	The potential economic impact of test use for patients, health plans, and/or public health programs, including the costs of testing, subsequent treatment decisions, and patient time
Evidence of need for an RCT	The likelihood that an RCT of the test will fill substantial gaps in the current body of evidence, and that there is no other timely research planned or in progress that will answer the key research questions. The study results will lead to reduced clinical uncertainties and to changes in the use and/or coverage of a technology or set of technologies (i.e., value of research)
Clinical trial implementation and feasibility	The operational feasibility of a clinical study and consideration of manageable barriers to implementation. Specific considerations include study designs, affordability of proposed research and burden on stakeholders, timeliness of topic, probability of securing funding, and ability to recruit sufficient numbers of patients
Market factors	Market factors suggest that relevant evidence concerning the test would not become available until after the technology is already in use. There is a high degree of pressure on payers to cover this technology, even with limited evidence

RCT, randomized controlled trial.

A modified Delphi process was used to assist the ESAG in creating a priority ranking of the genomic tests, which involved a mix of online voting and a full-day in-person meeting.¹⁹

The method enables group problem solving through an iterative process of discussion, feedback, and revisions. Four weeks before the in-person stakeholder meeting, the ESAG was provided information on each of the six candidate technologies in the form of background briefs and test target profiles (Supplementary Appendix S1 online). These materials were intended to provide stakeholders with a basic understanding of the candidate technologies and the key factors associated with each of the nine priority-setting criteria. Study investigators ensured that materials and engagement activities were accessible to all participants. ESAG members were asked to review the materials and initially rate each genomic test according to the nine priority-setting criteria using a Web-based survey tool.

The in-person, full-day stakeholder meeting was held in Seattle, WA, on 14 June 2010. The agenda included an explanation of meeting objectives from study investigators, overview of the priority-setting process, presentation by an ethics expert of the benefits and drawbacks of using deliberative methods to prioritize research, patient perspectives presentations by patient and consumer advocate ESAG representatives, and detailed presentation and discussion of the six candidate genomic technologies facilitated by study investigators. The test target profiles were used during the meeting to inform the discussion. Workshop discussions were recorded and transcribed.

At the conclusion of the in-person meeting, ESAG members were asked to reprioritize the genomic tests and to list the priority-setting criteria that carried the most weight in their decisions. Affer 2 weeks, the ESAG was requested to reprioritize the tests affer reading the meeting summary and using an online survey to allow the opportunity to reflect on the meeting and their rankings.

Qualitative analysis

Using the transcripts from the stakeholder meeting, a directed, qualitative content-analysis approach was used to analyze the discussion.²⁰ Coding was initially informed by the original nine priority-setting criteria and subsequently evolved as new themes emerged from the data. To increase the validity and accuracy of the results, a qualitative research technique called analytical triangulation was used. This technique is an iterative process in which two researchers independently code the text, then compare and adjust their findings to converge to the same analysis.^21,22 NVivo (QSR International, Cambridge, MA), a qualitative soff-ware program, was used to conduct the analysis.²³

RESULTS

All the 13 ESAG members completed the premeeting priority ranking of the candidate technologies based on the background briefs and test target profiles. Table 3 shows the priority ranking. These results were presented to the ESAG at the in-person meeting.

Table 3.

Premeeting and postmeeting priority-ranking results

Voting results
Premeeting	Postmeeting
ERCC1 expression test for NSCLC	ERCC1 expression test for NSCLC
EGFR mutation test for NSCLC	EGFR mutation test for NSCLC
BRAF for colorectal cancer	Breast cancer tumor markers

EGFR, epidermal growth factor receptor; ERCC1, excision repair cross complementation 1; NSCLC, non-small-cell lung cancer.

Eight stakeholders representing four different stakeholder groups attended the 1-day meeting: patient/consumer advocates (two), clinician/researchers (three), payers (one private, one government health technology assessment), and industry (one). Five ESAG members were unable to attend because of scheduling conflicts. All the 13 ESAG members completed the postmeeting online survey.

Discussion themes by prioritization criteria

Population impact

The population impact of the candidate technologies was frequently discussed (Table 4). Because the incidence of the various cancers and prevalence of the genetic variants differed greatly across candidate technologies, much of the discussion focused on how these criteria should be defined. A physician stakeholder noted the underlying issue in many of the candidate technology disease areas:

“It's tough to treat patients knowing you're probably not going to benefit the majority of them...so finding markers like these would be extremely helpful.”

Table 4.

Major discussion themes from the workshop session by priority-setting criteria

Priority-setting criterion	Major themes of discussion
Population health impact	The incidence of the disease under consideration
	The prevention on the genomic variant(s) targeted by the genomic technology
Adequacy of current standard of care	The proportion of patients who receive benefit under the current standard of care (e.g., tumor response, prolonged survival)
	Whether there are acceptable viable treatment options for all patients who undergo genomic testing
Analytical and clinical validity	Are additional validation studies required before a given genomic technology is ready to be evaluated in a CER trial?
	Are testing methods reliable and reproducible across different laboratories?
Benefits	Magnitude of health gains associated with targeting treatments to subgroups using genomic technologies
	How to compare the relative benefits of technologies that allow targeting agents to maximize benefit vs. technologies that are primarily intended to avoid harms (e.g., toxicity)
Harms	Psychological harms associated with anxiety about receiving (or not receiving) genomic test results
	How would clinical harms associated with false-positive and false-negative results be weighed?
Economic impacts	The role of payers in creating economic incentives for certain CER trials to be conducted
	Are tests that target more expensive interventions of higher priority than tests that target less expensive interventions?
Evidence of need for an RCT	How widespread is the use of the given genomic technology in current clinical practice and is an RCT likely to impact practice?
	Will RCT results provide timely and relevant information for stakeholders?
Clinical trial implementation and feasibility	What characteristics of tests and patient populations make them attractive for evaluating in CER trials?
	How can different study designs impact the feasibility of CER trials?
Market factors	How does public pressure to provide and pay for certain genomic technologies factor into prioritization decisions?
	What role does FDA clearance have in prioritizing genomic technologies (vs. laboratory developed tests)?

The left column shows the nine priority-setting criteria that were used to guide the workshop discussion, and the right column reports the major themes of the workshop discussion related to the given priority-setting criterion.

CER, comparative effectiveness research; FDA; Food and Drug Administration; RCT, randomized controlled trial.

There was a large degree of consensus that disease incidence, variant prevalence, and potential for prolonging survival are important in prioritizing candidate technologies. Population impact was offen discussed in terms of number needed to treat.

Similarly, a payer stakeholder argued that that genomic testing was more appropriate to inform use of targeted therapies like erlotinib, rather than cytotoxic agents like cisplatin in the ERCC1 testing example because “cytotoxic agents ...are more of a shotgun, rather than a laser beam.” Several stakeholders noted that rare diseases should not be ruled out as potential candidates for CER studies.

Analytical/clinical validity

Stakeholders acknowledged many problems with the analytic and clinical validity evidence for the candidate technologies, which impeded translational efforts in this area. A patient representative noted the lack of robust genomic technology validation, suggesting it must be addressed before these technologies are implemented in clinical practice. Much discussion focused on laboratory-developed tests that are performed at a single institution and are not subject to review by the US Food and Drug Administration. A physician noted:

“I do have concerns...(about) a local, homegrown test and quality control, it is always very difficult, and exploiting a single institution's technique is difficult.”

Specific concerns about analytic validity were discussed in the case of immunohistochemistry testing for ERCC1 expression in non-small-cell lung cancer (NSCLC) and fluorescence in situ hybridization testing for epidermal growth factor receptor in NSCLC. Stakeholders suggested that additional assay/test validation studies are needed before these technologies can move forward in a CER trial. In terms of clinical validity, several stakeholders considered EGFR mutation testing in NSCLC as a good candidate for a CER trial due to its strong association with survival in patients treated with erlotinib.

Potential clinical benefits

From a variety of stakeholder perspectives, the overall benefits of genomic tests were twofold: (i) to facilitate and select the most effective therapies for given patients and (ii) to reduce the time, cost, and quality of life lost to pursuing ineffective treatments. One patient stakeholder stated:

“...As the field of testing moves forward, it is becoming possible to do a better job of matching treatment to the profile of a particular patient's disease...It increases the chance of a strong positive response to treatment but...it will also reduce the opportunity cost of time lost with an ineffective treatment.”

Other discussions focused on reducing harm, particularly in relation to EGFR mutation testing. Clinicians and payers thought this test would enrich a population of responders to a targeted therapy, making it an attractive test case for CER. A physician representative stated:

“You've got a situation here where even though the test is expensive, you can show that it spares 90% of the patients from the expense and the side effects of the treatment.”

Economic impacts

Economic impacts were also mentioned frequently. Stakeholders focused on the aggregate costs faced by payers as opposed to the out-of-pocket expenses of patients. They noted the importance of economic considerations given the escalating costs of new diagnostic tests and cancer therapies and subsequent impacts on insurance premiums and ability to pay for alternative treatments. A payer representative noted:

“If I could find the people [who benefit most]...there would be a lot of savings, which could then be recycled to find the right drugs for other people.”

On the basis of this rationale, stakeholders showed considerable interest in EGFR mutation testing in NSCLC due to the potential for cost savings associated with erlotinib. In addition, when discussing the economic impact of the breast cancer tumor markers, stakeholders were concerned with the possibility of prescribing of tests that might not provide any clinical benefit. In this context, many stakeholders saw a CER trial as an opportunity to potentially highlight an ineffective technology. In addition, discussions about gene expression profiling in multiple myeloma highlighted the specific challenge in rare disease that therapy is offen very expensive, underscoring the need for predictive markers to target therapy.

Clinical trial implementation and feasibility

A major discussion theme under this criterion was whether genomic test results could be withheld from trial participants as part of the randomized study design. Stakeholders felt this could complicate the process of receiving institutional review board approval, lead to suboptimal treatment based on current standards of care, and limit patient willingness to participate in a trial.

Considerable discussion took place regarding the feasibility of various trials based on the design, required number of patient screenings, and required duration of follow-up. For example, many stakeholders thought a trial on ERCC1 expression testing was desirable due to the high prevalence of ERCC1-positive patients and rapid progression of NSCLC, which would facilitate achievement of trial end points in a relatively short period of time. Alternatively, several stakeholders noted that a breast cancer tumor marker study would need to be very large, long, and resource intensive given the slower progression of early-stage breast cancer. With respect to EGFR mutation testing, stake-holders viewed the targeted nature of erlotinib presenting additional complexities to study design. The low prevalence of the EGFR mutation (~10% of the NSCLC population) could present significant enrollment issues.

“I don't like the design of including all patients because fortunately or unfortunately, we have a population that we know tends to be selected for the mutation.”

One clinician stated that physicians would not be willing to enroll patients into a study in which they might not be randomized into an erlotinib treatment group even though they might be part of the 10% that would most likely benefit from the therapy.

Market factors

Stakeholders discussed the high level of evidence that would be required to delist the breast cancer tumor markers. The political challenges of delisting these markers were emphasized as well, citing the political backlash with the US Preventive Services Task Force mammogram recommendations.

“...Realistically, payers can't make change until they have some legitimate study backing, especially when you are talking about things that have already disseminated and have a lot of emotional influence such as this.”

From one payer's perspective, the low clinical validity of the test paired with high dissemination was one impetus for prioritizing the test for a CER study. On the basis of the stakeholder discussions, we subsequently revised the definition of “market factors” to “there is a high degree of pressure on payers to cover this technology (or continue to cover this technology), even with limited evidence.”

Potential clinical harms

Potential clinical harms were infrequently mentioned. Several brief discussions occurred regarding the potential for negative psychological impacts related to anxiety about test results and the harms that could result from treatments given on the basis of false-positive or false-negative genomic test results. Much of this discussion was focused on breast cancer tumor markers. Given the low levels of evidence, poor performance of some markers (e.g., CEA 15-3 marker sensitivity, 36%), and the potential for widespread use, the ESAG suggested that there was significant potential for harm.

Evidence of need for a randomized controlled trial

Since the initial list of candidate tests was limited to tests that appeared to have significant evidence gaps, little explicit discussion took place regarding the evidence of need for a randomized controlled trial (RCT). Stakeholders generally acknowledged that every study under consideration had an insufficient evidence base to support clinical implementation. Much of this discussion was interrelated with considerations of clinical and analytic validity, which was reported above.

New priority-setting criteria identified

Several new criteria emerged from the qualitative analysis and did not directly fall under any of the prespecified priority-setting criteria (Table 5).

Table 5.

New criteria identified from the workshop session

Priority-setting criterion	Major themes of discussion
Patient-reported outcomes	What is the relative importance of the psychological impacts of testing in prioritizing candidate genomic technologies?
	Which types of patient-reported outcomes would be most useful as trial end points?
Recruitment	Recruitment difficulties related to competition with industry trials for limited patient populations
	How to recruit adequate numbers of patients with rare genomic variants
Clinical trial ethics	In which situations is it ethically feasible to with-hold treatments on the basis of genomic status?
	Considerations for institutional review board perspectives when reviewing genomic technology comparative effectiveness research trials

The left column shows the new priority-setting criteria that were identified during the workshop discussion, and the right column reports the major themes of the workshop discussion related to the given criterion.

Patient-reported outcomes

First, there was considerable discussion about patient quality of life, and the potential role of patient-reported outcomes, particularly in relation to breast cancer tumor markers, for which several stakeholders suggested that patient concerns about recurrence and their desire for information drove the demand for testing. In fact, many stakeholders thought that the psychological impact of tumor marker testing and results was a compelling research area in itself. As a patient representative stated:

“When we're looking at the benefits of these tests...how the tests impact choices and decision making in care for both the patient and the provider...you really need to look at the decision-making process for the individual...and weigh that...in the incorporation of these tests.”

Recruitment

The topic of trial recruitment was offen discussed as well. Stakeholders noted the potential difficulty of conducting CER trials in NSCLC because many of the potential participants are recruited to competing industry-sponsored trials. In addition, stakeholders thought recruitment was a key issue in fluorescence in situ hybridization testing for epidermal growth factor receptor in NSCLC, arguing that the small target population would make enrollment quite slow.

Clinical trial ethics

Clinical trial ethics also emerged as a key theme. Stakeholders challenged the ethics of designing an ERCC1 expression testing study with a no treatment arm, particularly if enrolling stage II NSCLC patients who would receive treatment according to current standard of care.

“...Nobody will put a stage II patient on study that's not getting chemotherapy.”

Clinician representatives agreed that it would be difficult to obtain the institutional review board approval and to find stage II patients willing to be randomized in a study in which they might not receive chemotherapy (the current standard of care).

Final priority-ranking of candidate genomic technologies

On completion of the workshop, stakeholders identified and ranked the three highest priority candidate technologies (Table 3), on the basis of the rationale below:

1. ERCC1 testing: high population impact, high clinical validity, ability to quickly obtain end points in a trial due to rapid disease progression.

2. EGFR mutation testing: relatively high population impact, high clinical validity, opportunity costs are large as erlotinib is expensive and has significant harms.

3. Breast cancer tumor markers: high population impact, high degree of dissemination into current clinical practice, low clinical validity, opportunity to curb inappropriate use of markers and realize savings.

DISCUSSION

Overview

We evaluated a stakeholder discussion to prioritize six cancer genomic technologies for prospective CER trials. The in-person deliberations led to a revision of the priority ranking, suggesting that in-depth face-to-face deliberation can impact stakeholder priority rankings.

Of the nine original priority-setting criteria, six were the focus: clinical benefits, population health impacts, economic impacts, analytical and clinical validity, clinical trial implementation and feasibility, and market factors. Several criteria were rarely mentioned, with the potential harms of tests most notably absent. Failing to discuss the potential harms could lead to the underestimation of the impact of false positives and negatives of a genomic test on clinical practice and patient outcomes. We also identified three new themes from the workshop discussion: patient-reported outcomes, clinical trial ethics, and trial recruitment. The discussion about patient-reported outcomes highlighted the importance of the impact of genomic testing on quality of life and the selection of trial end points. By considering patient-reported outcomes, potential study designs could be broadened beyond RCTs to address, for example, behavior-related research questions. The theme of clinical trial ethics focused on the ethics of withholding the results of relatively established genomic tests in CER trials and how this might have an impact on patient and physician participation. Finally, the theme of trial recruitment emerged from concerns about adequate accrual given competition from industry-sponsored trials and finding patients with rare genomic variants.

Implications

On the basis of this study, future priority-setting processes may consider these nine prespecified and three new criteria into an explicit framework. Although the inclusion of our new priority-setting criteria can enhance the ability of our prespecified framework to facilitate informed decisions about CER trial priorities, future efforts should be made to identify additional priority-setting criteria that enhance stakeholders’ ability to prioritize research investments.

Comparison with previous findings

Few accounts of cancer genomics research prioritization are in the literature to date. The Evaluation of Genomic Applications in Practice and Prevention (EGAPP) program of the Centers for Disease Control and Prevention and Food and Drug Administration is the most notable example but differs from this study in its objective in key ways.^9,24 Primarily, this study sought to prioritize candidate genomic technologies for evaluation in a CER trial, whereas the EGAPP effort sought to prioritize genomic technologies for evidence reviews.²⁴ Although overlap exists in the priority-setting criteria involved in each, the difference in objective renders different criteria more relevant in each context. In addition, the EGAPP prioritization primarily involves academically oriented stakeholders, whereas this study involved a diverse array of genomic technology end users, including patients, payers, and technology developers.²⁴

Limitations

This study had limitations that are worth noting. First, the perspectives reported herein are not representative of all potential stakeholders in CER of cancer genomic technologies; however, stakeholders were purposely selected to best help the investigators obtain in-depth understanding of the issues that must be considered when prioritizing and designing CER studies in this area.¹⁷ Obtaining in-depth feedback and a high level of stakeholder engagement, which requires significant interaction among participants, would have been difficult to achieve with a larger group. Second, despite their participation through online voting, the 5 of the 13 ESAG members who could not attend in-person meeting did not have a chance to influence the deliberations, which may have had an impact on the final ranking. Still, given the multiple stakeholder groups already represented during the meeting, we believe that the goal of stakeholder engagement—to seek diverse perspectives to help make research more relevant for its end users—was achieved. To see if the results are similar or different, this process can be replicated with other stake-holders. Furthermore, to generalize these findings more broadly, the framework could be tested on a larger sample through alternative methods such as large-scale surveys. Finally, because we started with a preexisting list of criteria, and presented information about those criteria, we may have influenced and/or restricted the range of factors that stake-holders considered and discussed.

The process of prioritizing genomic technologies for future CER can be an extremely complex task to perform. Our findings and revised priority-setting framework can serve as a guide to researchers and stakeholders considering the use of priority-setting frameworks to guide research investments. Although this study specifically examined cancer genomic technologies, this type of framework can also be adapted to other disease areas. Ultimately, this type of approach can potentially assist decision makers in systematically allocating limited resources to CER studies with the intent to maximize public health benefit.