Abstract
A key to accelerating the appropriate integration of genomic applications into healthcare in the coming decades will be the ability to assess the tradeoffs between clinical benefits and clinical risks of genetic tests in a timely manner. Several factors limit the ability of stakeholders to achieve this objective, including the lack of direct evidence, the lack of a framework to quantitatively assess risk and benefit, and the lack of a formal analytic approach to assess uncertainty. We propose that a formal, quantitative risk-benefit framework may be particularly useful for assessing genetic tests intended to influence health outcomes, and communicating the potential clinical benefits, harms, and uncertainty to stakeholders. As part of the development process for such a framework, a stakeholder meeting was held in Seattle (Wash., USA) in December of 2008, with the objective of discussing a risk-benefit framework, using warfarin pharmacogenomics as a case study. Participants engaged in focused discussion to elucidate the potential role of genetic test risk-benefit analysis in informing decision-making, categorizing genetic tests and directing research prioritization. This research investigation focuses on qualitative analysis of responses elicited from workshop participants during the proceedings of the workshop session. The major findings of the workshop were: (1) stakeholder support for risk-benefit modeling as a tool to structure discussion of the clinical utility of genetic tests; (2) desire for the modeling process to be iterative, transparent, and parsimonious in its presentation to stakeholders, and (3) some concern with the use of quality-adjusted life-years in the evaluation process. The meeting's findings emphasize the potential utility of risk-benefit analysis in genetic test evaluation, and highlight key areas for future research and stakeholder consensus-building.
Key Words: Genetic testing, Pharmacogenomics, Quality-adjusted life years, Risk-benefit, Stakeholder, Warfarin
Introduction
The technological advances in genetic analysis over the past decade present a significant yet challenging opportunity to improve the public's health through genomics. A defining feature of this challenge is assessing a genetic test's clinical utility, defined as the impact of testing on health outcomes [1]. Most often the clinical utility of a genetic test is not well known or understood at the time of its availability to patients. The most likely reason is the lack of clinical evidence. Such evidence is absent for a multitude of related issues, including: the significant scientific challenges involved in developing evidence on the clinical outcomes of genetic testing, the lack of financial incentives by private industry to invest in such costly studies, the lack of a clear development pathway for genetic testing technologies, and the low regulatory hurdles for the marketing of genetic tests [2,3].
An equally challenging issue is lack of standard evidence criteria for evaluating the clinical utility of genetic tests. Many clinicians and policy-makers expect data from controlled clinical trials to support the use of a genetic test, while others are willing to rely on retrospective analyses and clinical plausibility [4,5,6,7,8]. This lack of consensus about the appropriate level of evidence for clinical benefits and harms, and the lack of a process for evaluating them in an explicit fashion, presents a roadblock on the translational pathway.
A key to accelerating the appropriate integration of genomic applications into healthcare in the coming decades will be the ability to assess the tradeoffs between clinical benefits and clinical risks of genetic tests in a timely manner. Traditional evidence-based methods that rely on direct evidence will have limited usefulness due to lack of data. Approaches for incorporating indirect evidence in the evaluation of preventive services have been developed, but they lack a summary measure of net health benefit and formal assessment of uncertainty [9]. We propose that a decision-modeling-based approach to risk-benefit analysis will be particularly useful for assessing genetic tests intended to influence health outcomes, and communicating the potential clinical benefits, harms, and uncertainty to stakeholders. A formal risk-benefit framework would serve as an adjunct to current evaluation processes, and facilitate the prioritization of genetic tests into risk-benefit categories. Although decision-analytic techniques are beginning to be used to explore the clinical utility of genetic tests, a framework for incorporating their findings into decision-making in a manner that consistently meets stakeholders’ needs has not been developed. Prior research has suggested that engaging stakeholders in collaborative framework development is essential to establishing methodological approaches that appropriately meet their needs in making decisions about genetic test use and coverage [7,8,10]. Thus, we conducted a qualitative research investigation of stakeholders’ perceptions of a decision-modeling-based risk-benefit framework. The overall objective of this study was to elicit and analyze stakeholder feedback on the strengths and weaknesses of this approach for evaluating genetic tests using a directed content approach. The specific objectives were to: (1) present the genetic test risk-benefit framework concept for evaluation by stakeholders; (2) present the risk-benefit framework as applied to a specific genetic testing case study for illustration of key concepts, advantages, and limitations of this method; (3) identify the evidence requirements for genetic testing to be integrated into the delivery of health care; (4) identify general points of stakeholder consensus and disagreement with regard to evaluation of genetic tests for clinical practice. The methodological approach utilized to achieve these objectives is described below.
Methodological Approach
To facilitate the collection of initial feedback regarding the development of a genetic test risk-benefit framework, a stakeholder workshop session was convened in Seattle (Wash., USA) during the University of Washington Genomics Conference held December 3–5, 2008. The University of Washington Resource Center for Health Policy hosted the event, in conjunction with the Center for Genomics and Healthcare Equality, the Pharmaceutical Outcomes Research and Policy Program, and the Center for Genomics and Public Health. This study was deemed institutional review board exempt by the University of Washington Institutional Review Board on the grounds that: (1) individual participants would not be identified; (2) specific stakeholder groups would not be identified (e.g. by organization name), and (3) the proceedings of the workshop session would not be recorded in any audio-visual format. Under these conditions, clinician, genetic test developer, researcher, consumer, and payer stakeholder representatives participated in the session (summarized in table 1). These groups represent the stakeholders who are the end users of recommendations on genetic tests.
Table 1.
Stakeholder attendance at the genomics workshop risk-benefit framework session
| Stakeholder Group | Number of rganizations epresented | Number of stakeholder participants |
|---|---|---|
| Clinicians | 6 | 8 |
| Consumers | 4 | 4 |
| Developers | 8 | 9 |
| Payers | 9 | 9 |
| Researchers | 9 | 12 |
The December 4th session began with a 60-min introduction to the proposed genetic test risk-benefit framework, overview of the rationale for its development, and explanation of a risk-benefit categorization matrix previously developed by the research team. This part of the session was intended to give participants a basic understanding of risk-benefit analysis concepts, methods, and metrics (e.g. quality-adjusted life years, QALYs). Following the introduction, workshop participants were instructed to consider a specific genetic test case study, as described below.
Warfarin Pharmacogenomics Case Study
To facilitate stakeholder feedback, a case study risk-benefit analysis of warfarin pharmacogenomic testing was presented. This case study applied risk-benefit analysis methods to assess the clinical benefits, harms, and uncertainty associated with genotype-guided dosing for patients receiving therapy with the anticoagulant drug warfarin. Warfarin is an effective drug for reducing the risk of clotting events in patients predisposed to clotting events, but it is one of the most common causes of serious adverse drug events leading to hospitalization [11]. There is considerable variation in required therapeutic dose between and within patients. Consequently, patients are required to undergo frequent measurement of anticoagulation status (using the International Normalized Ratio, INR) and subsequent dosage adjustments.
Recent studies have demonstrated an association between 2 specific genetic variants and required warfarin dose [12,13,14]. These studies found that patients who have variants of the genes CYP2C9 and VKORC1 require a lower dose of the drug to achieve values within the targeted INR range. Additionally, studies have demonstrated that patient's genotype status may impact the relative proportion of time spent above, within, and below their targeted INR range, which may in turn influence risk of bleeding and clotting events [15]. However, no randomized trial to date has demonstrated a significant reduction in out-of-range INR values or clinical events in patients randomized to genotype-guided dosing versus standard warfarin dosing [16]. As a result, there is significant uncertainty and controversy about the use of warfarin pharmacogenomic testing, making it an ideal case study for our purposes [4,5].
We presented data from a risk-benefit analysis of warfarin pharmacogenomics, based on a recently developed decision model [17]. The model compares the impact of time in the INR therapeutic range between genotype-guided dosing and standard of care warfarin dosing based on recently available data from a small randomized clinical trial. The estimated time in the therapeutic range was used to project the incidence of major bleeding events and major thromboembolic events in each dosing strategy, and this information was used to project QALYs. In terms of clinical events, the analysis estimated that in a population of 10,000 people, genetic testing would reduce the number of major bleeds by 17, yet increase the number of major thromboembolic events by 3 [17]. The analysis estimated genotype-guided dosing would increase QALYs by 0.004 (1–2 days) relative to standard of care dosing [17]. Uncertainty was evaluated using 1-way and probabilistic sensitivity analysis. The results of probabilistic sensitivity analysis indicated a 74% probability that QALYs would be increased with genotype-guided dosing, and a corresponding 26% probability they would decrease [17].
Data Collection Format
Following the presentation of the risk-benefit framework overview, warfarin case study, and initial stakeholder questions, participants were asked to respond to 8 questions (table 2) related to the genetic test risk-benefit framework. To qualitatively evaluate these stakeholder responses, directed content analysis methods were utilized. In this analytical approach, existing theory is used to identify key content area codes, stakeholder feedback is elicited through open-ended questions, and stakeholder feedback is coded and evaluated according to the pre-specified content area codes [18]. We employed this qualitative method because our goal was to evaluate feedback about key aspects of the risk-benefit framework, and directed content analysis provides a means to refine and conceptually extend such frameworks [18]. Each of the 8 questions posed to stakeholders was intended to elicit feedback related to 1 of 3 directed content area codes: decision-modeling process, projecting QALYs and clinical events, or the risk-benefit categorization matrix and policy applications. Each of these content area codes represents a key area of uncertainty identified in the literature and related to the application of risk-benefit analysis to genetic test evaluation [8,10,19].
Table 2.
Questions and content area codes from the genomic workshop risk-benefit framework session
| Content area code | Associated question |
|---|---|
| Decision modeling |
|
| Projecting QALYs and clinical events |
|
| Risk-benefit categorization matrix and policy applications |
|
The workshop participants were initially given 10 min to individually contemplate, record and organize their responses to the 8 questions regarding the use of formal risk-benefit analysis to assess the clinical utility of genetic tests (as outlined in table 2). Next, participants were grouped by professional affiliation and assigned to 1 of 5 breakout session stakeholder groups. Each stakeholder group was asked to select a group spokesperson to document major points of discussion and facilitate consensus building within the group. The stakeholder groups were then given 45 min to discuss the 8 questions, with each group member reporting their individual thoughts, and reacting to the thoughts of others. At the conclusion of the 45-minute stakeholder breakout session, each group spokesperson reported their group's findings to all participants. Discussion followed each group report, and 3 dedicated note-takers recorded major points. This process is depicted in figure 1.
Fig. 1.
Schematic diagram of stakeholder meeting.
Participants were informed that the focus of the quantitative framework were the clinical risks and benefits associated with genetic testing, but not costs, although such an approach can lay the foundation for cost-effectiveness analyses.
The workshop discussion points were compiled in a Microsoft Excel spreadsheet and analyzed using directed content analysis methods [18]. Participant comments were classified by predetermined content area codes (i.e. decision-modeling process, projecting QALYs and clinical events, and the risk-benefit categorization matrix and policy applications) and comments that did not fit into any of the pre-defined categories were compiled under an ‘other’ category. Within each content area code category, stakeholder comments were analyzed to determine points of consensus, as well as points of disagreement.
Results
The major findings of the stakeholder meeting are summarized below, and grouped according to the 3 directed content area codes.
Decision Modeling (Questions 1 and 2)
The stakeholder groups uniformly communicated a desire for genetic test decision models to balance the precision gained via structural complexity with considerations for making the model easy for stakeholders to understand and communicate to others. Clinician stakeholders initiated discussion of this issue with their comment that they would like to see a risk-benefit model that is relatively simple, but that is simultaneously trustworthy and reasonable. Clinicians also communicated a desire for an iterative modeling process that incorporates emerging evidence as it becomes available. The consumer, developer, payer and researcher stakeholder groups all supported the idea of a parsimonious and iterative modeling process.
The stakeholder groups also agreed that, at minimum, risk-benefit models provide a ‘starting point’ from which critical variables can be identified, and future research priorities can be explicitly evaluated. The clinician stakeholder group noted that a risk-benefit model could help to guide research investment to address shortcomings in clinical evidence identified during the modeling process. All stakeholder groups expressed a common sentiment that decision modeling could prove useful in genetic testing evaluation by highlighting key variables, explicitly stating assumptions, and providing timely feedback about emerging technologies. However, emphasis was placed on the issue of a ‘timely’ risk-benefit modeling process, as stakeholders noted that results would need to be made available rapidly in order to have optimal utility for decision-makers.
Despite concerted efforts to elicit feedback about the necessary level of evidence required for reliable decision modeling (e.g. randomized controlled trials, observational studies), there were few direct comments from stakeholders indicating the preferable level of evidence for model inputs. The developer stakeholder spokesperson noted that the group did not feel that randomized controlled trial data were necessary for all model inputs. Rather, the developer group felt that the necessity of randomized controlled trials was dependent on the strength of existing data. During this discussion, the researcher stakeholder spokesperson commented that ‘basic association data’ should be available for model inputs, but also specified that in the absence of such data, modeling could be carried out to attempt to understand downstream risks. Additionally, the developer stakeholder spokesperson suggested that as the potential risk and cost of tests rises, so should the level of evidence (e.g. reliance on surrogate markers would transition to reliance on clinical endpoints). The other stakeholder groups did not directly address the level of evidence question.
Several stakeholder groups commented on the need for a risk-benefit model to capture the intrinsic and psychosocial value provided to patients and healthcare providers via genetic test results. This position was strongly advocated by the consumer stakeholder group, and was supported by the clinician stakeholder group. The discussion of this issue did not progress to the point of specifying how the stakeholders would like to see psychosocial and/or intrinsic value incorporated as inputs into decision models. The payer, researcher, and developer stakeholder groups provided limited feedback on this issue.
Projecting QALYs and Clinical Events (Questions 3 and 4)
Stakeholders generally acknowledged the potential utility of QALY-based analysis, but expressed discomfort with using QALYs to inform decision-making. The researcher stakeholder group spokesperson summarized this mixed sentiment, stating ‘QALYs are a very useful measure’ but also noting trepidation because QALYs ‘don't capture all aspects of value’. In response to this statement, the developer group spokesperson noted that despite the fact that QALYs can be problematic because they do not capture all aspects of value, risk-benefit modeling with QALYs does have the merit of ‘providing explicit means to compare different technologies’. Throughout the group discussion of QALYs, stakeholders predominately focused on the ‘quality-adjusted’ aspect of the metric. The payer group spokesperson summarized this sentiment, noting ‘We like the “LY” part of the QALY more than the “QA” part; the QA part is open to interpretation in how it is measured and utilized’. When asked to expand on how the ‘quality-adjusted’ component of the QALY is ‘open to interpretation’, the payer group spokesperson noted that the health state utility values utilized in the calculation of QALY estimates are not always derived using the same methods and may not be consistent across disease states. The groups uniformly communicated a sense that QALYs are difficult to understand and communicate, are open to subjective interpretation, and may consequently detract from the potential value of a risk-benefit framework for genetic testing. The developer, payer, and researcher groups all noted that incorporating QALYs into the model would likely result in arguments about the methods utilized to arrive at given QALY values. Participants also questioned the ability of a quality-adjusted metric to capture all relevant value to stakeholders, specifically with regard to the psychosocial aspects of knowledge created by genetic testing results.
There was mixed stakeholder feedback regarding the use of surrogate markers in a genetic testing risk-benefit model, relative to estimates of clinical events. Clinician and consumer stakeholders commented that surrogate markers, like INR in the warfarin case study, might be more actionable for healthcare providers and patients compared to estimates of clinical events. However, the consumer stakeholder spokesperson also noted that many patients do not understand surrogate markers, and the risk-benefit model presents an opportunity to more clearly communicate with patients in terms of actual outcomes. Additionally, the consumer group commented that ‘absolute values are easier to understand compared to relative values’, indicating another reason why clinical outcomes data could be preferable to genetic test stakeholders.
Risk-Benefit Categorization Matrix and Policy Applications (Questions 5–8)
There was limited stakeholder feedback regarding the design and utilization of the draft risk-benefit categorization matrix (fig. 2). The researcher stakeholder group addressed this issue most directly, noting that even with a risk-benefit categorization matrix, the thresholds for establishing the level of uncertainty are still subjectively assigned. Additionally, consumers noted that even if a given genetic test falls into the neutral risk-benefit cell from a clinical utility perspective, it could still have personal utility to individual patients. The consumer stakeholder spokesperson noted, ‘we are not dealing with yes/no answers, we are dealing with uncertainty on many levels, and that is difficult for us’. Further discussion of the risk-benefit matrix led to the suggestion from the payer stakeholder spokesperson that the matrix be stratified by the potential cost of the genetic test in question, with higher-cost tests subjected to a more nuanced matrix (like the one displayed in fig. 2), and lower-cost tests subjected to a simplified version of the matrix. This suggestion was made with the underlying logic that payers may be willing to pay for lower-cost tests regardless of the clinical utility they provide, while higher-cost tests will be subjected to a greater level of scrutiny. The clinician and developer groups did not directly address the design or utility of the risk-benefit categorization matrix.
Fig. 2.
Draft risk-benefit categorization matrix.
There was a varied array of stakeholder comments regarding the use of a genetic test risk-benefit framework to inform policy and clinical decision-making. The clinician stakeholder group focused much of their feedback on the need for investments in genetic testing infrastructure and personnel before genetic testing can become a more common element of clinical practice. Specifically, they noted that genetic test results would need to be available in 8–10 h, not 8–10 days, and this level of performance would likely require more local laboratories to invest in equipment and staff to conduct genetic tests. Alternatively, the researcher stakeholder group was focused on ensuring that the ‘consumers’ of model output are adequately informed about the parameters and assumptions, so that clinical and policy decisions properly utilize risk-benefit information. The developer stakeholder group noted that risk-benefit modeling could prove useful in making comparisons between various genetic tests. The payer group noted that many organizations are already pursuing ways to evaluate the clinical utility of genetic tests, but risk-benefit modeling may significantly benefit smaller organizations that do not currently have the resources to invest in this type of evaluation.
Consumers expressed the opinion that genetic test results have intrinsic value, even if they do not provide clinically actionable information, and this intrinsic value should be given consideration in policy decisions. This theme was common throughout the comments of the consumer group. None of the other stakeholder groups directly responded to this point, or commented on this issue in reference to informing policy or medical decision-making.
The stakeholder groups presented a diverse array of concerns regarding the use of a risk-benefit framework in the evaluation of genetic testing. The clinician group made several comments about the shortcomings of applying aggregate data to individual patients in clinical practice. The clinician stakeholder group spokesperson noted, ‘relying on modeled data is like placing a series of bets, most patients are neither benefitted nor harmed, most get nothing from this’. The clinician group also noted that the risk-benefit framework does not have a metric to capture extrinsic factors, such as if there is an alternative therapy in a pharmaceutical/device/diagnostic manufacturer's pipeline that will soon make the given technology under evaluation obsolete. The consumer stakeholder group expressed concern that risk-benefit framework development process may fail to capture and utilize patient perspectives.
Miscellaneous Comments Not Directly Related to Content Area Codes
While the questions posed to the genetic test stakeholders were intended to elicit responses related to the pre-defined content area codes, we did receive many comments that were not classifiable within the three content areas. Many such comments focused on issues of uncertainty about test performance, model parameters, and clinical application of genetic tests. One clinician stakeholder noted that they were concerned about a ‘one size fits all’ approach to evaluating genetic tests, because different tests are accompanied by different levels and types of uncertainty. A researcher stakeholder noted that there is significant uncertainty surrounding the issue of whether patients will even be willing to undergo genetic testing, suggesting that the genetic test risk-benefit framework could only impact those willing to undergo such testing. Lastly, a clinician stakeholder noted that there might be so much uncertainty involved in the use of a genetic test, that a risk-benefit approach may not be able to produce valuable insights.
Other miscellaneous feedback focused on the potential drawbacks of over-reliance on the results of risk-benefit modeling. One clinician stakeholder noted that most practicing clinicians would not have the time to stay up to date on the studies used to inform model parameters, so there could potentially be a lack of scrutiny of model results. Additionally, there were questions of how risk-benefit model results would be communicated to clinicians and patients, and if such information transfer could occur in a manner that is clear, concise, and unbiased. Also, research, developer, clinician, and consumer stakeholder group members expressed concern that the assumptions and parameter ranges utilized in risk-benefit models would be difficult to understand and communicate.
Discussion
The stakeholder meeting helped elucidate and document the perspectives of a diverse array of genetic testing stakeholders regarding the use of a risk-benefit framework to evaluate the clinical utility of genetic tests. While aggregate stakeholder group perspectives were complementary in many respects, the stakeholder feedback process highlighted the reality that individual stakeholders, both within and between groups, have varied perspectives in a number of key areas. Specifically, there was a lack of consensus with regard to decision-modeling methods, appropriateness of policy and clinical decisions based on models, and the overall strengths and weaknesses of a modeling approach to evaluate genetic tests.
One of the workshop's most important findings was that stakeholders are generally receptive to risk-benefit modeling as a tool to evaluate the potential clinical utility of genetic tests. While more nuanced discussions of modeling methods and applications revealed disagreement both within and between groups, the vast majority of the workshop participants acknowledged the potential value of having a quantitative model-based tool to evaluate risk and benefit. Stakeholders noted that a risk-benefit framework has the potential to guide a structured discussion of the genetic testing evidence, highlight key drivers of health outcomes, and assist in prioritizing future research investments. However, stakeholders also expressed concerns about not considering cost in the modeling process, the use of QALYs, and issues of uncertainty in model parameters that may limit the impact of risk-benefit-modeling results.
Surprisingly, there was relatively limited stakeholder feedback regarding the necessary level of evidence to inform genetic test risk-benefit modeling. Some comments indicated that stakeholders would consider requiring varying levels of evidence based on the cost of the given test and the size of the patient population that would potentially undergo testing (in addition to the potential harm to patients). Specifically, several stakeholders suggested that the risk-benefit categorization matrix (fig. 2) be stratified by the potential budget impact of the given test, and the developer group spokesperson commented that the required level of evidence should be tied to the cost of the test in question. This feedback illustrates one way that cost considerations could be incorporated into risk-benefit analysis if stakeholders feel that different levels of cost warrant different risk-benefit considerations. However, varying the levels of evidence needed based on a test's cost or budget impact may make generalized conclusions about evidence requirements challenging. Additionally, the policy implications of an approach that stratifies genetic test evaluation by cost or budget impact are far-reaching. At the most fundamental level, a cost or budget impact stratified approach to risk-benefit evaluation implies that lower cost or lower budget impact tests can be evaluated with a variable (and presumably lower) standard of evidence relative to high-cost or high-budget-impact tests. The cost and/or budget impact of a given test does not, however, necessarily indicate the appropriate amount of scrutiny that test should receive. Consequently, it is plausible to envision circumstances where a stratified approach to risk-benefit categorization could either delay implementation of tests with high clinical utility or allow implementation of tests with low or highly uncertain clinical utility. Additionally, stratification by cost or budget impact raises questions of which costs to consider and which analytic perspective to assume (e.g. payer perspective vs. societal perspective) [20]. For these reasons, evaluation of genetic tests with a stratified risk-benefit matrix may lead to unintended policy consequences and prove to be an additional roadblock in building stakeholder consensus about risk-benefit modeling in genetic test evaluation. Additional research should seek to investigate the policy implications of risk-benefit matrix stratification based on cost, reimbursement, budget impact, or other factors that may potentially modify evidence considerations in the risk-benefit matrix.
Another critical finding from the workshop session was that a majority of stakeholders communicated a desire to avoid use of QALYs as a metric to value clinical benefit and risk in the modeling process. Stakeholders repeatedly noted that QALYs are often difficult to understand, discuss, and explain to others; and this factor limits the utility of a QALY-based modeling approach. Additionally, numerous stakeholders commented that they do not favor a QALY-based modeling approach because the quality adjustment metrics are subjective and open to considerable disagreement. While not all stakeholders overtly opposed the use of QALYs in the modeling process, there were only a few comments in support of a QALY-based approach. This finding is consistent with prior research evaluating the role of QALYs in US health policy [21,22]. Neumann and Greenberg [22] noted that despite endorsement by entities like the US Panel on Cost-Effectiveness in Health and Medicine, the socio-political culture and structure of the US healthcare system presents barriers to widespread incorporation of QALYs in decision-making.
Lastly, there was a clear desire among stakeholders for a risk-benefit modeling process that is rigorous, iterative, and transparent. However, while stakeholders want these conditions to be satisfied during the research and development of models, they would prefer to be presented with parsimonious model versions that are easy to understand and communicate to others. Stakeholders in numerous groups commented that overly complicated methods or model structures could hinder the use of risk-benefit modeling in clinical and policy decision-making. This feedback is consistent with prior research findings [8,23,24].
The findings of this work have several important limitations worth noting. Given the vast and diverse array of genetic test stakeholders, the study's primary limitation was that the workshop session was able to involve only a relatively small subset of stakeholders. Because there are many key informants that were not able to participate in this event, future work should focus on establishing a dialogue with a larger group of genetic test stakeholders. Specifically, future efforts should seek to achieve greater participation from patient groups, clinical guideline development groups, and pertinent specialist clinician groups. Additionally, this work was limited by the single (warfarin) case study discussed during the workshop session. While the warfarin case functioned to highlight many of the key issues involved in evaluating genetic tests, alternative case studies should be explored to reveal additional areas of uncertainty in the evaluation of genetic tests. A final limitation was that not all stakeholders had in-depth knowledge of risk-benefit analysis methods and metrics prior to the workshop session. Because stakeholder participants ranged from health economists to physicians to patient advocacy representatives, it is assumed that stakeholders had drastically varying levels of knowledge concerning risk-benefit analysis and measures such as QALYs. While we tried to address this issue at the beginning of the session by presenting a 60-minute overview of the risk-benefit framework approach, it is likely that some stakeholders were limited in their ability to respond to some questions based on a lack of familiarity with the given issue.
We believe that the findings of this study are important because they summarize key stakeholder perspectives that have not previously been reported in the genomics literature. The findings of this investigation indicate that future research efforts should seek to further examine stakeholder evidence requirements for risk-benefit model inputs, more clearly define the role of QALYs in informing evaluation of clinical utility, and further assess how stakeholders utilize genetic test results in ‘real-world’ clinical and policy decisions. The merits of QALY-based analytic methods should also be weighted against alternative analytic methods, such as discrete choice experiments and standard preference assessment approaches [23,25]. In accordance with the stakeholder perspectives asserted at this workshop session, all such efforts should strategically target genetic test evaluation methods that are accurate, clear, concise and readily applicable to clinical and healthcare policy decisions.
Acknowledgements
This publication was supported in part by Cooperative Agreement No. 1U18GD000005 from the Centers for Disease Control and Prevention. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the Centers for Disease Control and Prevention. This work was supported in part by the Center for Genomics and Healthcare Equality (Grant P50 HG003374 from the National Institutes of Health).
References
- 1.Holtzman NA, Watson MS. Promoting Safe and Effective Genetic Testing in the United States: Final Report of the Task Force on Genetic Testing. Baltimore: Johns Hopkins; 1998. [PubMed] [Google Scholar]
- 2.Garrison LP, Veenstra DL, Carlson RJ, Carlson J, Meckley L: Backgrounder on Pharmacogenomics for the Pharmaceutical and Biotechnology Industries: Basic Science, Future Scenarios, Policy Directions. University of Washington, Pharmaceutical Outcomes Research and Policy Program, 2007.
- 3.Garrison LP, Austin MJ. The economics of personalized medicine: a model of incentives for value creation and capture. Drug Inf J. 2007;41:501–509. [Google Scholar]
- 4.Lesko LJ. The critical path of warfarin dosing: finding an optimal dosing strategy using pharmacogenetics. Clin Pharmacol Ther. 2008;84:301–303. doi: 10.1038/clpt.2008.133. [DOI] [PubMed] [Google Scholar]
- 5.Garcia DA. Warfarin and pharmacogenomic testing: the case for restraint. Clin Pharmacol Ther. 2008;84:303–305. doi: 10.1038/clpt.2008.131. [DOI] [PubMed] [Google Scholar]
- 6.Evaluation of Genomic Applications in Practice and Prevention Working Group Recommendations from the EGAPP Working Group: can tumor gene expression profiling improve outcomes in patients with breast cancer? Genet Med. 2009;11:66–73. doi: 10.1097/GIM.0b013e3181928f56. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Tunis SR. Economic analysis in healthcare decisions. Am J Manag Care. 2004;10:301–304. [PubMed] [Google Scholar]
- 8.Rogowski WH, Grosse SD, Khoury MJ. Challenges of translating genetic tests into clinical and public health practice. Genetics. 2009;10:489–495. doi: 10.1038/nrg2606. [DOI] [PubMed] [Google Scholar]
- 9.Petitti DB, Teutsch SM, Barton MB, Sawaya GF, Ockene JK, DeWitt T. Update on the methods of the U.S. Preventive Services Task Force: insufficient evidence. Ann Intern Med. 2009;150:199–205. doi: 10.7326/0003-4819-150-3-200902030-00010. [DOI] [PubMed] [Google Scholar]
- 10.Phillips KA, Veenstra DL, Sadee W. Implications of the genetics revolution for health services research: pharmacogenomics and improvements in drug therapy. Health Serv Res. 2000;35:128–140. [PMC free article] [PubMed] [Google Scholar]
- 11.Gurwitz JH, Field TS, Judge J, et al. The incidence of adverse drug events in two large academic long-term care facilities. Am J Med. 2005;118:251–258. doi: 10.1016/j.amjmed.2004.09.018. [DOI] [PubMed] [Google Scholar]
- 12.Rieder MJ, Reiner AP, Gage BF, et al. Effect of VKORC1 haplotypes on transcriptional regulation and warfarin dose. N Engl J Med. 2005;352:2285–2293. doi: 10.1056/NEJMoa044503. [DOI] [PubMed] [Google Scholar]
- 13.Gage BF, Eby C, Milligan PE, Banet GA, Duncan JR, McLeod HL. Use of pharmacogenetics and clinical factors to predict the maintenance dose of warfarin. Thromb Haemost. 2004;91:87–94. doi: 10.1160/TH03-06-0379. [DOI] [PubMed] [Google Scholar]
- 14.Hillman MA, Wilke RA, Caldwell MD, Berg RL, Glurich I, Burmester JK. Relative impact of covariates in prescribing warfarin according to CYP2C9 genotype. Pharmacogenetics. 2004;14:539–547. doi: 10.1097/01.fpc.0000114760.08559.dc. [DOI] [PubMed] [Google Scholar]
- 15.Limdi NA, McGwin G, Goldstein JA, et al. Influence of CYP2C9 and VKORC1 1173C/T genotype on the risk of hemorrhagic complications in African-American and European-American patients on warfarin. Clin Pharmacol Ther. 2007;83:312–321. doi: 10.1038/sj.clpt.6100290. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Anderson JL, Horne BD, Stevens SM, et al. Randomized trial of genotype-guided versus standard warfarin dosing in patients initiating oral anticoagulation. Circulation. 2007;116:2563–2570. doi: 10.1161/CIRCULATIONAHA.107.737312. [DOI] [PubMed] [Google Scholar]
- 17.Meckley LM, Wittkowsky AK, Rieder MJ, Rettie AE, Veenstra DL. A policy model to evaluate the benefits, risks and costs of warfarin pharmacogenomic testing. Pharmacoeconomics. 2010;28:61–74. doi: 10.2165/11318240-000000000-00000. [DOI] [PubMed] [Google Scholar]
- 18.Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15:1277–1288. doi: 10.1177/1049732305276687. [DOI] [PubMed] [Google Scholar]
- 19.Garrison LP, Towse A, Breshahan BW. Assessing a structured, quantitative health outcomes approach to drug risk-benefit analysis. Health Aff. 2007;26:684–695. doi: 10.1377/hlthaff.26.3.684. [DOI] [PubMed] [Google Scholar]
- 20.Garrison LP, Mansley EC, Abbott TA, Bresnahan BW, Hay JW, Smeeding J. Good research practices for measuring drug costs in cost-effectiveness analyses: a societal perspective. Value Health. 2010;13:8–13. doi: 10.1111/j.1524-4733.2009.00660.x. [DOI] [PubMed] [Google Scholar]
- 21.Garrison LP. On the benefits of modeling using QALYs for societal resource allocation: the model is in the message. Value Health. 2009;12:S36–S37. doi: 10.1111/j.1524-4733.2009.00520.x. [DOI] [PubMed] [Google Scholar]
- 22.Neumann PJ, Greenberg D. Is the United States ready for QALYs? Health Aff. 2009;28:1366–1371. doi: 10.1377/hlthaff.28.5.1366. [DOI] [PubMed] [Google Scholar]
- 23.Johnson FR. Moving the QALY forward or just stuck in traffic? Value Health. 2009;12:S38–S39. doi: 10.1111/j.1524-4733.2009.00521.x. [DOI] [PubMed] [Google Scholar]
- 24.Kind P, Lafata JE, Matuszewski K, Raisch D. The use of QALYs in clinical and patient decision-making: issues and prospects. Value Health. 2009;12:S27–S30. doi: 10.1111/j.1524-4733.2009.00519.x. [DOI] [PubMed] [Google Scholar]
- 25.Grosse SD, McBride CM, Evans JP, Khoury MJ. Personal utility and genomic information: look before you leap. Genet Med. 2009;11:575–576. doi: 10.1097/GIM.0b013e3181af0a80. [DOI] [PMC free article] [PubMed] [Google Scholar]