Bridging the Gaps in Personalized Medicine Value Assessment: A Review of the Need for Outcome Metrics Across Stakeholders and Scientific Disciplines

Abstract

Despite the monumental advances in genomics, relatively few health care provider organizations in the United States offer personalized or precision medicine as part of the routine clinical workflow. The gaps between research and applied genomic medicine may be a result of a cultural gap across various stakeholders representing scientists, clinicians, patients, policy makers, and third party payers. Scientists are trained to assess the health care value of genomics by either quantifying population-scale effects, or through the narrow lens of clinical trials where standard of care is compared with the predictive power of a single or handful of genetic variants. While these metrics are an essential first-step in assessing and documenting the clinical utility of genomics, they are rarely followed-up with other assessments of health care value that are critical to stakeholders who use different measures to define value. The limited value assessment in both the research and implementation science of precision medicine is likely due to necessary logistical constraints of these teams; engaging bioethicists, health care economists, and individual patient belief systems is incredibly daunting for geneticists and informaticians conducting research. In this narrative review, we concisely describe several definitions of value through various stakeholder viewpoints. We highlight the existing gaps that prevent clinical translation of scientific findings generally as well as more specifically using two present-day, extreme scenarios: 1) genetically-guided warfarin dosing representing a handful of genetic markers and more than ten years of basic and translational research and 2) next generation sequencing representing genome-dense data lacking substantial evidence for implementation. These contemporary scenarios highlight the need for various stakeholders to broadly adopt frameworks designed to define and collect multiple value measures across different disciplines to ultimately impact more universal acceptance of and reimbursement for genomic medicine.

Introduction

Relatively few health provider organizations in the United States currently offer personalized or precision medicine as part of the routine clinical workflow[1]. Gaps between genomic research and clinical practice may be a result of a cultural gap across various stakeholders. Scientists ranging from basic to translational are trained to assess the health care value of genomics either by quantifying population-scale effects or through the narrow lens of randomized controlled trials (RCTs) where standard of care is compared with the predictive power of a single or handful of genetic variants. This narrow assessment of “value” in both the basic and translational sciences is likely due to necessary logistical constraints of the various teams that contribute to precision medicine. Within academia, limited overlap has traditionally existed across the disciplines of genomics, bioethics, and informatics[2–4]. Recent academic-industry partnerships in precision medicine drug development are bridging the chasm between data and the commercialization of that data[5], but gaps remain in the policy arena where regulations as well as coverage and reimbursement decisions have yet to keep pace with the rapid evolution of genomic technology.

The decision to use genetic information in clinical practice is not straightforward and requires the synthesis of highly fragmented and often non-overlapping pieces of evidence that span three major phases of research and implementation: 1) statistical association, 2) unbiased evaluation, and 3) clinical adoption. The decision to implement is also gauged independently and sometimes collaboratively by at least seven stakeholders, referred to here as the P7: principal investigators (i.e., scientists), providers, (individual) patients, payers, populations (defined by demographics or other dimensions of diversity), pharma (pharmaceutical industry), and policy makers (Figure 1). Several frameworks have been developed to address specific needs for specific stakeholders (e.g., [6]), but few, if any, are used to their fullest extent to ensure that the various metrics of value are measured and made available to the scientific community.

Figure 1. The P7 Stakeholders and Associated Value Metrics related to Precision Medicine Implementation.

At least seven stakeholders are involved in the pathway that leads from basic research findings to genetic testing for personalized or precision medicine: principal investigators (scientists), (individual) patients, populations (defined by dimensions of diversity such as demographics or genetic ancestry), providers, payers, pharma (pharmaceutical industry), and policy makers. The arrows show common interactions between and across stakeholders in studying, treating, selling, paying, and regulating activities related to genetic testing for precision medicine. For each stakeholder, we list a representative mindset (viewpoint) or goal and an associated representative metric.

To describe these challenges in greater depth, we first provide a narrative review of the various value metrics associated with specific stakeholders. We then illustrate this incomplete and fragmented data landscape and decision making process across various stakeholders for two scenarios representing clinical genomic tests available today: 1) genetically-guided warfarin dosing and 2) next generation sequencing. These scenarios represent extremes in data available for evidence-based personalized care while sharing similarities in hurdles to more universal adoption and reimbursement. Finally, we provide a critical analysis of the gaps in data needed for translating basic genetic discoveries into personalized care, and we propose that evaluation frameworks be both broadened and fully adopted to serve these very different stakeholders to better ensure that all value metrics are measured as may be required along the pathway to implementation.

Value metrics across various stakeholders

The two most obvious stakeholders measuring and assessing “value” in the early phases of genetic research and its clinical implementation are the policy makers who allocate funds for research and regulate drug development and the principal investigators or scientists who conduct the research itself. The definition of value for policy makers is often immeasurable and can be influenced by other stakeholders such as individual patients advocating for funds to be allocated for research into their disease or by pharmaceutical companies interested in current and future regulations that will impact their drug development pipeline, market share, and sales[7,8]. Policy makers such as the United States Congress can also be influenced by objective measures of value offered by principal investigators’ testimony on behalf of the larger scientific community. These objective measures include disease frequencies (prevalence and incidence) and disease consequences (morbidity and mortality), sometimes couched in terms of short-term and long-term economic impacts[9].

For the principal investigator or scientist, the emphasis is first on statistical strength of the relationship between the genetic variant(s) and outcome of interest. Metrics at this stage include estimates of the genetic effect size (commonly expressed as odds ratios and betas with accompanying confidence intervals and standard errors, respectively) and statistical significance (e.g., p-values, false discovery rates)[10,11]. Evidence of statistical association is further strengthened by replication studies in independent datasets[12–15].

Genetic association studies are necessary for genomic discovery but are insufficient for immediate translation into clinical care. A statistical relationship between genetic variation and the outcome of interest does not necessarily imply causality. Even if the true causal genetic variant has been identified and associated with the outcome of interest, the burden of actionability or translational urgency is not clear. Most principal investigators emphasize the above-described strength of statistical association coupled with known or presumed (based on in silico data) biological consequence[16–18] to argue for actionabililty and the development of clinical genetic tests. In this phase of research, emphasis is placed on the predictive power of targeted genetic variants alone or in combination with known demographic and clinical factors. Statistical metrics for this phase include variance explained (model R²) and area under the receiver operating curve (AUC).

For many principal investigators or scientists, the strength of genetic association and predictive modeling is the extent of evidence offered to the scientific community in the assessment of a genetic variant’s effectiveness and value in clinical genetic testing. Very few genetic tests, whether assaying a single genetic variant, multiple variants, or the entire genome, have peer-reviewed published data related to analytic validity, clinical validity, and clinical utility, the cornerstones of the evidence review analytic framework for genetic tests[6,19] and metrics valued by several P7 stakeholders, including providers, individual patients, heath care payers, and policy makers. Analytic validity refers to the genetic assay itself, including the laboratory protocol and interpretation of the data. Metrics and other requirements that assure analytic validity in the United States are governed by the Centers for Medicare & Medicaid Services (CMS) Clinical Laboratory Improvement Amendments (CLIA) in collaboration with Centers for Disease Control and Prevention (CDC) and Food and Drug Administration (FDA), and these standards must be met by laboratories that process genetic tests offered in the clinic.

Clinical validity and clinical utility refer to the genetic test’s ability to identify or predict a patient’s status and the risks and benefits associated with the test, respectively[19]. The metrics required to assess clinical validity of a genetic test considered in diagnostics mimic the metrics used to evaluate any clinical laboratory test and include measures such as sensitivity, specificity, positive predictive value, and negative predictive value[20]. While clinical pathology laboratories recognize these standards, both the standards and data required for these metrics are unfamiliar to most scientists conducting genetic association studies for discovery. Strength and predictive power of genetic associations seemingly equate these standards, but neither offer the data nor the metrics required for strict evaluation of a genetic test’s clinical validity.

Clinical utility is arguably the most difficult and expensive metric to measure and is the metric most scrutinized by providers and health care payers. The gold standard study design to discern the benefits of genomics-guided therapy or intervention is the RCT. In a typical RCT study design, genetically-guided drug dosing, for example, is compared with standard of care where outcomes of interest could include adverse events or time to therapeutic dose. The requirement of RCTs for genetic tests has been heavily criticized as many in the field recognize that timely collection of these data is not feasible for all gene-drug pairs in all possible populations[21,22], and that the study design cannot be easily applied to diagnostic genetic tests such as whole exome or whole genome sequencing. Furthermore, RCTs are by design small in sample size and limited in multiple dimensions of diversity such as race/ethnicity and age, consequently limiting their generalizability[23], which potentially impacts both individual patients as well as larger groups or populations. In lieu of RCTs, alternative study designs and data collection efforts such as the Implementing Genomics In Practice (IGNITE) Network’s CYP2C19 multi-center pragmatic (nonrandomized) study[23,24] are being implemented to provide much needed data towards the clinical utility valued by evidence-based medicine.

For tests or treatments with evidence of clinical utility, economic evaluations offer additional dimensions of data relevant to clinical decision making (e.g., providers). As opposed to measures of morbidity and mortality assessed in traditional RCTs to establish clinical utility, economic evaluations known as cost-effectiveness analysis, cost-utility analysis, and cost-benefit analysis define benefits of treatment in terms of quantity of life and/or quality of life[25]. Cost-effectiveness analyses consider unidimensional or straightforward outcomes such as life years gained, cases of disease, or number of deaths prevented whereas cost-utility analyses consider multidimensional variables known as quality-adjusted-life-years or QALYs, a measure that captures a patient’s perceived well-being[26]. Both cost-effectiveness analyses and cost-utility analyses can include other clinically intuitive measures such as number needed to treat and number needed to harm[27]. Cost-effectiveness is not synonymous with cost-savings; that is, a genetically-guided treatment or intervention may be cost-effective, but it may not be less expensive (expressed in monetary terms) than the alternative. Formal economic evaluations of genetic tests using cost-effectiveness analyses with or without QALYs have increased in the past decade and a half, although not at the pace of the development and availability of genetic tests[22,28,29].

Of the three economic evaluations described here, cost-benefit analyses are the only evaluations that examine whether the benefits exceed the costs of intervention in monetary terms. Economic evaluations of genetic tests are generally performed as cost-effectiveness analyses or cost-utility analyses and less so as cost-benefit analyses, as the latter is not heavily used in clinical decision making[30]. While cost-benefit analyses may be valuable to payers and policy makers, it is interesting to note that monetary costs cannot be considered by some stakeholders (such as CMS, a health care payer) in making coverage decisions. Similarly, reimbursement decisions, whether based on value-based care or fee-for-service[31,32], are not considered in formal evaluations of genetic tests. Reimbursement levels and payments in general, however, are of great interest to many stakeholders, including providers, payers, policy makers, and patients[33–35].

Variability in the decision making process

Evidence within and across phases of the decision to implement genetic testing is weighed or valued differently across a variety of stakeholders. For example, different stakeholders rely on different data summaries for the evaluation phase, which can include medical society endorsements, individual RCTs, and systematic reviews or evidence reviews. Even within systematic or evidence reviews there is variability in data regarding types and emphasis. For pharmacogenomics, the Pharmacogenomics Knowledgebase (PharmGKB) offers a synthesis of data related to strength of statistical associations and minor allele frequencies[36] whereas the Clinical Pharmacogenetics Implementation Consortium (CPIC) additionally offers prescribing and dosing recommendations if genetic data on the patient are already available[37]. While both of these resources offer evidence ratings for each drug-gene pair considered, neither synthesize nor offer data related to clinical utility[38]. In contrast, CDC synthesizes available data, including FDA labeling language, CMS coverage decisions, clinical practice guidelines, and systematic reviews, and offers interpretations regarding the strength of evidence for clinical implementation[39].

Regardless of available data, the decision making process is rarely transparent as few stakeholders document the rationale (e.g., [38]) or frameworks considered in their assessments[40]. Decision to implement genetic testing may also hinge on other unmeasured factors such as the provider’s ability to offer the services, where individual-level clinical resources may determine if the testing is performed at point-of-care or preemptively. A provider’s decision to implement could also depend on the availability of a genetic test or testing service, a decision impacted by pharmaceutical companies or third party investment in developing a for-profit test. Decision to invest in the development of clinical genetic tests is a complex calculation based on basic research results (e.g., strength of the genetic association) and population impact (e.g., frequency of outcome and frequency of genetic variants being tested) or market segmentation to predict commercial viability[41]. Policy makers driven by distributive justice can both encourage and enable basic research, translational research, and pharmaceutical drug development through government incentives (such as the Orphan Drug Act for rare diseases). On the other end of the spectrum, patients may consider specific beliefs, concerns (e.g., genetic discrimination), and limitations (e.g., ability to pay both in monetary and other terms) when considering genetic testing if offered by the provider and even when covered by the payer.

To illustrate the above-described variability and availability of value metrics, we describe and consider two contemporary scenarios in genetic testing: 1) genetically-guided warfarin dosing and 2) next-generation sequencing. These two scenarios illustrate two kinds of data extremes along the spectrum that genetic testing offers. The first data extreme is the number of genetic variants assayed in the actual test (in this case, a few versus an entire genome). The second data extreme is the availability of the basic and translational research data needed to assess the genetic test’s suitability for implementation. These scenarios underscore the challenges in filling the data gaps for the various stakeholders involved in moving even well-established basic research findings towards precision medicine.

Scenario 1: Genetically-guided warfarin dosing

Available since the mid-1950s[42], warfarin (Coumadin®) is a common anticoagulant prescribed to prevent stroke and blood clots in at-risk patients[43]. Warfarin is known to have a narrow therapeutic window where too much warfarin is associated with serious bleeding events and too little warfarin leaves the patient vulnerable to the outcome that is being avoided[44,45]. Since the identification of the drug’s target gene VKORC1 [46,47] and the landmark genetic association studies of 2005 that followed[48,49], the pharmacogenomics of warfarin dosing is now well known, particularly for populations of European descent where known genetic variation in and around VKORC1 and CYP2C9 accounts for up to 54% of the observed variability in warfarin dosing compared with 22% of the variance explained by clinical variables alone[50,51]. Based on these strong findings, the FDA quickly revised its label in 2007 to consider the genetic testing for initial warfarin dosing, and again in 2010 to consider individual patient genotypes (if already known) in initial dosing. Despite the strong, replicable genetic effect size and the FDA’s recommendations, CMS issued a 2009 memo that genotyping for warfarin dosing was not beneficial to Medicare recipients and therefore not reasonable and necessary. Instead, CMS declared that the existing evidence supports the Coverage with Evidence Development label, which provides coverage for first-time genotyping of participants in pharmacogenomic RCTs for warfarin dosing. Currently, major medical societies such as the American College of Chest Physicians recommend against the use of genetically-guided warfarin dosing[52,53]. Meanwhile, several large clinics with strong biomedical research resources (e.g., Mayo Clinic[54]) have become early adopters of genetic testing for warfarin dosing. CDC classifies genetic testing for warfarin dosing as “tier 2,” a designation for genomic tests with insufficient evidence to support routine implementation but enough evidence that could be used by providers and patients for decision making[39]. Recent RCTs have had conflicting results in the clinical utility of genetically-guided warfarin dosing[55,56]. Economic studies of the cost-effectiveness of genetically-guided warfarin dosing also conflict or are inconclusive[57]. Most recently, an RCT of genetically-guided warfarin dosing among patients undergoing elective hip or knee arthroplasty treated with perioperative warfarin demonstrated reduced adverse events compared with clinically-guided dosing[58].

Today, few payers reimburse for this genetic test[59,60] labeling it as “investigational,” and most United States healthcare provider organizations do not offer genetically-guided warfarin dosing. Given the American Heart Association, American College of Cardiology, and the Heart Rhythm Society’s 2019 endorsement of non-vitamin K oral anti-coagulants as the standard of care for atrial fibrillation[61], many stakeholders such as providers, patients, populations, and payers now no longer need value metrics related to genetically-guided warfarin dosing. However, despite this recent change in recommendations, it is important to note that there are specific patient groups (e.g., those with mechanical heart valves and moderate to severe mitral stenosis) where warfarin remains the standard of care, and for them, gaps remain across the decision-making framework to implement genetically-guided warfarin dosing.

At the basic genomic discovery or association phase, more data are required in populations of non-European descent (e.g., African-decent populations) to ensure genetically-guided dosing algorithms are optimal for all patients[62–65]. Population differences, including differences in genetic variants, are thought to be a major driver in the differences observed between two large RCTs for genetically-guided warfarin dosing[66]. Additional diverse RCTs are warranted to determine the clinical utility of genetically-guided warfarin dosing, particularly if novel or additional population-specific genetic variants are discovered in the association phase of this process. Also, supporting prospective observational studies as well as real-world economic studies can be undertaken given that a few large providers are offering warfarin-related genotyping pre-emptively. Finally, additional studies are needed to assess patients’ attitudes and beliefs towards genetic testing for warfarin dosing. Recent surveys suggest that there are several barriers to pharmacogenomic testing[67,68], including insurance coverage and cost. In a recent survey of diverse participants in Chicago, most (~82%) patients surveyed wanted the genetic testing performed at no cost to them or for up to $20 USD[69]. Similarly, 44% of surveyed Mayo Clinic patients indicated that they would not pay out-of-pocket for pharmacogenomic testing[70]. Further studies are needed to probe deeper into patient beliefs specifically for genetically-guided warfarin dosing where tangible benefits such as achieving therapeutic dose more quickly and the aversion of life-threatening events such as bleeding may outweigh concerns of out-of-pocket expenses or other concerns such as privacy.

Scenario 2: next generation sequencing

At the other genetic testing extreme are next generation sequencing technologies, which make clinically-ordered whole exome (WES) and now whole genome sequencing (WGS) a possibility. Compared with genotyping or targeted sequencing (such as FDA’s approved and now reimbursable under CMS FoundationOne CDx for solid tumors), WES and WGS are still relatively rare and limited to clinics with strong academic ties[71]. WES/WGS is mainly ordered to diagnose pediatric patients with undiagnosed rare diseases[72], but its potential for pharmacogenomics and individual risk prediction is well recognized[73]. Unlike the warfarin dosing scenario, clinical trials for WES/WGS are not obvious; consequently, little data are available to define value for most stakeholders[74,75]. According to one of the few surveys available[76,77], older adult respondents were interested in WGS if it was covered by their insurance; if not reimbursable, they would be willing to pay on average $1,035 USD for the service[77]. Little other data are available on willingness to pay for WES or WGS across the full spectrum of ages, education level, socioeconomic status, and other dimensions of diversity.

The sparsity of value metric data available for WES and WGS make near-future universal adoption unlikely in the United States under the currently accepted process that requires extensive and often lengthy periods of evaluation followed by issuance of coverage policies by the largest and most influential of all US payers, CMS. For this and other examples of possible precision medicine implementation, the CMS Innovation Center established in 2010 under the Affordable Care Act[78] may be the venue to accelerate such innovative medical technologies as part of the Initiatives to Speed the Adoption of Best Practices. As others have proposed[79], principal investigators and providers could propose CMS demonstration projects around genotyping or next generation sequencing before a patient becomes enrolled in Medicare as part of a routine standard of care. Under this scenario, hospitals systems and providers who opt in would receive a bonus reimbursement credit to their Medicare patient population that is based on the percentage of their non-Medicare population who has participated in the demonstration project. Expansion of this program to actual Medicare recipients would be the next logical step, and a similar bonus credit would be offered. In doing so, providers and their organizations would have more control in decisions to adopt innovative genomic technologies[80], and the payer (CMS under this scenario) would facilitate the creation of real world evidence that could be shared with all stakeholders in the P7 model.

Conclusions

In the two scenarios presented, gaps remain across the three phases of the decision making process and the extent of the gaps is specific to each scenario. The decision to implement genetic testing in a clinical setting is not straightforward and is often based on arbitrary and non-uniform criteria across a variety of stakeholders. The two scenarios presented here encapsulate the major challenges various stakeholders face in amassing or contributing to the information critical for implementation of genetic testing in the clinic. To bridge the gaps between various stakeholder viewpoints and those conducting basic and translational research in precision medicine, existing frameworks designed for the evaluation of genomic tests must be embraced by all stakeholders presented here to ensure that the underlying data for their individual required or desired metrics will be readily available. Ideally, a P7 stakeholder metric inventory for genetic tests would be used to both identify gaps in required knowledge as well as to inspire common language and data references bridging the science and economics of precision medicine. As most measures of “value” for genetic testing are not yet available in the scientific literature, there is substantial opportunity to bridge the cultural and data gaps to define value very much needed for evidenced-based personalized medicine.