Summary
Prior to integration into clinical care, a novel medical innovation is typically assessed in terms of its balance of benefits and risks, often referred to as utility. Members of multidisciplinary research teams may conceptualize and assess utility in different ways, which has implications within the translational genomics community and for the evidence base upon which clinical guidelines groups and healthcare payers make decisions. Ambiguity in the conceptualization of utility in translational genomics research can lead to communication challenges within research teams and to study designs that do not meet stakeholder needs. We seek to address the ambiguity challenge by describing the conceptual understanding of utility and use of the term by scholars in the fields of philosophy, medicine, and the social sciences of decision psychology and health economics. We illustrate applications of each field’s orientation to translational genomics research by using examples from the Clinical Sequencing Evidence-Generating Research (CSER) consortium, and we provide recommendations for increasing clarity and cohesion in future research. Given that different understandings of utility will align to a greater or lesser degree with important stakeholders’ views, more precise use of the term can help researchers to better integrate multidisciplinary investigations and communicate with stakeholders.
Introduction
Rapid innovation in genetics and genomics over the past two decades has generated a significant need for translational research. Given that evidence of the utility of a novel medical innovation or diagnostic test is generally required prior to clinical integration, the generation of such evidence is central to clinical translational genomics research.1 The term “utility” is ubiquitously used in discussions about investigations that attempt to understand and describe various impacts of genomic sequencing. Utility can, however, take on multiple connotations that vary according to disciplinary culture. Without an accompanying definition, use of the utility construct can thus create ambiguity within multidisciplinary translational research teams. Lack of specificity in definition, usage, and approach to measurement of utility can be a barrier to cohesive multidisciplinary research efforts and exacerbate the challenges faced by clinicians, administrators, and policy makers in integrating genomic sequencing into patient care.2
The question of how to achieve optimal clinical integration of genomic sequencing is being addressed across the globe in health care systems with various approaches to health technology assessment and payment for clinical services.3,4 In the United States, the Clinical Sequencing Evidence-Generating Research (CSER) consortium is one example of a multidisciplinary effort to generate evidence that can support clinical integration of genomic sequencing.5 The CSER consortium is comprised of seven research projects taking place within a clinical framework, operating at the “research-clinical interface” of genomic medicine.6 The consortium aims to generate data on the utility, or impacts, of clinical genomic sequencing, focusing on the inclusion of patient groups who have been historically underrepresented in genomics research.5 To ensure a broad understanding of all relevant impacts on patients, their families, and society, the CSER consortium has brought together researchers with diverse disciplinary backgrounds.7
Diversity of perspective and a multidisciplinary team science approach are substantial strengths of the consortium as it works to develop a more complete and precise understanding of the utility of clinical genomic sequencing.7,8 These strengths, however, revealed a related challenge: the various ways in which utility can be conceptualized and measured. Investigators held different ideas about precisely what to assess and which methodological approach to use, which indicated a need for clarification of the conceptual foundations of utility to support collaborative, theoretically grounded research efforts with minimal opportunities for misunderstanding and ambiguity about research design and result interpretation.
This paper describes how the term “utility” is defined and used within the fields of philosophy, medicine, and two related social sciences—decision psychology and health economics. These fields are of particular interest because the concept of utility is integral to each yet spoken about differently. Moreover, these fields have produced scholars that have helped shape evaluations of genomic sequencing applications, including as a part of the CSER consortium. For each field, following the description of the term, we draw upon our own experience as CSER investigators and provide examples to illustrate how conceptualizations of utility are operationalized within one translational genomics research effort. We then discuss how various perspectives, properly understood, can be integrated into translational research collaborations. We ultimately seek to encourage conceptual clarity, foster more precise use of the term, and illuminate how findings from various perspectives inform clinical integration. Awareness of similarities and differences in the ways in which utility is discussed, as well as the implications for evaluative criteria, can advance evidence development and communication with stakeholder groups.
Utility in philosophy
Philosophy definition and measurement
The concept of utility as it is now used in philosophy was developed by Bentham in his work on utilitarianism. This consequentialist theory holds that “the greatest happiness of the greatest number is the foundation of morals and legislation,” meaning that in any given circumstance, the morally right action is the one that produces the most good, or utility, for the most people.9 Bentham defined utility as “a property associated with any object or action associated with an increase of pleasure or the avoidance of pain.”9 Under this conceptualization, anything that holds positive value or avoids things with negative value has utility. His understanding of utility is univocal and hedonistic, leading him to argue that pleasure and pain are the only phenomena that matter in the ethical evaluation of an action.
Mill followed Bentham’s utilitarian approach, using the terminology of “happiness” while connecting that concept closely to pleasure.10 He stated that, “‘utility’ or the ‘greatest happiness principle,’ holds that actions are right in proportion as they tend to promote happiness; wrong as they tend to produce the reverse of happiness. By happiness is intended pleasure and the absence of pain; by unhappiness, pain and the privation of pleasure.”10 Mill’s understanding of pleasure, however, is more complex than Bentham’s in that some types of pleasure are more valuable than others. The pleasure deriving from intellectual or social pursuits, he argues, is of a different and more substantive nature than physical enjoyment. In making this claim, Mill triggers an evolution in the concept of utility from a simple property into a multi-faceted idea.
Later works by Moore11 and Sen12 develop this idea further, advocating for a pluralistic view of utility. This approach opens the door for inclusion of a wider set of phenomena that could constitute utility. It incorporates not only pleasure but also aesthetics, desire fulfillment, and basic capabilities. Further, the view holds that these various facets of utility can be integrated into a whole—understood as a complex, cohesive unit—instead of simply an aggregate of individual utilities. Moore’s and Sen’s views enable a broader array of considerations to be identified as morally relevant and therefore allow more complete and nuanced ethical evaluation.
Griffin fleshes out and clarifies this pluralistic understanding of utility into three types of goods.13 Although he uses the term “well-being” rather than “utility,” his work distinguishes among three ways in which something can hold positive value for someone. First, something that brings about positive mental states such as happiness or pleasure contributes to a person’s well-being and therefore has utility. Second, an object or state of affairs can have utility if it helps to satisfy a person’s preferences, regardless of whether the person experiences a positive mental state as a result. Finally, there are some things that are objectively good for people even if they do not desire them or experience pleasure when they have them. Griffin’s approach can be used to delineate different sources of utility under the broader umbrella of a person’s well-being.
While the concept of utility is most often described and used in a positive sense to refer to people’s well-being, it can also be framed in a negative sense with the term “disutility,” or ill-being. Very little work has been done by philosophers on the idea of ill-being, perhaps on the assumption that it is simply the inverse of utility and follows the same patterns and rules that utility does. Some, however, have argued that this assumption may not be justified.14,15 Regardless of its precise operationalization, the concept of disutility can be described as capturing those things that bring a person pain, frustrate their desires, or are objectively bad for them.
Bentham’s hedonistic calculus, an algorithm by which the amount of pain or pleasure resulting from an action is determined, lays out a process for measuring utility. It uses various characteristics of pleasure or pain, such as intensity and duration, to assign that experience a univocal value that can then be compared with the value of other experiences. These comparisons form the basis for the moral evaluation of actions: the right or praiseworthy action is the one that produces the most utility. Utilitarian moral theory derives from this approach.
Subsequent philosophers have proposed refinements of and reinterpretations of Bentham’s classic version of utilitarianism. While some have argued that the rightness of an act should be judged on the net utility of that particular act (the consequences associated with an individual incident—e.g., a person’s telling the truth), others have said that judgment should be based on whether the action is consistent with a utility-maximizing rule (the consequences that would occur if everyone followed a given rule—e.g., the general agreement that people will tell the truth).16,17 Scholars have also debated whether utility or disutility should weigh more in the ethical calculus.18 Implications of distributing utility in different ways among the members of a population have also been considered,19 including whether it is morally appropriate to promote the average utility among its members, justifying a small population of very happy people, or the total utility of a population, justifying a large population of people with relatively lower average happiness.20
In philosophy, then, utility is used as a generic term that points to something of positive value. Under all accounts, it provides a single descriptor to summarize the varied positive and negative outcomes of an action. Utility therefore allows otherwise incommensurate experiences, options, and objects to be compared. As a result, it can be used to ground moral evaluation of decision making. If a decision or action generates utility—specifically, more utility than the other options—we have good moral reasons to make that choice.
Philosophical perspective on utility in genetic medicine
From a philosophical perspective, genomic medicine has the potential to produce a wide range of utilities and disutilities. The health benefits for patients made possible by genetic diagnosis and therapy are some of the most conceptually obvious yet difficult to systematically define and measure. Distributional considerations are also relevant. For example, different theories about how to allocate resources to or within genomic medicine could lead to different policy decisions regarding health services for patients with a particular rare disease.
Most would agree that improved health is objectively good for people regardless of whether they value it because it reduces pain, increases pleasure, and enhances their basic capabilities. Philosophers would also count the degree to which a technology improves or compromises a person’s mental state among its utilities and disutilities. For example, if a genomic test result offers a patient peace of mind or hope or causes symptoms of anxiety or depression, those effects should be included in the ethical calculus. While genomic sequencing generally does not cause psychological harm for the average patient, some individuals may be more likely to experience distress. Psychological benefits have been less studied, and it is unclear how persistent either positive or negative effects might be.21
Finally, the incorporation of desire-fulfillment as a good under the philosophical understanding of utility implies that the use of genomic technology has utility or disutility if using it satisfies the preferences of a patient. Patients or their families may want to use this technology for multiple reasons, including a desire for information, a need to explore all available options for diagnosis and treatment, or to fulfill what they see as the moral requirements of being a good parent.22,23 On the other hand, however, concerns about privacy or the potential for discrimination in insurance, the workplace (or school), or social environments may be perceived as disutilities.24
Philosophy’s broad understanding of the concept of utility therefore incorporates all of the many different potential implications of genomic technology for patients and their families in its analysis of their impact, typically assessed through qualitative methods or psychometric outcome measures administered via surveys. In the CSER consortium, there are several qualitative research efforts guided by philosophical understandings, as well as quantitative instruments to evaluate patient-reported utility (Table 1).7,23, 24, 25, 26 None of these existing instruments, however, measure all relevant domains of utility and disutility and none were developed with as clinically or sociodemographically diverse a population as CSER.
Table 1.
Examples of outcomes in assessments of utility of genomic medicine
Field | Example stakeholders who share orientation | Utility operationalization | Measurement approach | Example CSER measurement |
---|---|---|---|---|
Philosophy | institutional review boards, patients, health technology assessment agencies | balance of benefit and harm | qualitative methods (interviews, focus groups), surveys, psychometric outcome instruments | interviews on patient/parent perceived utility |
Medicine | patients, clinicians, clinical administrators, payers | usefulness (clinical utility, personal utility) | changes in medical management, psychometric outcome instruments | clinician-reported recommended clinical actions, provider perceived utility, PrU, FACToR |
Decision psychology/ health economics | clinical administrators, payers, therapeutic developers | health-related quality of life, health state utility | preference-based instruments or direct elicitation | PedsQL, SF-12 |
CSER, Clinical Sequencing Evidence-Generating Research consortium; PrU, patient-reported utility; FACToR, feelings about genomic testing results; PedsQL, pediatric quality of life inventory; SF-12, 12-item short form health survey.
Utility in medicine
Medicine definition and measurement
Clinicians have traditionally taken a reductive view of outcomes when evaluating clinical interventions, emphasizing the ability to forestall death, reduce length of hospital stays, or improve observable clinical signs and symptoms. However, the field of medicine is not monolithic and includes both physicians and non-physician healthcare workers within the same clinical enterprise. The diversity of views from various perspectives and specialty areas makes it difficult to describe a general medical conceptualization of utility. Moreover, the definition of utility tends to be left implicit; there is no standardized definition of utility that is widely accepted by all clinicians. Rather than debating whether a medical intervention carries utility, clinicians tend to consider whether it is clinically indicated. By this, they mean that an intervention would be expected to reduce or eliminate symptoms, morbidity, mortality, or burden of disease. The implicit definition of utility in medicine, then, is the ability of an intervention to reduce or eliminate one or more of these outcomes. In utility terms, this is the clinical utility of an intervention, defined as the effectiveness to impact medical management or downstream health outcomes.1,27,28
Emphasis on objective clinical features can be to the exclusion of outcomes that might be of more immediate importance to patients, such as the frequency or severity of subjective symptoms, usefulness in life planning or everyday decisions, or even alleviation of guilt. In response to patients and patient advocates, clinicians have recently begun to take a more patient-centered approach. As a result, the definition of utility in medicine also increasingly reflects a broader set of outcomes intended to capture the perspectives of patients.
Greater emphasis on patient experience data is reflected in several recent initiatives. For example, the National Institutes of Health has supported development of patient-reported outcome measure repositories to aid the assessment of patient perspectives across a range of constructs such as pain, fatigue, self-efficacy, and social participation ability.29 Additionally, the Patient-Centered Outcomes Research Institute has advanced research and clinical practice that highlights outcomes of interest to patients.30 Through the Patient-Focused Drug Development initiative, the Food and Drug Administration has also increased emphasis on patient experience data on the basis of the understanding that patients’ perspectives should inform the assessment of benefits and risks of an intervention for their condition.31,32 Thus, if there is a perspective on utility that is distinctive to medicine, it appears to be trending away from a sole clinician-determined evaluation of physical health-related outcomes and moving increasingly toward acknowledgment of the patient’s self-reported experience of their condition as a key outcome.
The element of utility that is both fundamental and distinctive to medicine is its epistemological aspect. Medicine tends to focus on how one would know that an intervention leads to utility and how to justify that claim to others. For at least the last 30 years, evidence-based medicine (EBM) has provided the epistemological framework for how medicine defines claims of utility and generates evidence for its evaluation. In fact, some have argued that the demand for empirical evidence to support claims of utility is itself the value that defines medicine as a domain of inquiry and practice.33,34
Medical perspective on utility in genomic medicine
A focus on clinical utility, narrowly defined, in genomic medicine emphasizes that decisions like whether to perform genetic or genomic testing for a specific clinical indication and whether to analyze genes unrelated to the clinical indication for testing (i.e., secondary findings) should be based on conventional outcomes of interest to physicians: changes in medical management, reduction in morbidity or mortality, and perhaps prognostication.35,36 A purely clinical focus also deemphasizes the practical or informational value to patients and their families that genetic and genomic testing may provide.
To guide evidence generation, there is currently neither a universally accepted definition of clinical utility in genomic medicine nor a standardized measurement approach.27,37,38 To guide review of existing evidence, a working group supported by the National Office of Public Health Genomics at the Centers for Disease Control and Prevention articulated a framework to consider evidence in four domains: analytic validity; clinical validity; clinical utility; and ethical, legal, and social implications. In this “ACCE” framework, clinical utility is conceptualized to include improved clinical outcomes, informed clinical or personal decision-making, and ending the diagnostic odyssey, as well as “outcomes of value to patients” based on changes in clinical care guided by test results.1 While this definition mostly comports with the traditional EBM approach, it also incorporates the notion that outcomes related to the concept of personal utility of genetic and genomic testing should be included in assessments of its use.
Clinicians and clinical researchers in genomic medicine increasingly accept such patient-centered outcomes as relevant, acknowledging that genetic information can also have personal utility, the value of which should be considered alongside clinical outcomes.39 The personal utility view highlights that interventions in this field may be useful even if they are not expected to lead to clinical intervention and improved objective markers of health. These personal outcomes may include alleviation of guilt, pragmatic life planning, or even just the ability of genetic and genomic tests to provide information that is “good to know.”23,24,40,41 There are also calls to broaden the concept of utility beyond the individual patient given the relevance of genomic information for biological relatives and communities with shared ancestry.42 However, while this broader conception of the utility of genetics and genomics is now widespread within the field, clinical outcomes for the patient remain the primary focus, especially given the need to justify implementation of testing to healthcare systems and funders.
The appropriate evidentiary threshold for adoption of new genomic medicine interventions is also debated. At one end of the spectrum, advocates for EBM argue that diagnostic or therapeutic interventions should not be adopted until large randomized controlled trials demonstrate their utility; there is a high evidence threshold for demonstration of an impact on clinically relevant outcomes. However, conventional evidentiary standards for EBM may be troublesome in the context of genomic medicine. In particular, treatments for rare genetic conditions are not amenable to comparative effectiveness research given that the rarity of these conditions limits the sample sizes that can be obtained for such studies. Comparative effectiveness research on precision therapeutics and predispositional genetic testing encounter similar challenges given that the genetic variants underlying these interventions may be uncommon.
Advocates for genomic medicine, therefore, argue that diagnostic information should be considered useful even if the empirical evidence for its clinical use is still limited; there is a low evidence threshold for demonstration of an impact on clinically relevant outcomes. Rather than demonstrating improved health outcomes, some have advanced the concept of actionability as an alternative standard. Actionability emphasizes that genetic results may be considered useful if they create an opportunity to take an action of some sort,43,44 even if it is currently not possible to empirically determine whether that action will bring about a downstream change in patient health.45,46 In the strictest sense, clinical actionability excludes consideration of personal utility to patients.
The CSER consortium is collecting data on objective endpoints, such as whether a patient received a diagnosis and whether any of a set of clinical management changes occurred, as well as clinicians’ and patients’ perceived utility of the test (Table 1).7 Measurement approaches geared toward understanding the perceived utility on the part of the patient or family are aligned with those guided by philosophical understanding of benefits and harms, described above, as well as aspects of personal utility.
Utility in decision psychology and health economics
Decision psychology and health economics definition and measurement
Much of decision psychology and health economics is concerned with the study of how individuals make choices. In economics, as in philosophy, the classical utilitarian definition from Bentham and Mill is commonly used to define utility as happiness or satisfaction. According to Mill, when perfectly rational agents with full information are given a choice of outcomes that each have some probability of occurring, they will make the choice has the highest chance of leading to a preferred outcome, maximizing the expected value of the utility. Expected utility theory, formulated by Bernoulli, can be applied as a normative theory to explain how people should make decisions under conditions of uncertainty.
Because humans are not perfectly rational agents, Samuelson’s revealed preference theory, in which utility is inferred through observation of choices,47 and von Neumann and Morgenstern’s utility theorem, which described the conditions under which the expected utility hypothesis holds,48,49 have helped illuminate how individuals make choices in the real world. According to modernized conceptions, utility can be defined as a measure of preference or choice that incorporates an individual’s risk attitude.50
As the basis for judging resource allocation with applications to health policy, there are two major schools of thought within normative (welfare) economics: welfarist and extra-welfarist,51 representing two different views on the role of health versus other goods in society.52 Welfarists believe that the output of health care should be judged according to the extent to which it contributes to overall welfare. Utility is calculated by adding up individual utilities via a classical utilitarian approach and has the goal of maximizing the sum of all individual utilities subject to a budget constraint.
In contrast, extra-welfarists consider outcomes in addition to utility, such as Sen’s capabilities conceptualization of well-being53 or health. The overall goal is to maximize a chosen social objective, such as health, within a fixed budget constraint. Thus, the output of health care is measured according to its contribution to health. Extra-welfarists try to maximize health by choosing interventions that are cost-effective at a given threshold chosen by policymakers. Given the greater relevance to translational research and cost-effectiveness analysis, we focus here on how extra-welfarists employ expected utility theory to derive the valuation of health states for use in economic evaluation of health care interventions.
In health economic evaluation, utility most often refers to health state utility, which is the value placed on health outcomes by actual or potential patients.54 The extra-welfarist approach to economic evaluation is cost-utility analysis (CUA), in which quality-adjusted life years (QALYs) are used as a measure of health. While CUA gets its name from the incorporation of utility ratings of health states that are used to calculate QALYs, QALYs themselves are a measure of health in specified domains, rather than a measure of utility, that theoretically enable systematic comparison of levels of health across all interventions and disease areas.52 QALYs are calculated by weighting the time spent in each health state with the utility/preference weight of that health state, which are elicited through preference assessment methods.54 Preferences for health states are measured in numeric terms (cardinal utility) rather than mere preference ordering (ordinal utility) with the formulation of expected utility theory developed by von Neumann and Morgenstern.48
To understand predictive applications of expected utility theory, decision psychologists use psychological methods to build knowledge of the cognitive processes underlying utility formation and to measure subjective experience, including judgments and decision making under uncertainty.55 Psychologists have described the ways in which people’s choices do not conform to theories of decision-making that emphasize formal rationality.50 For example, Simon’s concept of bounded rationality recognizes that people will “satisfice,” or stop searching for the utility-maximizing option when they find an option that is perceived to be good enough, given the tradeoff between accuracy of information on all possible alternatives and the effort required to obtain it.56
Decision psychology has also contributed to the conceptualization of forms of utility in relation to the process of decision-making.57, 58, 59 For instance, Kahneman and colleagues argued for the importance of distinguishing between Bentham’s concept of utility, termed experienced utility, and decision utility, in which utility is inferred from observable choices rather than the hedonic experience of choices.60 They described concepts of utility along a process timeline: instant utility, the immediate assessment of hedonic and affective experience; remembered utility, retrospective assessment of the hedonic and affective experiences associated with past outcomes; and total utility, constructed from assessment of instant utility over time.60
Decision psychological and health economic perspective on utility in genomic medicine
Health economists evaluating genomic sequencing applications have been chiefly concerned with modeling health gains that arise as a result of genomic sequencing, ideally over patients’ lifetimes. Decision analytic modeling can be used to bridge the gap between diagnostic yield, a commonly reported intermediate clinical outcome measure in genetics and genomics,61,62 and QALYs, a definitive measure of health outcome, through evaluation of the effects on quality of life associated with health state changes that result from receipt of clinical care informed by genetic findings.54 Decision analytic models allow changes in medical management and clinical event rates, which are sourced from existing literature or registries, to be incorporated into health economic evaluations, even in the absence of lifetime follow-up data on the patients who have themselves been sequenced.44 Examples of such models include BRCA1/BRCA2 testing for hereditary breast and ovarian cancer63 and return of secondary findings.64 However, modeling requires data sources appropriate for informing long-term clinical trajectories and parameter values, and these data sources are scarce for many of the disorders diagnosable via genomic sequencing. A further complication when modeling multiplex technologies like genomic sequencing is that medical management changes and impacts on patient health are highly condition specific and dependent upon assumptions about clinician and patient behavior and variant penetrance.
Personal utility aspects of genomic medicine are also relevant to economic evaluation, and US methodological guidelines call for both health and non-health effects to be included.65 This raises the question of whether off-the-shelf utility measures used to calculate QALYs are capable of capturing non-health, personal utility impacts. There is emerging interest in the use of stated preference techniques to quantify and assign value to broad, non-health effects of testing that are not captured in health-related quality of life instruments.66,67
In light of the need to collect health-related quality of life data via standardized measures for cost-effectiveness and cost-utility analysis, CSER surveys include the Euro-QoL VAS68 and PedsQL69 for pediatric patients and the SF-12 for adults (Table 1).70 Cognizant of the debate about whether these quality of life measures capture effects on the scale of prognostication and management changes, CSER is also collecting data on attributable clinical actions and measuring clinicians’ perceived utility of genomic sequencing results to describe the impact on clinical decision-making included in original definitions of clinical utility and decision utility.1,60 Additionally, a new patient-reported outcome measure designed to assess a broad range of benefits and harms from patients’ perspectives is in development. To capture data on resource use that is useful for decision analytic modeling, time and motion studies to measure the resources required to deliver genomic services, including clinician time, are being performed. A set of frameworks for conducting economic evaluations of genomic sequencing in various clinical contexts are also in development.
Utility in genomics translational research
Translational research investigates the application of basic science discoveries in patient care.71 Traditionally described as linear stages that range from basic biomedical research (T0) to population-level outcomes research (T4), translational research spans multiple fields and methodologies.72 Through the analysis of clinical outcomes, comparative-effectiveness and cost-effectiveness, and policy change, investigations are designed to inform evidence-based clinical guideline development and establishment of efficient service delivery models that ultimately improve population health. For complex interventions such as genomic sequencing, however, the stages of translational research are often non-linear. While research questions from multiple stages may overlap, enumeration and measurement of relevant benefits and harms are nonetheless crucial aspects of research design. The need to produce various forms of evidence on benefit and harm to fulfill diverse stakeholders’ informational needs underpins the importance of multidisciplinary approaches to translational research (Table 1). Evidence generated through various approaches may be relevant to some stakeholders but not to others depending upon the intended use.
Not only do notions of utility and its associated measurement constructs differ among researchers, they differ among genomic medicine stakeholders as well.37 Robust evidence of utility, assessed from several disciplinary perspectives, is needed to achieve multiple goals: inform decisions of clinicians and patients about whether to pursue genomic sequencing, address patient needs and concerns, and inform decisions about whether genomic sequencing will be offered or reimbursed. In genomic medicine, conversations regarding the varied and sometimes fragmented orientations toward conceptualization and measurement of utility mirror those regarding value in many ways. Utility itself can be framed as a measure of value.27 Yet, stakeholders embrace different definitions of the value of genomic medicine73 and attach varying levels of importance to several outcomes.74 Awareness of the variety of understandings of both utility and value, as well as how perspectives might be integrated, is critical to advancing the discussion of evidentiary needs in various contexts.
Moreover, there is a crucial need to understand optimal ways to conduct genomics research that effectively engages and incorporates sociopolitically diverse and medically underserved communities. This work, which is central to the aim of the CSER consortium, involves understand differing priorities, values, and social contexts in populations that have historically been underrepresented in genomic research and ensuring that evidence reflects diverse populations and the subgroups within them. Given that many of the instruments related to utility measurement were developed in populations (e.g., healthy adults) that do not reflect the sociopolitical makeup of the consortium, there are validation efforts underway to assess whether and how instruments might perform differently in the CSER projects. Additionally, many projects have used qualitative studies to provide greater depth of understanding of patient experiences.
Utility in genomic sequencing translational research: A path forward
There is mounting pressure to generate evidence of the utility of genomic sequencing, yet what it means to establish utility remains somewhat unclear. To date, as highlighted at the end of each disciplinary section above, distinct knowledge traditions have shaped early evaluation efforts in translational genomics research (Table 1). By bringing together diverse perspectives in a meaningful way, clearer articulation of the evidence required to establish utility would allow for the design of assessments that can be justified to and consumed by a wider range of stakeholders and applied in a broader range of contexts. As research on clinical integration of genomic medicine continues to mature, there are several ways in which translational research can aim to increase precision related to, for example, the operational definition of utility, level of analysis, and measurement methods (Figure 1).
Figure 1.
How understandings of utility can inform each other for the design of genomic translational research
At the outset of a project, both research funders and researchers should clearly communicate how the project will conceptualize utility. If the operational definition includes evaluations from multiple perspectives, they should be clearly distinguished through the use of precise language, including modifiers such as economic utilities or personal utilities. In addition to the definition of utility itself, it is important to be explicit about the level of analysis that is of interest to researchers and stakeholders. For example, health economic evaluation is conducted and interpreted at the population level to assess which interventions lead to the highest aggregate health for populations, considering their associated costs. In contrast to the economic perspective on the average patient, research from philosophical or clinical perspectives might place greater emphasis on the impact of genomics on individual patients and their families. Additionally, the identification of appropriate endpoints and measurement tools can present challenges if funders, researchers, and stakeholders do not have a shared understanding of the measurement purpose.
When carefully addressed, varying understandings can be catalysts for growth in the research enterprise. Interdisciplinary, team science approaches to translational research are a strength from both a study design and result interpretation perspective; they can strengthen creativity and innovation and enhance interoperability of measurement models.8,75 When met with knowledge of concepts used in colleagues’ primary field of training, different perspectives facilitate the design of investigations that thoughtfully and holistically explore the implications of clinical translation.
When equipped with broad understanding of key concepts relevant to important outcomes of the study, research teams can work together in a transcendent way, at best, and with mutual understanding, at least, that turns diversity of disciplinary backgrounds from a liability into an asset. As the exploration of the term “utility” reveals, the underlying sentiment across the three fields of philosophy, medicine, and social science is more alike than different. Through creative engagement with colleagues and input from stakeholders, all understandings may be meaningfully incorporated into future analytical frameworks for translational genomics research.
Acknowledgments
The Clinical Sequencing Evidence-Generating Research (CSER) consortium is funded by the National Human Genome Research Institute (NHGRI) with co-funding from the National Institute on Minority Health and Health Disparities (NIMHD) and The National Cancer Institute (NCI), supported by U01HG006487 (UNC), U01HG007292 (KPNW), U01HG006485 (Baylor), U01HG009599 (UCSF), U01HG007301 (HudsonAlpha), and U24HG007307 (Coordinating Center). The contents of this paper are solely the responsibility of the authors and do not necessarily represent the official views of the NIH. More information about CSER can be found on the CSER website (see web resources).
Declaration of interests
A.L.M. is a member of The American Journal of Human Genetics editorial board. The other authors declare no competing interests.
Web resources
References
- 1.Teutsch S.M., Bradley L.A., Palomaki G.E., Haddow J.E., Piper M., Calonge N., Dotson W.D., Douglas M.P., Berg A.O., EGAPP Working Group The Evaluation of Genomic Applications in Practice and Prevention (EGAPP) Initiative: methods of the EGAPP Working Group. Genet. Med. 2009;11:3–14. doi: 10.1097/GIM.0b013e318184137c. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Horowitz C.R., Orlando L.A., Slavotinek A.M., Peterson J., Angelo F., Biesecker B., Bonham V.L., Cameron L.D., Fullerton S.M., Gelb B.D., et al. The Genomic Medicine Integrative Research Framework: A Conceptual Framework for Conducting Genomic Medicine Research. Am. J. Hum. Genet. 2019;104:1088–1096. doi: 10.1016/j.ajhg.2019.04.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Stark Z., Dolman L., Manolio T.A., Ozenberger B., Hill S.L., Caulfied M.J., Levy Y., Glazer D., Wilson J., Lawler M., et al. Integrating genomics into healthcare: a global responsibility. Am. J. Hum. Genet. 2019;104:13–20. doi: 10.1016/j.ajhg.2018.11.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Manolio T.A., Abramowicz M., Al-Mulla F., Anderson W., Balling R., Berger A.C., Bleyl S., Chakravarti A., Chantratita W., Chisholm R.L., et al. Global implementation of genomic medicine: we are not alone. Sci. Transl. Med. 2015;7:290ps213. doi: 10.1126/scitranslmed.aab0194. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Amendola L.M., Berg J.S., Horowitz C.R., Angelo F., Bensen J.T., Biesecker B.B., Biesecker L.G., Cooper G.M., East K., Filipski K., et al. The Clinical Sequencing Evidence-Generating Research Consortium: Integrating Genomic Sequencing in Diverse and Medically Underserved Populations. Am. J. Hum. Genet. 2018;103:319–327. doi: 10.1016/j.ajhg.2018.08.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Wolf S.M., Amendola L.M., Berg J.S., Chung W.K., Clayton E.W., Green R.C., Harris-Wai J., Henderson G.E., Jarvik G.P., Koenig B.A., et al. Navigating the research-clinical interface in genomic medicine: analysis from the CSER Consortium. Genet. Med. 2018;20:545–553. doi: 10.1038/gim.2017.137. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Goddard K.A.B., Angelo F.A.N., Ackerman S.L., Berg J.S., Biesecker B.B., Danila M.I., East K.M., Hindorff L.A., Horowitz C.R., Hunter J.E., et al. Lessons learned about harmonizing survey measures for the CSER consortium. J. Clin. Transl. Sci. 2020;4:537–546. doi: 10.1017/cts.2020.41. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Disis M.L., Slattery J.T. The Road We Must Take: Multidisciplinary Team Science. Sci. Transl. Med. 2010;2:22cm29. doi: 10.1126/scitranslmed.3000421. [DOI] [PubMed] [Google Scholar]
- 9.Bentham J. T. Payne; 1789. An introduction to the principles of morals and legislation: printed in the year 1780, and now first published. [Google Scholar]
- 10.Mill J.S. 1863. Utilitarianism, Liberty, and Representative Government. 1859; pp. 7–9. [Google Scholar]
- 11.Moore G.E., Baldwin T. Cambridge University Press; 1993. Principia ethica. [Google Scholar]
- 12.Sen A. Plural utility. Proceedings of the Aristotelian Society. 1980;81 [Google Scholar]
- 13.Griffin J. Clarendon Press; 1986. Well-being: Its meaning, measurement and moral importance. [Google Scholar]
- 14.Kagan S. An Introduction to Ill-Being. Oxford studies in normative ethics. 2015;4:261–288. [Google Scholar]
- 15.Mathison E. Department of Philosophy, University of Toronto; 2018. Asymmetries and Ill-Being. Doctoral Dissertation. [Google Scholar]
- 16.Brandt R.B. Morality and the Language of Conduct; 1963. Toward a credible form of utilitarianism; pp. 107–143. [Google Scholar]
- 17.Harsanyi J.C. Rule utilitarianism and decision theory. Erkenntnis. 1977;11:25–53. [Google Scholar]
- 18.Popper K.R. Princeton University Press; 2020. The open society and its enemies, Volume 119. [Google Scholar]
- 19.Temkin L.S. Oxford University Press; 1993. Inequality. [Google Scholar]
- 20.Sidgwick H. Fourth Edition. MacMillian and Co; London: 1890. The Methods of Ethics. [Google Scholar]
- 21.Robinson J.O., Wynn J., Biesecker B., Biesecker L.G., Bernhardt B., Brothers K.B., Chung W.K., Christensen K.D., Green R.C., McGuire A.L., et al. Psychological outcomes related to exome and genome sequencing result disclosure: a meta-analysis of seven Clinical Sequencing Exploratory Research (CSER) Consortium studies. Genet. Med. 2019;21:2781–2790. doi: 10.1038/s41436-019-0565-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Malek J., Pereira S., Robinson J.O., Gutierrez A.M., Slashinski M.J., Parsons D.W., Plon S.E., McGuire A.L. Responsibility, culpability, and parental views on genomic testing for seriously ill children. Genet. Med. 2019;21:2791–2797. doi: 10.1038/s41436-019-0570-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Malek J., Slashinski M.J., Robinson J.O., Gutierrez A.M., Parsons D.W., Plon S.E., McCullough L.B., McGuire A.L. Parental perspectives on whole-exome sequencing in pediatric cancer: a typology of perceived utility. JCO Precis. Oncol. 2017;1:1–10. doi: 10.1200/PO.17.00010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Kohler J.N., Turbitt E., Lewis K.L., Wilfond B.S., Jamal L., Peay H.L., Biesecker L.G., Biesecker B.B. Defining personal utility in genomics: A Delphi study. Clin. Genet. 2017;92:290–297. doi: 10.1111/cge.12998. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Lupo P.J., Robinson J.O., Diamond P.M., Jamal L., Danysh H.E., Blumenthal-Barby J., Lehmann L.S., Vassy J.L., Christensen K.D., Green R.C., McGuire A.L., MedSeq Project team Patients’ perceived utility of whole-genome sequencing for their healthcare: findings from the MedSeq project. Per. Med. 2016;13:13–20. doi: 10.2217/pme.15.45. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Li M., Bennette C.S., Amendola L.M., Ragan Hart M., Heagerty P., Comstock B., Tarczy-Hornoch P., Fullerton S.M., Regier D.A., Burke W., et al. The Feelings About genomiC Testing Results (FACToR) Questionnaire: Development and Preliminary Validation. J. Genet. Couns. 2019;28:477–490. doi: 10.1007/s10897-018-0286-9. [DOI] [PubMed] [Google Scholar]
- 27.Burke W., Laberge A.M., Press N. Debating clinical utility. Public Health Genomics. 2010;13:215–223. doi: 10.1159/000279623. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Grosse S.D., Khoury M.J. What is the clinical utility of genetic testing? Genet. Med. 2006;8:448–450. doi: 10.1097/01.gim.0000227935.26763.c6. [DOI] [PubMed] [Google Scholar]
- 29.Snyder C., Brundage M., Rivera Y.M., Wu A.W. A PRO-cision Medicine Methods Toolkit to Address the Challenges of Personalizing Cancer Care Using Patient-Reported Outcomes: Introduction to the Supplement. Med. Care. 2019;57(Suppl 5 Suppl 1):S1–S7. doi: 10.1097/MLR.0000000000001089. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Frank L., Basch E., Selby J.V., Patient-Centered Outcomes Research Institute The PCORI perspective on patient-centered outcomes research. JAMA. 2014;312:1513–1514. doi: 10.1001/jama.2014.11100. [DOI] [PubMed] [Google Scholar]
- 31.Hunter N.L., O’Callaghan K.M., Califf R.M. Engaging Patients Across the Spectrum of Medical Product Development: View From the US Food and Drug Administration. JAMA. 2015;314:2499–2500. doi: 10.1001/jama.2015.15818. [DOI] [PubMed] [Google Scholar]
- 32.US FDA . 2017. Plan for Issuance of Patient-Focused Drug Development Guidance Under the 21st Century Cures Act, Title III, Section 3002. May 2017.https://www.fda.gov/files/about%20fda/published/Plan-for-Issuance-of-Patient%E2%80%90Focused-Drug-Development-Guidance.pdf [Google Scholar]
- 33.Polychronis A., Miles A., Bentley P. Evidence-based medicine: reference? Dogma? Neologism? New orthodoxy? J. Eval. Clin. Pract. 1996;2:1–3. doi: 10.1111/j.1365-2753.1996.tb00022.x. [DOI] [PubMed] [Google Scholar]
- 34.Angell M., Kassirer J.P. Alternative medicine—the risks of untested and unregulated remedies. N. Engl. J. Med. 1998;339:839–841. doi: 10.1056/NEJM199809173391210. [DOI] [PubMed] [Google Scholar]
- 35.Kalia S.S., Adelman K., Bale S.J., Chung W.K., Eng C., Evans J.P., Herman G.E., Hufnagel S.B., Klein T.E., Korf B.R., et al. Recommendations for reporting of secondary findings in clinical exome and genome sequencing, 2016 update (ACMG SF v2.0): a policy statement of the American College of Medical Genetics and Genomics. Genet. Med. 2017;19:249–255. doi: 10.1038/gim.2016.190. [DOI] [PubMed] [Google Scholar]
- 36.Ravitsky V., Wilfond B.S. Disclosing individual genetic results to research participants. Am. J. Bioeth. 2006;6:8–17. doi: 10.1080/15265160600934772. [DOI] [PubMed] [Google Scholar]
- 37.Hayeems R.Z., Dimmock D., Bick D., Belmont J.W., Green R.C., Lanpher B., Jobanputra V., Mendoza R., Kulkarni S., Grove M.E., et al. Medical Genome Initiative Clinical utility of genomic sequencing: a measurement toolkit. NPJ Genom. Med. 2020;5:56. doi: 10.1038/s41525-020-00164-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Stevens Smith H., Russell H.V., Lee B.H., Morain S.R., and the Value of Exome Sequencing Delphi Panel Using the Delphi method to identify clinicians’ perceived importance of pediatric exome sequencing results. Genet. Med. 2020;22:69–76. doi: 10.1038/s41436-019-0601-3. [DOI] [PubMed] [Google Scholar]
- 39.Grosse S.D., McBride C.M., Evans J.P., Khoury M.J. Personal utility and genomic information: look before you leap. Genet. Med. 2009;11:575–576. doi: 10.1097/GIM.0b013e3181af0a80. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Kohler J.N., Turbitt E., Biesecker B.B. Personal utility in genomic testing: a systematic literature review. Eur. J. Hum. Genet. 2017;25:662–668. doi: 10.1038/ejhg.2017.10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Torkamani A., Wineinger N.E., Topol E.J. The personal and clinical utility of polygenic risk scores. Nat. Rev. Genet. 2018;19:581–590. doi: 10.1038/s41576-018-0018-x. [DOI] [PubMed] [Google Scholar]
- 42.ACMG Board of Directors Clinical utility of genetic and genomic services: a position statement of the American College of Medical Genetics and Genomics. Genet. Med. 2015;17:505–507. doi: 10.1038/gim.2015.41. [DOI] [PubMed] [Google Scholar]
- 43.Burke W., Pinsky L.E., Press N.A. Categorizing genetic tests to identify their ethical, legal, and social implications. Am. J. Med. Genet. 2001;106:233–240. doi: 10.1002/ajmg.10011. [DOI] [PubMed] [Google Scholar]
- 44.Veenstra D.L., Roth J.A., Garrison L.P., Jr., Ramsey S.D., Burke W. A formal risk-benefit framework for genomic tests: facilitating the appropriate translation of genomics into clinical practice. Genet. Med. 2010;12:686–693. doi: 10.1097/GIM.0b013e3181eff533. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Jarvik G.P., Amendola L.M., Berg J.S., Brothers K., Clayton E.W., Chung W., Evans B.J., Evans J.P., Fullerton S.M., Gallego C.J., et al. eMERGE Act-ROR Committee and CERC Committee. CSER Act-ROR Working Group Return of genomic results to research participants: the floor, the ceiling, and the choices in between. Am. J. Hum. Genet. 2014;94:818–826. doi: 10.1016/j.ajhg.2014.04.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Hunter J.E., Irving S.A., Biesecker L.G., Buchanan A., Jensen B., Lee K., Martin C.L., Milko L., Muessig K., Niehaus A.D., et al. A standardized, evidence-based protocol to assess clinical actionability of genetic disorders associated with genomic variation. Genet. Med. 2016;18:1258–1268. doi: 10.1038/gim.2016.40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Samuelson P.A. A Note on the Pure Theory of Consumer’s Behaviour. Economica. 1938;5:61–71. [Google Scholar]
- 48.Morgenstern O., Von Neumann J. Princeton University Press; 1953. Theory of Games and Economic Behavior. [Google Scholar]
- 49.Wheeler G. 2020. Bounded Rationality.https://plato.stanford.edu/archives/fall2020/entries/bounded-rationality/ [Google Scholar]
- 50.Briggs R. 2019. Normative Theories of Rational Choice: Expected Utility.https://plato.stanford.edu/archives/fall2019/entries/rationality-normative-utility/ [Google Scholar]
- 51.Boadway R.W., Bruce N. B. Blackwell New York; 1984. Welfare economics. [Google Scholar]
- 52.Brouwer W.B., Culyer A.J., van Exel N.J.A., Rutten F.F. Welfarism vs. extra-welfarism. J. Health Econ. 2008;27:325–338. doi: 10.1016/j.jhealeco.2007.07.003. [DOI] [PubMed] [Google Scholar]
- 53.Al-Janabi H., Flynn T.N., Coast J. Development of a self-report measure of capability wellbeing for adults: the ICECAP-A. Qual. Life Res. 2012;21:167–176. doi: 10.1007/s11136-011-9927-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Drummond M.F., Sculpher M.J., Claxton K., Stoddart G.L., Torrance G.W. Oxford university press; 2015. Methods for the economic evaluation of health care programmes. [Google Scholar]
- 55.Einhorn H.J., Hogarth R.M. Behavioral decision theory: Processes of judgement and choice. Annu. Rev. Psychol. 1981;32:53–88. [Google Scholar]
- 56.Simon H.A. A behavioral model of rational choice. Q. J. Econ. 1955;69:99–118. [Google Scholar]
- 57.Edwards W. The theory of decision making. Psychol. Bull. 1954;51:380–417. doi: 10.1037/h0053870. [DOI] [PubMed] [Google Scholar]
- 58.Peterson C.R., Beach L.R. Man as an intuitive statistician. Psychol. Bull. 1967;68:29–46. doi: 10.1037/h0024722. [DOI] [PubMed] [Google Scholar]
- 59.Tversky A., Kahneman D. Judgment under uncertainty: Heuristics and biases. Science. 1974;185:1124–1131. doi: 10.1126/science.185.4157.1124. [DOI] [PubMed] [Google Scholar]
- 60.Kahneman D., Wakker P.P., Sarin R. Back to Bentham? Explorations of experienced utility. Q. J. Econ. 1997;112:375–406. [Google Scholar]
- 61.Payne K., Gavan S.P., Wright S.J., Thompson A.J. Cost-effectiveness analyses of genetic and genomic diagnostic tests. Nat. Rev. Genet. 2018;19:235–246. doi: 10.1038/nrg.2017.108. [DOI] [PubMed] [Google Scholar]
- 62.Smith H.S., Swint J.M., Lalani S.R., Yamal J.M., de Oliveira Otto M.C., Castellanos S., Taylor A., Lee B.H., Russell H.V. Clinical Application of Genome and Exome Sequencing as a Diagnostic Tool for Pediatric Patients: a Scoping Review of the Literature. Genet. Med. 2019;21:3–16. doi: 10.1038/s41436-018-0024-6. [DOI] [PubMed] [Google Scholar]
- 63.Guzauskas G.F., Garbett S., Zhou Z., Spencer S.J., Smith H.S., Hao J., Hassen D., Snyder S.R., Graves J.A., Peterson J.F., et al. Cost-effectiveness of Population-Wide Genomic Screening for Hereditary Breast and Ovarian Cancer in the United States. JAMA Netw. Open. 2020;3:e2022874. doi: 10.1001/jamanetworkopen.2020.22874. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 64.Bennette C.S., Gallego C.J., Burke W., Jarvik G.P., Veenstra D.L. The cost-effectiveness of returning incidental findings from next-generation genomic sequencing. Genet. Med. 2015;17:587–595. doi: 10.1038/gim.2014.156. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65.Neumann P.J., Sanders G.D., Russell L.B., Siegel J.E., Ganiats T.G. Oxford University Press; New York: 2016. Cost-Effectiveness in Health and Medicine. [Google Scholar]
- 66.Regier D.A., Weymann D., Buchanan J., Marshall D.A., Wordsworth S. Valuation of Health and Nonhealth Outcomes from Next-Generation Sequencing: Approaches, Challenges, and Solutions. Value Health. 2018;21:1043–1047. doi: 10.1016/j.jval.2018.06.010. [DOI] [PubMed] [Google Scholar]
- 67.Grosse S.D., Wordsworth S., Payne K. Economic methods for valuing the outcomes of genetic testing: beyond cost-effectiveness analysis. Genet. Med. 2008;10:648–654. doi: 10.1097/gim.0b013e3181837217. [DOI] [PubMed] [Google Scholar]
- 68.Wille N., Badia X., Bonsel G., Burström K., Cavrini G., Devlin N., Egmar A.C., Greiner W., Gusi N., Herdman M., et al. Development of the EQ-5D-Y: a child-friendly version of the EQ-5D. Qual. Life Res. 2010;19:875–886. doi: 10.1007/s11136-010-9648-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 69.Varni J.W., Seid M., Rode C.A. The PedsQL: measurement model for the pediatric quality of life inventory. Med. Care. 1999;37:126–139. doi: 10.1097/00005650-199902000-00003. [DOI] [PubMed] [Google Scholar]
- 70.Ware J., Jr., Kosinski M., Keller S.D.A. A 12-Item Short-Form Health Survey: construction of scales and preliminary tests of reliability and validity. Med. Care. 1996;34:220–233. doi: 10.1097/00005650-199603000-00003. [DOI] [PubMed] [Google Scholar]
- 71.Rubio D.M., Schoenbaum E.E., Lee L.S., Schteingart D.E., Marantz P.R., Anderson K.E., Platt L.D., Baez A., Esposito K. Defining translational research: implications for training. Acad. Med. 2010;85:470–475. doi: 10.1097/ACM.0b013e3181ccd618. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 72.Khoury M.J., Gwinn M., Yoon P.W., Dowling N., Moore C.A., Bradley L. The continuum of translation research in genomic medicine: how can we accelerate the appropriate integration of human genome discoveries into health care and disease prevention? Genet. Med. 2007;9:665–674. doi: 10.1097/GIM.0b013e31815699d0. [DOI] [PubMed] [Google Scholar]
- 73.Bush W.S., Cooke Bailey J.N., Beno M.F., Crawford D.C. Bridging the Gaps in Personalized Medicine Value Assessment: A Review of the Need for Outcome Metrics across Stakeholders and Scientific Disciplines. Public Health Genomics. 2019;22:16–24. doi: 10.1159/000501974. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 74.Scheuner M.T., Russell M.M., Chanfreau-Coffinier C., Peredo J., Yano E.M., Hamilton A.B., Lerner B., Provenzale D., Knight S.J., Voils C.I. Stakeholders’ views on the value of outcomes from clinical genetic and genomic interventions. Genet. Med. 2019;21:1371–1380. doi: 10.1038/s41436-018-0344-6. [DOI] [PubMed] [Google Scholar]
- 75.Lungeanu A., Contractor N.S. The effects of diversity and network ties on innovations: The emergence of a new scientific field. Am. Behav. Sci. 2015;59:548–564. doi: 10.1177/0002764214556804. [DOI] [PMC free article] [PubMed] [Google Scholar]