A FRAMEWORK FOR PRIORITIZING RESEARCH INVESTMENTS IN PRECISION MEDICINE

Abstract

Introduction

The adoption of precision medicine (PM) has been limited in practice to date, and yet its promise has attracted research investments. Developing foundational economic approaches for directing proper use of PM and stimulating growth in this area from multiple perspectives is thus quite timely.

Methods

Building on our previously developed Expected Value of Individualized Care (EVIC) framework, we conceptualize new decision-relevant metrics to better understand and forecast the expected value of PM. Several aspects of behavior at the patient, physician and the payer level are considered that can inform the rate and manner in which PM innovations diffuse throughout the relevant population. We illustrate this framework and the methods using a retrospective evaluation of the use of OncotypeDx genomic test among breast cancer patients.

Results

The enriched metrics can help inform many facets of PM decision making, such as evaluating alternative reimbursement levels for PM tests, implementation and education programs for physicians and patients, and decisions around research investments by manufacturers and public entities. We replicated prior published results on evaluation of OncotypeDx among breast cancer patients, but also illustrated that those results are based on assumptions that are often not met in practice. Instead, we show how incorporating more practical aspects of behavior around PM could lead to drastically different estimates of value.

Conclusion

We believe that the framework and the methods presented can provide decision makers with more decision-relevant tool to explore the value of PM. There is a growing recognition that data on adoption is important to decision makers. More research is needed to develop prediction models for potential diffusion of PM technologies.

INTRODUCTION

Individualizing care for patients is important for improving the nation’s health in a cost effective manner.^{1,2,3,4,5,6,7} Individualization takes many forms, including predictions using of clinical characteristics and patient preferences, but genomics-based individualized care (precision medicine, PM) offers the highest potential and is in the midst of a paradigm shift. Next-generation sequencing technology can sequence the whole exome, the 30 million protein coding bases in the genome, in clinical care for about $1,000.^8,9,10 Whole genome sequencing is just around the corner.^11,12 The Precision Medicine Initiative, recently announced by president Obama indicates that the development of genomic markers and medicine are in fast track.^13,14 Patients will be confronted with the possibility of receiving multiple clinically actionable and non-actionable genomic results that have a variety of clinical and personal implications.^15,16

Despite this promise,¹⁷ the adoption of genomic markers has been limited in practice to date.^18,19 Healthcare stakeholders – from technology developers to policymakers to providers and patients – are thus confronted with this dichotomy: despite the tremendous advances in genomics, it is not clear how to best pursue the goal of PM within our existing healthcare system. A failure or delay in capitalizing on the technologic and scientific advances in genomics would represent a tremendous opportunity loss.²⁰ Yet the inappropriate clinical implementation of genomics could lead to significant increases in healthcare costs with no concomitant improvement in patients’ health or quality of life.²¹ Developing foundational economic approaches for directing proper use of PM and stimulating growth in this area is thus quite timely.

In this paper, building on our previously developed Expected Value of Individualized Care (EVIC) conceptual framework,²² we develop a conceptual framework in order to better understand the value of PM and optimal research investment opportunities in PM. We enrich the theory underlying EVIC and develop parallel methods to convert it into a positive tool for forecasting the expected value of precision medicine by incorporating several aspects of behavior at the patient, physician and the payer levels, which culminate in the way in which PM innovations diffuse through the relevant population and over time. Our methods highlight that the traditional approach (e.g. cost-effectiveness) to looking at the value of genomic test, even from an extra-welfarist social insurer’s perspective, may be misleading if the subsequent influence of the social insurer’s decisions on individual-level decision making is not accounted for. We illustrate the use of these methods and the implications for coverage by an insurer and research investments by a manufacturer using an empirical case study of using a genomic test (OncotypeDx) to choose the use of chemotherapy in breast cancer patients. Discussion follows including recommendation of future research that can help estimate the parameters in the new EVIC metrics.

EXPECTED VALUE OF INDIVIDUALIZED CARE (EVIC)

The concept of the Expected Value of Individualized Care (EVIC) was previously developed by Basu and Meltzer (2007).²² Inherently, it compares two approaches to patient management: all patients receive the same treatment (paternalistic model) vs. each patient receives the treatment that leads to optimal outcomes for individual patients (individualized model). Since prescribing a perfect individualizing mechanism or test could also be seen as “paternalistic”, we drop this distinction and classify the two approaches to patient management as the non-use (homogeneous care) versus use (individualized care) of an individualizing mechanism or PM. The assumptions that are inherent to both the homogeneous care model and the individualized care model are that:

1. From a clinician’s point of view, observably identical patients, denoted by i, differ in the incremental net benefits (benefits − harms), B_i, from a new treatment compared to a standard one. This variation is driven by a vector of patient attributes denoted by θ, which are not readily observed by a clinician, through a production functions B_j,i = B_j(θ_i), where j (j =1,2) denotes the two treatment options, with j =1 denoting the new treatment

2. The physician makes the treatment decisions by maximizing the expected net health benefits (NHB), (NHB=Net Benefits − Costs/λ) from a patient perspective in the absence of a PM test. (Costs/λ), denoted by C, represents the benefit-equivalent of costs, with λ being the social willingness to pay for improvement in the scale of B. Physicians do not know about individual willingness to pay.

3. A physician has no knowledge of θ_i, the individual levels of patient attributes, although the physician knows about the population distribution of these attributes p(θ);

4. Patients always accept physician’s recommendations on treatments.

5. Patients bear the full costs of treatments; i.e. treatments are not covered by insurance.

6. A perfect (100% specificity, 100% sensitivity) PM test is available and its properties are known with absolute certainty; and

7. The PM test, when available, is always used by the physician to prescribe treatment.

Under Assumption 1–7, the expected value of individualized care (EVIC) is then given by (see Basu and Meltzer (2007)²² and Appendix for details):

$$\mathit{E}\mathit{V}\mathit{I}\mathit{C}=\mathrm{V}(\text{Individualized Care})-\mathrm{V}(\text{Homogeneous Care})=\int \{{\text{max}}_{\mathrm{j}}\mathrm{N}\mathrm{H}\mathrm{B}(\theta )\}\mathrm{p}(\theta )\mathrm{d}\theta -{\text{max}}_{\mathrm{j}}\int \mathrm{N}\mathrm{H}\mathrm{B}(\theta )\mathrm{p}(\theta )\mathrm{d}\theta ,$$ (Eq. 1)

where NHB are evaluated at the social threshold for willingness to pay for B. EVIC represents, from a societal perspective, a normative upper bound on the value of research for developing a perfect PM test that helps to identify individualized information about each patient to the physician so that she can choose the best treatment for each patient. Naturally, EVIC, as expressed in (1), is greater than zero. EVIC from a societal perspective, accounting for the cost of the PM test, would be given as:

$$\mathit{E}\mathit{V}\mathit{I}{\mathit{C}}_{\mathit{\text{Societal}}}=EVIC-{\mathrm{C}}_{\mathrm{P}\mathrm{M}},$$ (Eq. 2)

where C_PM is the price of the PM test.

To make these calculations more decision-relevant, we evaluate the expected value of individualized care from a social insurer’s point of view, who may not share the same perspective an all-encompassing social planner, but rather evaluate costs and benefits that fall under her jurisdiction. A social insurer covers patients throughout their lifetime. We do so to tie potential patient-physician behaviors to calculate the potential returns to decision making by an insurer and relate it to the potential for commercial success for a manufacturer of a PM test.

Under assumptions 1–7, the EVIC from a social insurer’s point of view, who is interested in maximizing NHB but also bears the total costs of the PM test but not the cost of treatments, is given as

$$\mathit{E}\mathit{V}\mathit{I}{\mathit{C}}_{\mathit{\text{Insurer}}}=\int [\{{\text{max}}_{\mathrm{j}}\mathrm{N}\mathrm{H}\mathrm{B}(\theta )\}+{\mathrm{C}}_{\mathrm{k}}-{\mathrm{C}}_{\mathrm{P}\mathrm{M}}]\mathrm{p}(\theta )\mathrm{d}\theta -[{\text{max}}_{\mathrm{j}}\int \mathrm{N}\mathrm{H}\mathrm{B}(\theta )\mathrm{p}(\theta )\mathrm{d}\theta +{\mathrm{C}}_{\mathrm{k}\text{'}}],$$ (Eq. 3)

where C_k reflect the costs of treatments corresponding to the one that produces the maximum net health benefits for level of θ (i.e. C_k = C_{argmaxjNHB(θ)}), while C_k’ reflects the cost of the treatment that corresponds to the maximum net health benefits on average.

EXTENSIONS TO ORIGINAL EVIC FRAMEWORK

In what follows, we keep assumption 1 and 2 fixed throughout this paper as they represent fundamental structural assumptions of the EVIC framework. Relaxing these assumptions is relegated to future work. We now explore how EVIC_Insurer is altered as we start relaxing the other assumptions implicit in the EVIC framework in order to create a tool that can be used to inform prioritization decisions for PM in practical settings.

Potential uptake of treatments when patients bear the full costs of treatments (Relax Assumption 4)

If patients bear the costs of treatment, the price elasticity of demand, driven by the income distribution in the population, will bring down the use of the treatment even if its expected NHB or individualized NHB is the highest among alternatives. With Assumption#4 relaxed, the EVIC_Insurer will be altered by a new factor k (Table 1, Row 1, Eq. 4), where (1−k) indicates a factor by which EVIC_Insurer in (1) is altered because of the probability that a treatment may not be used even if a physician recommends the treatment. In our breast cancer example, the use of chemotherapy will fall, irrespective of the presence of a diagnostic test, if patients bear the full cost of the drugs.

Table 1.

Alternative EVIC Formulas with Different Assumptions Relaxed

Assumptions Relaxed Compared to Baseline (Eq(3))	EVICInsurer
Treatment uptake reduces as patient bear the full costs of treatments (#4)	$\mathrm{k}·\{\int [\{{\text{max}}_{\mathrm{j}}\mathrm{N}\mathrm{H}\mathrm{B}(\theta )\}+{\mathrm{C}}_{\mathrm{k}}-{\mathrm{C}}_{\mathrm{P}\mathrm{M}}]\mathrm{p}(\theta )\mathrm{d}\theta -[{\text{max}}_{\mathrm{j}}\int \mathrm{N}\mathrm{H}\mathrm{B}(\theta )\mathrm{p}(\theta )\mathrm{d}\theta +{\mathrm{C}}_{\mathrm{k}\text{'}}]\}$	Eq. 1
Treatment covered by insurance (#5)	$\int [\{{\text{max}}_{\mathrm{j}}\mathrm{B}(\theta )\}-{\mathrm{C}}_{\mathrm{k}}-{\mathrm{C}}_{\mathrm{P}\mathrm{M}}]\mathrm{p}(\theta )\mathrm{d}\theta -[{\text{max}}_{\mathrm{j}}\int \mathrm{B}(\theta )\mathrm{p}(\theta )\mathrm{d}\theta -{\mathrm{C}}_{\mathrm{k}\text{'}}]\}$	Eq. 2
Treatment covered by insurance (#5) + Imperfect and uncertain PM (#6)	$\int [\{{\text{max}}_{\mathrm{j}}\mathrm{B}(\mathrm{\theta })|\mathrm{P}\mathrm{M}\}-{\mathrm{C}}_{\mathrm{k}}-{\mathrm{C}}_{\mathrm{P}\mathrm{M}}]\mathrm{p}(\mathrm{\theta })\mathrm{d}\mathrm{\theta }-[{\text{max}}_{\mathrm{j}}\int \mathrm{B}(\mathrm{\theta })\mathrm{p}(\mathrm{\theta })\mathrm{d}\mathrm{\theta }-{\mathrm{C}}_{\mathrm{k}\text{'}}]\}$	Eq. 3
Treatment covered by insurance (#5) + Imperfect and uncertain PM (#6) + Passive personalization (#3)	$\int [\{{\text{max}}_{\mathrm{j}}\mathrm{B}(\mathrm{\theta })|\mathrm{P}\mathrm{M}\}-{\mathrm{C}}_{\mathrm{k}}-{\mathrm{C}}_{\mathrm{P}\mathrm{M}}]\mathrm{p}(\mathrm{\theta })\mathrm{d}\mathrm{\theta }-\int [\{{\text{max}}_{\mathrm{j}}\mathrm{B}(\mathrm{\theta })|\mathrm{P}\mathrm{P}\mathrm{M}\}-{\mathrm{C}}_{\mathrm{k}\text{'}}]\mathrm{p}(\mathrm{\theta })\mathrm{d}\mathrm{\theta }$	Eq. 4
Treatment covered by insurance (#5) + Imperfect and uncertain PM (#6) + Passive personalization (#3) + Variable adoption trajectories (#7)	$\text{Pr}(\mathrm{P}\mathrm{M})·(\int [\{{\text{max}}_{\mathrm{j}}\mathrm{B}(\mathrm{\theta })|\mathrm{P}\mathrm{M}\}-{\mathrm{C}}_{\mathrm{k}}-{\mathrm{C}}_{\mathrm{P}\mathrm{M}}]\mathrm{p}(\mathrm{\theta })\mathrm{d}\mathrm{\theta }-\int [\{{\text{max}}_{\mathrm{j}}\mathrm{B}(\mathrm{\theta })|\mathrm{P}\mathrm{P}\mathrm{M}\}-{\mathrm{C}}_{\mathrm{k}\text{'}}]\mathrm{p}(\mathrm{\theta })\mathrm{d}\mathrm{\theta })$	Eq. 5

Treatments are covered in full by the insurer (Relax Assumption 5)

In this case, two things happen. First, patient choice of treatments is no longer influenced by the costs of treatment as the demand price to the patients is zero. Therefore, factor k drops out of Eq(4). Second, the objective function that the physician is trying to maximize, i.e. patient welfare, may deviate from the social welfare function that a social planner/insurer may try to maximize. For example, the insurer wants to maximize average NHB in the population but the physician may act to maximize only benefits B since patients do not bear the costs of treatment any longer. Therefore, EVIC_Insurer should account for the implicit physician/patient behavior that is induced by covering treatments and is shown in Table 1 (Row 2, Eq. 5), where C_k reflect the costs of treatments corresponding to the one that produces the maximum benefits for level of θ, while C_k’ reflects the cost of the treatment that corresponds to the maximum benefits on average. For example, if breast cancer patients do not bear the costs of chemotherapy, physicians may prescribe treatment only based on expected life years or QALYs, although the insurer would bear the costs of those choices.

EVIC_Insurer in (5) will be lower than EVIC_Insurer in (2) due to the welfare loss from moral hazard induced by covering treatments. In fact, in certain circumstances it is possible that the EVIC_Insurer in (5) can be negative.²² However, the change in magnitude for EVIC in (5) compared to EVIC in (4) is ambiguous. That is by removing the patient’s sensitivity to treatment costs, EVIC in (5) could be greater as the PM test may be able to induce enough benefits to recuperate the costs of treatment now borne by the insurer. Henceforth we will continue to assume that both treatments and PM are covered in full by the insurer, as this is typically the case in most US settings. One can easily use this formulation to capture differential co-pay or coinsurance for the treatments.

Imperfect PM test (Relax Assumption 6)

In this case, say the new PM test has a sensitivity of S_n and a specificity of S_p, where S_n < 1 and S_p < 1. Consequently, the test may lead a physician to choose a treatment for an individual patient that does not produce the maximum benefits for that patient, i.e. max_jB(θ) | PM < max_jB(θ) for some patients. Interestingly, the insurer’s point of view, EVIC follows Eq(6) in Table 1 (Row 3), which means that EVIC_Insurer can be higher with an imperfect test (as in (6) over a perfect test (as in (5) because the imperfection lead to the reduction in inefficiency that occurs due to the failure by physicians to internalize the costs of treatments while making treatment decisions for their patients. While this is not a general result, it illustrates that such an outcome is possible. For example, an imperfect diagnostic test in breast cancer can reduce the inefficiencies generated due to the moral hazard of insurance coverage of chemotherapy.

Uncertain PM test (Relax Assumption 6)

In reality, not only are PM tests less than perfect, their properties are often estimated from samples with considerable uncertainty.¹ In this case, both the imperfect and the uncertain information on the test may lead a physician to choose a treatment for an individual patient that does not produce the maximum benefits for that patient, i.e max_jB(θ) | PM < max_jB(θ) for some patients. Therefore, EVIC_Insurer will have the same expression as in (6).

The EVIC framework can be directly linked to the other concepts of Value of Information (e.g. Expected Value of Perfect Information (EVPI) and the Expected Value of Sample Information (EVSI) that are used to prioritize and value future research that aims to estimate the test properties with greater precision.^23,24 In our formulation:

$$\mathrm{E}\mathrm{V}\mathrm{P}\mathrm{I}=EVI{C}_{\mathit{\text{insurer}}}(\text{with perfect certainty about PM properties})-EVI{C}_{\mathit{\text{insurer}}}(\text{with current levels of uncertainty about PM properties})$$

$$\mathrm{E}\mathrm{V}\mathrm{S}\mathrm{I}=EVI{C}_{\mathit{\text{insurer}}}(\text{with current levels of uncertainty about PM properties})-EVI{C}_{\mathit{\text{insurer}}}(\text{with current levels of uncertainty about PM properties})$$ (Eq. 4)

Note that the difference between this formulation of EVIC and EVSI and those that are typically reported in the literature is that our formulation accounts for the contextual behavior of patients in choosing treatments given their coverage scenarios in establishing the value of research on PM properties. This becomes an important consideration given our next section.

EVIC with passive personalization (PPM) (Relax Assumption 3)

Homogeneous care model leads to the same treatment choice for all patients (either based on maximum expected benefits or net health benefits) as θ_i is unobserved to the physician. However, in clinical practice there is existence of passive personalization (PPM).^25,26 PPM involves a passive approach to personalization where, in the absence of explicit PM test, patients and physicians “learn by doing” mostly because of the repeated use of similar products on similar patients or employ other readily available prediction algorithms. Thus, PPM signifies relaxing Assumption 3, where physicians obtain some knowledge about θ and prescribes treatments accordingly. Any PPM mechanism is deemed to be successful from a patient perspective if patients receiving the new treatment are benefiting more than the average patient.^27,25,26 Therefore, an important aspect of understanding the incremental value of new PM test is to understand the baseline levels of communication or the passive levels of personalization that exists in clinical practice.

In most cost-effectiveness studies of pharmacogenetic testing strategies, the outcomes under the “no-testing” comparator are usually informed by efficacy outcomes for the control arm from randomized trials of the targeted pharmaceutical drug or by the uniform use of a specific dose of the drug.^28,29,30 To the extent that these comparators differ from the reality of clinical practice, they would tend to produce erroneous evaluations of PM test without any clear sense of the direction of the biases. Warfarin pharmacogenomic testing to guide initial dosing (to avoid serious bleeding events) is a good case study. Should warfarin pharmacogenomic testing be compared to warfarin dosing based on a patient’s demographic and clinical characteristics, which is current clinical standard of care, and as was done in a recent RCT,³¹ or should it be compared to a single standard dose (not individualized), as was done in another recent RCT?³² Basing the comparison on the former study is likely to provide a more accurate estimate of the impact of pharmacogenomic testing in this setting.

Presence of PPM is not sufficient to guarantee that the average levels of outcomes will be higher compared to when everyone received the treatment that maximizes average outcome. Thus, PPM can alter the calculation of incremental value between two treatments but not necessarily produce higher average results of health compared to the maximum average health produced by a treatment. For that to happen, the sensitivity and specificity of PPM need to be very high.

Consequently, in our current formulation:

$$\int \{{\text{max}}_{\mathrm{j}}\mathrm{B}(\theta )|\mathrm{P}\mathrm{P}\mathrm{M}\}\mathrm{p}(\theta )\mathrm{d}\theta \underset{<}{\overset{<}{=}}{\text{max}}_{\mathrm{j}}\int \mathrm{B}(\theta )\mathrm{p}(\theta )\mathrm{d}\theta ,$$ (Eq. 5)

depending on the sensitivity and specificity and the uncertainty of PPM. However, even if the extent of PPM is often not known with certainty, ∫{max_jB(θ)|PPM}_p(θ)dθ can be known with greater ease as it represents the average level of outcomes observed in current practice. This can be readily used as the baseline level of outcomes in the EVIC calculations, which would then be more relevant for decision making in practice. For example, such EVIC calculations could be used to assess the value of a pragmatic trial to evaluate a new PM technology where the control group mimics current clinical practice.

An EVIC formulation that accounts for PPM in clinical practice is given by Eq. 9 in Table 1 (Row 4). Therefore, consideration of PPM can actually increase the value of a PM test, especially when the first condition in Eq. 8 holds. Similarly, it would decrease the value of PM test when the last condition in Eq. 8 holds. In our breast cancer example, the genomic test should not be compared to the chemotherapy use, but rather the existing mechanism, such as the NCCN prediction algorithm, of allocating chemotherapy use.

EVIC under alternative diffusion/adoption trajectories of PM (Relax Assumption 7)

We now arrive at the most critical part of the EVIC formulation that is fundamental to prioritizing investments in PM. As Hall (2004) points out,³³ the contribution made by innovation and new technologies to economic growth and welfare is largely determined by the rate and manner by which innovations diffuse throughout the relevant population, a concept largely ignored in the literature on health technology assessments and cost-effectiveness analysis. Understanding the adoption and uptake of a PM test when one becomes available is critical to understand the economics of PM innovation and investments. There is a large literature on technology diffusion.^34,35 In this section we plan to relax Assumption#7 and develop a reduced form model for adoption and uptake among physicians and patients. A PM test, when available, is often not universally adopted. Therefore, the population EVIC calculations must change to account for this selective uptake since the basic premise of value of information calculations is that new information is deemed to have value only if it changes behavior.

The expected population outcomes with a PM test at any point in time is a weighted average of outcomes among those who use the PM test and those who do not. Denoting the proportion of the eligible population receiving PM test as Pr(PM), we have Eq. 10 in Table 1, Row 5. In the absence of any information about the selective uptake of a PM test, a non-selective uptake can be assumed (which means that the {max_jB(θ)|PM} in Eq. 10 is the same as {max_jB(θ)|PM} in Eq. 9) and therefore the overall EVIC_Insurer in Eq. 9 is merely scaled down in Eq. 10 using the fraction of the population receiving the PM test.

A framework for prioritization of research and development (R&D) of PM based on diffusion trajectory and EVIC

Using the refined EVIC framework developed in the last section, we now develop metrics for population level decision making for an insurer making a coverage decision and also a manufacturer making R&D investment decisions for a PM test.

Insurer’s coverage decision

A social insurer cares both about the clinical benefits and also the budget impact due to the costs of technology. We assume that at any point in time (denoted at t = 0), an insurer determines level of reimbursement (e.g by making patient bear a coinsurance rate r) for the PM test by maximizing the net present value of the population

$$r^{*}=\underset{\mathrm{r}}{\text{argmax}}\{\mathrm{p}\mathrm{o}\mathrm{p}\mathrm{E}\mathrm{V}\mathrm{I}{\mathrm{C}}_{\text{Insurer}}(\mathrm{r})=\mathrm{P}·\mathrm{E}\mathrm{V}\mathrm{I}\mathrm{C}{(\mathrm{r})}_{\text{Insurer},0}+{\sum }_{\mathrm{t}=1}^{\mathrm{T}}{\mathrm{d}}^{\mathrm{t}}·{\mathrm{I}}_{\mathrm{t}}·\mathrm{E}\mathrm{V}\mathrm{I}\mathrm{C}{(\mathrm{r})}_{\text{Insurer},\mathrm{t}}\}$$ (Eq. 6)

where d represents a discount rate; P represents the prevalent size of the target population while I represent the time-period specific incident size of the target population over a time horizon T; and following (10), EVIC_Insurer(r) at any point in time is given by:

$$\mathit{E}\mathit{V}\mathit{I}\mathit{C}{(\mathit{r})}_{\mathit{\text{Insurer}}}=\text{Pr}(\mathrm{P}\mathrm{M}|\mathrm{r})·(\int [\{{\text{max}}_{\mathrm{j}}\mathrm{B}(\mathrm{\theta })|\mathrm{P}\mathrm{M},\mathrm{r}\}-{\mathrm{C}}_{\mathrm{k}}-\mathrm{r}·{\mathrm{C}}_{\mathrm{P}\mathrm{M}}]\mathrm{p}(\mathrm{\theta })\mathrm{d}\mathrm{\theta }-\int [\{{\text{max}}_{\mathrm{j}}\mathrm{B}(\mathrm{\theta })|\mathrm{P}\mathrm{P}\mathrm{M}\}-{\mathrm{C}}_{\mathrm{k}\text{'}}]\mathrm{p}(\mathrm{\theta })\mathrm{d}\mathrm{\theta }$$ (Eq. 7)

The time period specific estimate of EVIC_Insurer,t is driven by the time-period specific probability of use of the PM test and can be informed by the estimates from the reduced form models described above. In sum, for an ex ante analysis, these estimates should reflect the anticipated diffusion patterns of the PM test in the population.

Denoting r = 1 to be full coverage and r=0 as no coverage the benefit-cost metric for covering a PM test from the insurer’s perspective is given as: NPV(r=1) − NPV(r =0), which implies comparing Eq(12) under alternative diffusion curve of the PM test with and without coverage. Alternative decisions on partial coverage based on instituting copays or a coinsurance rate can also be studied using this framework as predictions of diffusion rates will be altered based on the price elasticity of demand.

PM test manufacturer’s R&D investment decision

A PM test manufacturer’s decision to invest in developing a PM would depend on the probability of regulatory success and the extent of commercial success for the PM test as they represent returns to such investments.² Since only a relatively small number of available companion diagnostic tests have gone through the Food and Drug administration regulatory approval route, we mostly focus on commercial success here.³ These approaches can be easily extended to estimating returns of developing PM used for diagnosis of disease of disease risks.

Commercial success would be based on the revenues the manufacturer can expect though the sales of the PM test. Consequently, expected revenues of a PM test sales is given by:

$$\mathrm{R}\mathrm{E}\mathrm{V}={\int }_{\mathrm{V}}\{\mathrm{P}·\text{Pr}(\mathrm{P}\mathrm{M}|{\mathrm{r}}^{*})·{\mathrm{C}}_{\mathrm{P}\mathrm{M}}+{\sum }_{\mathrm{t}=1}^{\mathrm{T}}{\mathrm{d}}^{\mathrm{t}}·{\mathrm{I}}_{\mathrm{t}}·\text{Pr}(\mathrm{P}\mathrm{M})|{\mathrm{r}}^{*})·{\mathrm{C}}_{\mathrm{P}\mathrm{M}}\}\mathrm{f}({\mathrm{r}}^{*})\mathrm{d}{\mathrm{r}}^{*}$$ (Eq. 8)

Where r^*, the coinsurance rate for the PM test that the insurer will find optimal, is a stochastic quantity for the manufacturer at the point of development of the test. Let f(r^*) denotes the anticipated probability distribution function of r^*. One potential method to estimate f(r^*) is based on the stochastic values of popEVIC_Insurer from (11) under alternate levels of r and then evaluate the Pr(r = r^*).

We now illustrate these EVIC concepts by applying them to a case study of OnctotypeDX, a PM test used to identify breast cancer patients for chemotherapy treatment.

EVIC calculations for gene-expression profiling for early stage breast cancer

In this section, we illustrate how the EVIC estimates change as we relax some of the assumptions built into the original EVIC framework. We also study the role of diffusion and coverage policies on the expected population value of precision medicine and expected revenues to a manufacturer that may help guide optimal R&D investment decisions for precision medicine.

Our illustration is based on a gene expression test to identify patients with early stage breast cancer who are likely to benefit from adjuvant chemotherapy vs. no test. Although adjuvant chemotherapy has been shown to improve recurrence-free and overall survival,³⁶ it also may cause significant toxicity, sometimes fatal, in patients. Personalization approaches have focused on identifying women who are at a higher risk of distant recurrence and therefore the prime candidates for adjuvant chemotherapy while sparing low-risk patients from these treatments.

In the absence of a genomic test, we start by assuming that there is no other way to personalize chemotherapy. All patients receive chemotherapy. We then relax this assumption based on clinical practice that followed established guidelines for the use of chemotherapy, which includes a form of personalization based on phenotypical prediction rules (e.g., tumor size). A number of cost-effectiveness studies have looked at the net value of specific genomic assays (OncotypeDx, MammaPrint etc) compared to phenotype-based prediction models developed by National Institutes of Health (NIH) or the Adjuvant! Online Program (AOL).^37,38,39,40 We developed a simple Markov simulation model mirroring the features developed in the earliest works^{37, 38} and then use it to estimate different value of information metrics over a 20-year horizon under different assumptions. We then use this model to explore the different EVIC concepts discussed in this paper. All analyses are conducted from an “ex ante” perspective, i.e. before the introduction of the genomic test.

Markov Model for Breast Cancer Treatment

Figure 1 describes the model used for our analysis. A cohort of lymph-node-negative, estrogen-receptor-positive patients averaging 44 years of age with early-stage breast cancer is considered to either go through standard of care where they receive phenotypical predictions about their recurrence prognosis or receive a genomic test to obtain their recurrence prognosis. If prognosis is poor under either scenario, based on the threshold established under each scenario, patients always receive treatment as long as they do not bear the costs of treatment. If the patients bear the costs of treatment, demand for treatment is reduced depending on the price-elasticity of demand for oncology drugs.⁴¹ If patients receive treatment, they may experience fatal toxicity. Otherwise, their true underlying risk status and treatment status determine their lifetime recurrence and death probabilities. Quality of life in each year is determined by their treatment receipt and their recurrence statuses. All parameters in this model are expressed as probability distributions in order to reflect the uncertainty in their estimates. Table 3 and the Appendix provide a comprehensive list of all parameters. Net monetary benefits are calculated using a $100,000/QALY threshold.⁴²

Figure 1.

Table 3.

EVIC-Relevant Parameters

		Assumptions Relaxed
Parameters	Base Case	#5	#5, #6	#5,#6,#3	#5,#6,#3,#7
Pr(High Risk)	0.10+
Pr(Poor Prognosis |PM)	0.49*
Pr(Poor Prognosis |PPM)	0.93*
Specificity of PM	1.0		0.51	0.51	0.51
Sensitivity of PM	1.0		0.84	0.84	0.84
Specificity of PPM	0.5†			0.05‡	0.05
Sensitivity of PPM	0.5†			0.98‡	0.98
Pr(PM Use)	1.0				<1.0
Patient coinsurance rate for treatment costs	1.0	0.0	0.0	0.0	0.0

⁺Derived based on Paik et al. (2004);¹ Hornberger et al. (2005);³⁷ Oestreicher et al. (2005)³⁸

^*Hornberger et al. (2005);³⁷

^†No phenotypical predictions available;

^‡Oestreicher et al. (2005).³⁸

Estimating diffusion of OncotypeDx

We identify the use of the genomic test (HCPCS: S3854) in the population of lymph-node-negative, estrogen-receptor-positive female patients with early-stage breast (index date identified by the first occurrence of breast cancer ICD-9 code 174.XX or 233 with no such occurrence of these codes in the past 12 months) during 2006 to 2010 within a large integrated healthcare system. Based on the literature on technology diffusion that puts forth a S-shaped diffusion pattern for most technologies,^34,35 we model the rate of use of this test, S(t), using a Gompertz functional form that asymptotes to 1:

$$\mathrm{P}(\mathrm{t})=\text{exp}(\alpha ·\text{exp}(\beta ·t)$$ (Eq. 9)

Let y_it denote an indicator for the use of the genomic test by patient i at time t. Therefore, E(y_it) = P(t). Since the genomic test was introduced to the market in 2003 for these patients, we use data corresponding to years 5 to 8 (2008 – 2011) of the product life-cycle. In order to fit the Gompertz functional form, we fit a complementary log-log link function model to y'_it = 1 − y_it as a function of t (expressed in product life cycle year). Since, 1 − E(y'_it) = P(t), we have

$$c\text{loglog}(y{\text{'}}_{it})=\text{ln}(-\text{ln}(\mathrm{P}(\mathrm{t}))=\text{ln}(-\alpha )+\beta ·t$$ (Eq. 10)

Alternative Scenarios for EVIC Calculations

Our evaluation of EVIC follows the theory discussed in the previous section and summarized in Table 2. We start with estimating the simple EVIC concept in (3) with assumptions 1–7. All values are expressed in 2004 US dollars (when OncotypeDx was introduced) and QALYs were valued at $100,000/QALYs. In this case, we assume that the comparator is such that no other form of personalization is available (to be relaxed subsequently). We then proceed by relaxing the assumptions as illustrated in Table 2. For passive personalization, we change the comparator to that where phenotypical predictions are used for personalization of chemotherapy.

Table 2.

Evaluation of EVIC Concepts for a Genomic Test for Breast Cancer Patients

EVIC Concepts (Eq.)	Comparator	Assumptions Relaxed Compared to Baseline	Per Patient
			Estimated Incr. Benefits (QALYS)	Estimated Incr. Costs (in 2003 US $)	Estimated EVIC Value* (in 2003 US $)
EVICInsurer(Eq.3)	No phenotypical predictions	Baseline (Assumptions 1 – 7 enforced)	0.29	6,478	23,182
EVICInsurer(Eq.4)	No phenotypical predictions	Treatment uptake reduces as patient bear the full costs of treatments (#4)	0.089	4,376	4,534
EVICInsurer(Eq.5)	No phenotypical predictions	Treatment covered by insurance (#5)	0.29	−13,222	42,883
EVICInsurer(Eq.6)	No phenotypical predictions	Treatment covered by insurance (#5) + Imperfect and uncertain PM (#6)	0.11	−5309	15,806
EVICInsurer(Eq.9)	Phenotypical predictions (PPM)	Treatment covered by insurance (#5) + Imperfect and uncertain PM (#6) + Passive personalization (#3)	0.09	−4,441	13,659
EVICInsurer(Eq.10)	Phenotypical predictions (PPM)	Treatment covered by insurance (#5) + Imperfect and uncertain PM (#6) + Passive personalization (#3) + 50% uptake (#7)	0.045	−2,220	6,830
EVICInsurer(Eq.12)	Phenotypical predictions (PPM)	Treatment covered by insurance (#5) + Imperfect and uncertain PM (#6) + Passive personalization (#3) + 50% uptake at r = 0 (#7) + PM covered partially (r = 0.50)**	0.031	−2,133	5,315

^*QALYS valued at $100000K/QALYS;

^**Corresponds to a 34.5% uptake. r coinsurance rate.

To relax assumption #7 on the diffusion of use of the genomic test, we use the estimated diffusion rate of the genetic test from its introduction. Estimates of diffusion patterns beyond the observed life cycle of the test will be based on the Gompertz parametric form and the estimated parameters. Uncertainty in this extrapolation will be accounted for in our probabilistic analyses. The observed coinsurance rate for the genomic test among the beneficiaries of the large integrated healthcare system in the US was 20%. If there were a higher coinsurance rate for this genetic test, the use probabilities would have been lower. Similarly, full coverage would have increased the use rates. Note that even though our analyses perspective in “ex ante”, we anticipate these estimated diffusion patterns at the point of analyses. Predicting such diffusion curve prospectively for other genomic tests is discussed in the Conclusions Section.

We study the effect of alternative coinsurance rates for the genetic test using the EVIC concept in (12) which is based on relaxing assumptions #5, 6, 3, 7, and also introduce coinsurance rate r~Uniform(0,1) for the genomic test. In the absence of a direct estimate of price elastic of demand for a genomic test, we apply an estimate of the arc price elasticity of demand for oncolytic drugs (Goldman et al, 2010), which reports a 5% increase in use with every 25% decrease in price. Since we observe the diffusion curve, S(t)_Obs, at r =0.20, for any other given level of r and time t in the product life cycle, we shift the estimated empirical use rate for the genomic test downward or upward by multiplying S(t)_Obs with (1.05)^{((ln(r)/ln(0.75)) − 5.5)}.⁴ To approximate the scenario where r = 0, we consider r = 0.001, which corresponds to an inflation factor of (1.05)^{((ln(r)/ln(0.75)) − 5.5)} = 2.46 to be applied to S(t)_Obs, truncating the rates to lie between [0,1].

Next, we study the PopEVIC_Insurer(r), expressed in (11) as a function of coinsurance rate r to test whether immediate coverage of the genetic test would have been warranted, which was not done in practice. That is, whether r = 0 produces the maximum PopEVIC_Insurer. Specifically, we calculate population EVIC over 20 annual cohorts of 178,520 patients (based on 2002 SEER), each cohort followed over 20 years and using discounting rate of 3%. However, each cohort enters the model at different points on the diffusion curve of the genetic test.

Finally, we study the manufacturer’s incentive to develop this genomic test based on anticipation of coverage decision by an insurer. We estimate the Pr(r = r^*) for each value of r to reflect the anticipated probability of a given level of coverage. We then estimate the expected revenues for the manufacturer, internalizing this probability distribution.

Results

Table 2 presents estimates of EVIC (and corresponding Eq) of a genomic test for breast cancer patients as we sequentially relax the assumptions from the original framework.

EVIC_Insurer (3) enforces Assumptions 1–7. It compares the expected value of a perfect genomic test, which is always used, to having no personalization for choosing chemotherapy (which is not covered but yet perfect uptake happens) among breast cancer patients is estimated to be $23,182 per patient, with an incremental expected benefit of 0.29 QALYS and an expected incremental costs of $6478. Note that this hypothetical scenario is only useful when determining the value of developing a test, when no other test exits.

Potential uptake of treatments when patients bear the full costs of treatments (Relax Assumption 4)

Since the patients bear the cost of the chemotherapy treatments, it is natural to imagine that uptake of chemotherapy will not be perfect and will be driven by the price elasticity of demand (Relaxing Assumption #4). On the genomic test side, this decreases the use of chemotherapy even when prognosis is poor, thereby reducing overall value. On the no personalization side, it helps reduce the use of chemotherapy among those who would have been harmed by them without any commensurate benefits, thereby increasing value. Consequently, the net effect is ambiguous, and we find that EVIC_Insurer(4) reduces to $4,534 as the expected incremental benefits reduces to 0.089 while the expected incremental costs decreases to $4,376.

Treatments are covered in full by the insurer (Relax Assumption 5)

However, when the chemotherapy is also covered fully by insurance (relaxing Assumption#5), chemotherapy uptake recovers to baseline levels, but insurers now bear the costs of chemotherapy. EVIC_Insurer(5) increases to $42,883 per patient. This is because, in the presence of a perfect personalizing mechanism, promoting the use of chemotherapy when it would really improve outcomes and reduce costs is likely to generate more value.

Imperfect and Uncertain PM test (Relax Assumption 6)

Naturally, if that personalizing mechanism were not perfect, then some of the expected value would be lost. For example, when we introduce the test properties of the OncotypeDx test for this genomic test (relaxing Assumptions #5,6), EVIC_Insurer(6) decreases to $15,806 per patient.

EVIC with passive personalization (PPM) (Relax Assumption 3)

Note that till now the genomic personalization is being compared to no personalization at all. However, clinical practice of prescribing chemotherapy to these patients would follow the standard NCCN guidelines, which tries to personalize treatment decisions based on phenotypical factors. Therefore, when EVIC for the genomic test is evaluated with respect to the baseline levels of personalization that exist in the absence of the genomic test (relax Assumptions #5,6,3), EVIC_Insurer(9) decreases further to about $13,659.

EVIC under alternative diffusion/adoption trajectories of PM (Relax Assumption 7)

Next, we study how EVIC would change if the uptake of the genomic test occurs only in 50% of the patients (even when the test is fully covered by the insurer), but this constant uptake applies throughout the life time of the test. In this case (relaxing Assumptions #5,6,3,7), the insurer loses value as not all patient are getting the test but saves money on the cost of the test. EVIC_Insurer(10) reduces to $6830. On top of this, if the insurer places a high coinsurance rate of, say 50%, it would imply an uptake of the genomic test of only 34% in the population and an EVIC_Insurer(12) of $5,315.

Finally, we study how expected diffusion rates (in contrast to a constant uptake rate) would influence the population EVIC calculations. With perfect uptake of the genomic test under full coverage, which corresponds to an EVIC_Insurer(9) of about $13,659, the popEVIC_Insurer is estimated to be $37.4 Billion (Table 4). The corresponding expected revenue for the manufacturer of the test is estimated to be $9.5 Billion. However, the empirical diffusion rates observed for the genomic testing among breast cancer patients suggest that diffusion of this technology follows the traditional S-shaped pattern of technology diffusion (Figure 2) at the observed coinsurance rate of 20%. By the 7^th year since its introduction in 2003, 20% of the targeted breast cancer patients were getting genomic tests.

Table 4.

Population Returns to Insurer and Manufacturer under Alternate Diffusion Scenarios

	With Perfect Uptake Pr(PM) =1	With Predicted Uptake++ (at r ~0.20)	With Predicted Uptake++ (at r ~0.001)
Parameters
Eq12: popEVICInsurer (in Bilions $)+	37.4	16.8	23.9
Eq. 13: Expected revenue to manufacturer (in Billions $)+	9.5	4.25	6.08

⁺Over 20 annual cohorts of patients, each cohort followed over 20 years, Discounting at 3%.

⁺⁺Follows observed patterns of uptake (See Figure 2).

Figure 2.

Our empirical model fits the observed data well (Figure 2). This model was used to predict the potential diffusion pattern of this technology over the first twenty years since its introduction under the fixed coinsurance rate of 20% (Figure 2). If diffusion follows this trend, it would take nearly 20 years for uptake of this genetic test to be universal. Incorporating this predicted diffusion pattern produces as estimate of popEVIC_Insurer to be only $16.08 billion and the expected revenue for the manufacturer to be $4.25 billion.

Borrowing an estimate of the price elasticity of demand for oncolytic drug and applying it to the demand for the genomic test (along with it uncertainty), we predict that if the genomic test were to be fully covered (coinsurance rate ~0), diffusion of the genomic test would be much faster that what is empirically observed (Figure 2). In fact, these estimates (going from 5% in 2005 to 40% in 2009) match closely to recent estimates produced for the youngest patients in Medicare (who bear the closest resemblance to our target population), where the test is fully covered.⁴³ Based on our predictions, under full coverage, universal uptake of the genomic test could happen by year 8 since its introduction. With such an altered prospect of diffusion, popEVIC_Insurer would have increased to $23.9 billion and the expected revenue for the manufacturer to $6.08 billion.

Finally, in order to understand the expected revenue for the manufacturer when internalizing the stochastic nature of coverage decisions that may follow, we studied the Pr(r = r^*) for a range of coinsurance rates starting from r = 1 (no coverage) to r = 0 (full coverage) by increments of 0.10. Since we do not model selective uptake of the test with respect to r, i.e., phenotypically identified high/intermediate risk patients have the same price elasticity of demand as those identified as low risk,⁵ the probability distribution for likelihood of coverage converges to a bimodal distribution at r = 1 with probability 0.44 and r = 0 with probability 0.56. The expected revenue for the manufacturer then is calculated as: 0.44 * $3.25B + 0.56 * $6.08B = $4.83Billion, about half of the expected revenue under perfect uptake of the genomic test (Table 4).

Implications for EVIC calculations

The range of EVIC results show that decision-relevant metrics for evaluating the value of PM would produce very different assessments of the PM as various behavioral assumptions are relaxed from a traditional economic evaluation framework. For example, traditional cost-effectiveness analyses may not be informative about the range of decisions or policy contexts that are relevant to an insurer. In fact, our results from EVIC based on Eq (9), where using the genomic test produces incremental QALYs of 0.09 and a cost savings of about $4000, is identical to results that Hornberger et al.³⁷ reported using the same assumptions implicit in Eq (9). However, as more behavioral variables, such as diffusion of technology and reimbursement levels, are tested and built into this framework, EVIC estimates change substantially. We believe that these methods can provide decision makers with more decision-relevant tool to explore the value of PM.

Discussions

In summary, we developed an economic framework that captures the value of individualized care derived from genomics-based precision medicine. We started with an established framework, the expected value of individualized care (EVIC), that attempts to estimate the upper bound on the value of a PM test. We show that many of the assumptions built into such a framework may not be met in practice. We, therefore, start relaxing these assumptions, and present a more decision-relevant metric of valuing PM tests. Application of this framework to the evaluation of the value of OncotypeDx among breast cancer patients replicated some of the prior results of it’s evaluation, but also illustrated that those results are based on assumptions that are often not met in practice. Instead we show how incorporating more practical aspects of behavior around PM could lead to drastically different estimates of value. In fact, our framework could be used to directly evaluate alternative reimbursement levels for PM tests, implementation and education programs for physicians and patients, and decisions around research investments by manufacturers and public entities using the metrics expressed in Eqs. 11 and 13. Although, in this work, we have highlighted the social insurer’s and manufacturer’s perspectives, similar calculations could be made from the perspective of a payer, clinician or a patient.

There are however, some limitations in using this framework prospectively in actual decision making, but which can be easily resolved with research that is currently underway. Perhaps the biggest is the need of predicting the diffusion rate of a technology. There is a growing recognition that data on adoption is important to decision makers.⁴⁴ In our example, we had the luxury of looking retrospectively and using the observed diffusion rate of OncotypeDx. In prospective settings, decision making on coverage policies or investments needs to happen before observing the diffusion rate. In fact, as we have shown in our example, those diffusion rates would inherently depend on the decision that is being made. These types of exercise are not uncommon in the literature. Most budget impact analyses make an attempt to predict diffusion rate of a technology.⁴⁵ However, more research is needed to better understand adoption decisions of PM by physicians and patients as a function of current evidence and also other idiosyncratic factors. Currently, we are working towards developing such models as part of a large NIH–funded U01 project, that, when validated based on its ability to replicate observed diffusion rates of PM, could be incorporated in the EVIC framework to evaluate new PM and identify critical areas for research investment – particularly around evidence generation.

Another limitation of this framework, as illustrated, is the reliance on the net monetary benefits framework to inform decision making. Like cost-effectiveness analyses, it should always be pointed out that metrics such as EVIC are meant only to inform decision making, not to be the sole guide. The framework can easily be adopted to any objective function that a decision maker chooses to employ. Even when such an objective function is not explicit and multi-criteria decision making is employed to making policy decisions, EVIC results can provide an important component of that decision making processes. Furthermore, extensions of this work to evaluate policies such as FDA labeling, direct to consumer marketing, integration into clinical practice guidelines and pathways, and incorporate the potential impact of the Affordable Care Act and future proposed payment models on diffusion of precision medicine technologies are possible and relegated to future work.

With the growth of PM and the increasing investments in this area, as highlighted by President Obama’s Precision Medicine Initiative,¹⁴ flexible, decision-relevant metrics that capture the complexity of value and ultimately uptake of PM are needed to help guide research investment and coverage policy decisions. Here, we propose such a framework that could help decision makers in this process.

APPENDIX

Conceptual framework for original EVIC (Basu and Meltzer, 2007)

Let the patients face two treatment options, with j =1 denoting the new treatment. Let the net health benefits (NHB_j) from any treatment j be given by the difference between the benefits (B_j) from the treatment minus the benefits-equivalents of the costs (C_j) of the treatment. For simplicity, let’s assume that C_j includes only the price of treatment (although, the analysis could easily be extended to include all the downstream costs of treating the patients with treatment j). C_j is obtained by dividing costs by λ, which is the marginal social willingness to pay for health improvement: NHB_j(θ) = B(θ) − C_j. (Costs could also depend on θ, but we keep it constant for simplicity). Alternatively, one could express the value to patients in net monetary benefits (NMB) terms where, instead of calculating benefits-equivalents of costs, one calculates the cost-equivalents of benefits by multiplying B_j with λ and subtracting costs from it. Under assumption 1–7, we consider NHB to be the welfare measure for the patients and to the society. Alternate welfare measures, such as only net benefits, could be easily accommodated in this framework.

Under the homogeneous care model, physicians are unaware of the values of θ for individual patients but base their decisions on the distribution p(θ), trying to maximize the patient’s net health benefits. Thus, physicians may choose the treatment that maximizes expected net health benefits and prescribe it to all patients. The average per patient value (V) obtained from physician’s choice under this model is given by:

$$\mathit{V}\mathit{(}\mathit{\text{Homogeneous Care}}\mathit{)}={\mathit{\text{max}}}_j\int \mathit{N}\mathit{H}B(\theta )p(\theta )d\theta$$

Under the individualized care (IC) model, individualized care is attained by identifying the levels of patient attributes using the perfect PM test (100% specificity, 100% sensitivity), and the physician using the individual levels for θ to make treatment decisions. Thus, the physician chooses different treatments for different patients so that the net health benefits are maximized for each patient given his true value of θ. Hence, under the IC model, the average per patient value (V) obtained is the expected maximum net health benefits as given by:

$$\mathbf{V}(\mathbf{I}\mathbf{C})=\int \{{\mathit{\text{max}}}_{j}NHB(\theta )\}p(\theta )d\theta .$$

Under Assumption 1–7, the expected value of individualized care (EVIC) is then given by:

$$\mathit{E}\mathit{V}\mathit{I}\mathit{C}=\mathrm{V}(\text{Individualized Care})-\mathrm{V}(\text{Homogeneous Care})=\int \{{\mathrm{\text{max}}}_{\mathrm{j}}\mathrm{N}\mathrm{H}\mathrm{B}(\mathrm{\theta })\}\mathrm{p}(\mathrm{\theta })\mathrm{d}\mathrm{\theta }-{\text{max}}_{\mathrm{j}}\int \mathrm{N}\mathrm{H}\mathrm{B}(\mathrm{\theta })\mathrm{p}(\mathrm{\theta })\mathrm{d}\mathrm{\theta }.$$