Emerging Therapies for COVID-19: The Value of Information From More Clinical Trials

Abstract

Objectives

The COVID-19 pandemic necessitates time-sensitive policy and implementation decisions regarding new therapies in the face of uncertainty. This study aimed to quantify consequences of approving therapies or pursuing further research: immediate approval, use only in research, approval with research (eg, emergency use authorization), or reject.

Methods

Using a cohort state-transition model for hospitalized patients with COVID-19, we estimated quality-adjusted life-years (QALYs) and costs associated with the following interventions: hydroxychloroquine, remdesivir, casirivimab-imdevimab, dexamethasone, baricitinib-remdesivir, tocilizumab, lopinavir-ritonavir, interferon beta-1a, and usual care. We used the model outcomes to conduct cost-effectiveness and value of information analyses from a US healthcare perspective and a lifetime horizon.

Results

Assuming a $100 000-per-QALY willingness-to-pay threshold, only remdesivir, casirivimab-imdevimab, dexamethasone, baricitinib-remdesivir, and tocilizumab were (cost-) effective (incremental net health benefit 0.252, 0.164, 0.545, 0.668, and 0.524 QALYs and incremental net monetary benefit $25 249, $16 375, $54 526, $66 826, and $52 378). Our value of information analyses suggest that most value can be obtained if these 5 therapies are approved for immediate use rather than requiring additional randomized controlled trials (RCTs) (net value $20.6 billion, $13.4 billion, $7.4 billion, $54.6 billion, and $7.1 billion), hydroxychloroquine (net value $198 million) is only used in further RCTs if seeking to demonstrate decremental cost-effectiveness and otherwise rejected, and interferon beta-1a and lopinavir-ritonavir are rejected (ie, neither approved nor additional RCTs).

Conclusions

Estimating the real-time value of collecting additional evidence during the pandemic can inform policy makers and clinicians about the optimal moment to implement therapies and whether to perform further research.

Introduction

Amidst >250 million cases worldwide and 3000 to 5500 daily hospitalizations in the United States alone, the COVID-19 pandemic represents the greatest global public health crisis since 1918.^{1 , 2} In the absence of known effective pharmaceutical interventions during early stages of the pandemic, many clinicians prescribed treatments off label. Since the start of the outbreak, > 3500 clinical trials investigating potential therapies have been registered,³ and new trials continue to emerge. These trials are all competing for resources and patient enrollment. Decisions on early implementation of promising treatments have been the source of substantial academic and public debate; nevertheless, objective criteria for research prioritization remain absent.^{4 , 5}

Policy makers and clinician-researchers face a difficult choice—giving emergency use authorization⁶ (approval of the drug conditional on conducting more research), approval of the drug for widespread clinical implementation, approval of the drug only in research (OIR), or rejection of the drug based on limited existing data.⁴ A “study, then treat or reject” approach would optimize expected benefit by gaining more certainty about treatment effects, whereas a “treat first, investigate later” approach seeks to prevent lives being lost because of delayed implementation and denial of potentially beneficial treatments. Nevertheless, this strategy increases the risk of harm from implementation of a possibly ineffective or deleterious treatment.

At a given point in time, findings from completed clinical trials—representing the current body of evidence—can be modeled to provide an estimate of the potential (health) benefits of further research or implementing findings of existing research.⁷ A key tool to quantify this cost-benefit trade-off is value of information (VOI) analysis. VOI quantifies the value of treatment choices made with the expected evidence from additional research compared with making the choice based on currently available information.7, 8, 9 VOI is increasingly applied as part of health economic evaluations,10, 11, 12, 13, 14, 15, 16, 17 both to aid the determination of the optimal sample size and to direct research efforts to where the greatest return can be expected from finite resources.¹⁰ Although meta-analyses investigate drug efficacy and effectiveness and commonly conclude that further research is needed based on a lack of statistical significance,⁷ VOI results consider both current uncertainty relevant to the decision and the potential consequences of making decisions with and without further evidence.¹⁶ These VOI analyses can be used to quantify the benefit of further research, in terms of reducing uncertainty around treatment efficacy and avoiding unintended harm that would result from premature use of a therapy that turns out to be ineffective or deleterious. The benefits of further research are then balanced against possible forsaken benefits because of delayed implementation and research costs.⁷

In this study, we apply VOI to express the value of performing further randomized controlled trials (RCTs) with delayed approval decisions (use only in RCTs), emergency use while performing RCTs (approval with research), or immediate approval of treatments for COVID-19 versus rejection without further research. Focusing on drug therapies for hospitalized patients with COVID-19 for which large RCTs or meta-analyses have been published to date, we examined hydroxychloroquine,¹⁸ remdesivir,^{19 , 20} casirivimab-imdevimab,²¹ dexamethasone,²² baricitinib-remdesivir,²³ tocilizumab,²⁴ lopinavir-ritonavir,²⁵ and interferon beta-1a²⁰ compared with usual care. We considered this as a noncompeting choice problem, given that each of these drugs may be beneficial in the armamentarium of drug therapies for COVID-19. Our analysis aims to inform both treatment decisions and research prioritization decisions regarding therapies for hospitalized patients with COVID-19 and demonstrate how a VOI approach can inform clinical and public health decision making during a pandemic. Glossary of terms were illustrated in Table 1 .

Table 1.

Glossary of terms.

WTP	A threshold that represents what the decision maker or society is willing to pay for a unit of health outcome. The threshold is expressed in monetary units per health outcome.
ICER	A ratio demonstrating the trade-offs between costs and benefits, calculated as the ratio of the incremental cost of an intervention to the incremental benefit in health outcomes
iNHB	A summary statistic representing the impact of an intervention on a population’s health for a given WTP threshold, compared with an alternative intervention, calculated as follows: incremental health benefit – incremental cost of the intervention/WTP threshold
iNMB	A summary statistic representing the value of an intervention in monetary terms for a given WTP threshold, compared with an alternative intervention, calculated as follows: incremental health benefit × WTP threshold – incremental cost of the intervention.
iNB	A summary statistic representing the impact of an intervention on population outcome compared with an alternative intervention, calculated as either incremental net health benefit or incremental net monetary benefit
PA	A technique used to propagate uncertainty from model inputs to model outcomes, also referred to in the literature as PSA.
VOI analysis	The estimation of decision uncertainty and the value of collecting more information on key parameters influencing a decision, expressed in monetary or health terms
Overall strategy	The combined choice of strategy with respect to both treatment and research. The options for the overall strategy are as follows: - OIR, where the drug is made available to patients participating in further research trials, but no emergency use authorization or widespread use is granted - AWR, where emergency use authorization is granted, the drug is made available to patients, and in addition further research is being conducted - Approval, where the drug is immediately approved and no further research is conducted - Reject, where the drug is rejected and no further research is conducted. This situation is the default strategy where usual care does not change.
Population EVPPI	The value of collecting perfect information on selected parameter(s) or subset(s) of parameters in the model, extrapolated to the size of the target population that can benefit from the information (future patients) over a specific time horizon
Population EVSI	The value of collecting additional information on selected parameter(s) or subset(s) of parameters in the model with a trial with finite sample size, extrapolated to the size of the target population that can benefit from the information (future patients) over a specific time horizon
Costs of performing research	Resources required to perform a new trial (fixed cost and variable cost per participant) plus, for study participants, the foregone benefit because of randomized assignment to suboptimal treatment in the trial
Net benefit due to implementation	Incremental net monetary (or health) benefit that is gained because implementation of a beneficial therapy is approved, through either emergency use authorization or definitive approval. This net benefit is foregone in current patients if approval and implementation are delayed whereas more evidence is obtained from further RCTs (OIR strategy).
Net value of research (new RCT)	Expected value of performing further research, in this analysis a new RCT (population EVSI), minus the cost of performing the RCT
Net value of the overall strategy	The net value of the combined treatment-and-research strategy that equals the net value of performing an RCT if further research is performed plus the net benefit of treatment if approved

AWR indicates approval with research; EVPPI, expected value of partial perfect information; EVSI, expected value of sample information; ICER, incremental cost-effectiveness ratio; iNB, incremental net benefit; iNHB, incremental net health benefit; iNMB, incremental net monetary benefit; OIR, only in research; PA, probabilistic analysis; PSA, probabilistic sensitivity analysis; RCT, randomized controlled trial; VOI, value of information; WTP, willingness to pay.

Methods

Decision Trade-off

We performed a model-based cost-effectiveness analysis to examine both costs and health outcomes for all treatments considered. Next, we used the VOI framework to determine the net value of research. This estimate quantifies the trade-off between resources required for another RCT and the added value of the RCT to gain more solid evidence (Fig. 1 ).²⁶ If the net value of research exceeds zero, performing a new RCT is worthwhile and the new evidence should be incorporated into the decision making (Fig. 1: upper quadrants). Ideally, trials are performed until the cost of future research exceeds the expected benefits (lower quadrants). This process can be applied to trials that demonstrate potential beneficial treatment effects (right quadrants) and those demonstrating no beneficial effects (left quadrants).

Figure 1.

Trade-off between implementation of promising COVID-19 treatments and conducting further research. (A) Net value equations and description of trade-offs. Demonstrates the equations used to quantify the net value for the overall strategy options, compared to reject as default strategy. These equations take iNB, EVSI, RCT cost, and number of patients (current and future) into account. iNB may be expressed in monetary units (NMB) or health units (NHB). One could also consider irrecoverable costs for the implementation of a new treatment or the possible reversal of implementation. However, in our analysis implementation and reversal costs are assumed negligible and therefor, shown in gray. The figure additionally shows the advantages and disadvantages of the corresponding implementation and research strategy. These quadrants are based on whether the drug’s current evidence suggests benefit versus standard care or placebo (the right quadrants) or not (left quadrants). Within these right and left quadrants, the upper and lower quadrants indicate whether the value of doing additional research to reduce the uncertainty in benefit exceeds the “cost” of performing additional research expressed economically or (as quality-adjusted) life years lost) in the upper quadrants or not in the lower quadrants. (B) Net value results for optimal strategy. The net value results for the currently existing evidence and its uncertainty for eight drugs are calculated and each drug is placed in the resulting optimal health policy quadrant. Other factors, in particular ethical issues, also need to be considered to decide whether a strategy is desirable. For our study this is particularly true for Hyrdroxychloroquine. H∗ = Hydroxychloroquine: OIR has the highest net value if further research would demonstrate decremental cost-effectiveness (that is, saving costs but with loss of quality-adjusted life years). The ethics of investigating such decremental cost-effectiveness should be considered. If not justifiable, then hydroxychloroquine would move to the Reject category, where the net value would be 0.

Model Description

We simulated the effect of treatment for a cohort of hospitalized patients with COVID-19 based on meta-analyses or large multicenter RCT’s that reported in-hospital mortality. For this simulation, we used a Markov cohort state-transition model with 4 health states (hospitalized, recovered from hospital ward as highest level of care, recovered from intensive care unit [ICU], and dead) (Appendix Fig. 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.03.016). As in the RCTs, hydroxychloroquine,¹⁸ remdesivir,^{19 , 20} casirivimab-imdevimab,²¹ dexamethasone,²² baricitinib-remdesivir,²³ tocilizumab,²⁴ lopinavir-ritonavir,²⁵ and interferon beta-1a²⁰ were compared with the control arm (usual care) rather than to each other. We considered the implementation and research decision as a noncompeting choice problem. Each drug has the potential to be a valuable component in the armamentarium against COVID-19. The various drugs have different therapeutic mechanisms, for example, as an antiviral or corticosteroid, and may be useful in sequence, in combination, or in different contexts. Similarly, from a regulatory perspective, approval is based on available safety and efficacy evidence. Given that it is a noncompeting choice problem, the incremental cost-effectiveness ratio was calculated for each drug individually compared with care as usual. This choice allowed treatments investigated in earlier trials to become integrated in control arms in later trials. Furthermore, this modeling choice allowed for differences in context and trial populations, such as percent of patients in the ICU and hospital wards.

All model parameters were based on best-available evidence as of November 1, 2021 (Appendix Table 1 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.03.016). The probabilities to transition between health states are based on a large UK cohort study²⁷ of hospitalized patients with COVID-19 with a mean age of 72 years. For dexamethasone and tocilizumab, we assumed that only patients in the ICU received treatment with these drugs. This is in accordance with current treatment recommendations.²² During hospitalization, patients either remained in their respective recovered states or died of COVID-19 or other causes.²⁸ The maximum hospitalization duration was 73 days.²⁷ After hospitalization, patients were followed over their lifetime.

The model was developed in the statistical programming language R based on the Decision Analysis in R for Technologies in Health framework.29, 30, 31 We followed the Consolidated Health Economic Evaluation Reporting Standards³² and ISPOR³³ reporting and analysis recommendations. We validated the model by performing internal validation using the observational cohort data²⁷ and replicating the model’s cohort transitions in the decision-analytical software Amua.³⁴

Costs and Effects

Health outcomes in the model were expressed as life-years (LYs) and quality-adjusted LYs (QALYs). Costs were estimated in 2020 US dollars ($).³⁵ The analyses were performed from a US healthcare perspective. Costs during hospitalization included daily costs per person in a hospital ward or in the ICU and depended on the estimated mean length of hospital stay based on trial data. Treatment costs were based on the price proposed by the manufacturer or public pharmacy databases. After hospitalization, patients in the ICU recovery state accrued a one-time rehabilitation cost based on treatment needs as estimated by the Dutch National Health Authority guidelines³⁶ and converted to US prices for equivalent services. Recovered patients incurred mean healthcare costs for US citizens according to their age group annually until their death.³⁷ A review of 59 novel therapeutic drugs informed the costs for additional research, where the fixed cost estimate of trials up to 26 weeks was adjusted pro rata to represent a shorter trial duration of 3 months.^{38 , 39} We applied a 3% annual discount rate for both costs and effects⁴⁰ and a $100 000 willingness-to-pay (WTP) threshold.41, 42, 43

Uncertainty Analyses and VOI

To assess the benefit of conducting further research, VOI analysis quantifies the opportunity cost of suboptimal decisions because of uncertainty. It takes a random sample of the value of each model parameter (Appendix Table 1 and Supplemental Excel file in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.03.016) and evaluates the resulting outcomes to determine the optimal strategy (treatment or usual care) for that iteration (probabilistic analysis [PA]). We performed a PA and calculated the expected value of each strategy for each of the 10 000 iterations. Input parameter distributions were lognormal (treatment effects), beta (utilities, transition probabilities), uniform, or triangular (costs or when distributions were not available from data sources).²⁶ Detailed information on all parameters is included in the Supplemental Excel file found at https://doi.org/10.1016/j.jval.2022.03.016.

With perfect knowledge of the parameter values chosen from their distributions for each of the 10 000 iterations of the PA, we determined the optimal strategy and its expected value for each iteration. Averaging these yielded the average expected value of the 10 000 decisions made with perfect information. Next, we calculated the difference between the average expected value of the 10 000 decisions made with perfect information and the decision based on the average expected value of each of the strategies, that is, the decision made with current information. This difference yielded the loss in expected value due to suboptimal decisions as a result of parameter uncertainty, also known as the expected value of perfect information. We expressed the VOI results on a single scale of net monetary benefit by converting QALYs to a monetary amount by multiplying these QALYs by a societal WTP, for example, $100 000 per QALY gained, and subtracting the resource costs.

Similarly, we calculated the expected added value of performing an RCT to reduce only the uncertainty surrounding treatment-related decrease in mortality as partial perfect information (EVPPI). For drugs with identified potential positive value of further research (EVPPI > 0), we determined the value of collecting additional information on treatment efficacy with a trial of finite sample size (expected value of sample information [EVSI]). We performed an EVPPI estimation with a linear-regression meta-model and EVSI with a Gaussian approximation approach as proposed by Jalal and Alarid-Escudero.44, 45, 46 In this approach, the opportunity loss from a suboptimal decision is approximated by a linear relation of the parameters of interest.⁴³

This process was followed by a Gaussian approximation that simplifies the traditional Bayesian approach by computing the posterior mean for each of the parameters of interest (ie, treatment efficacy in our analysis).⁴⁴ The approximation allows for multiple correlated parameters and parameters by different sample sizes and a wide range of univariate and multivariate non-Gaussian distributions and is computationally substantially more efficient than traditional Bayesian updating in EVSI.

Next, we calculated the cost of further trials based on fixed and variable cost across sample sizes. Given that it is impossible to obtain perfect information, VOI places an upper bound on the cost of additional research aimed at reducing uncertainty.¹⁶ The optimal sample size of a new RCT was calculated as the size at which the net value of the optimal overall strategy is highest. Given reported concerns of insufficient trial enrollment,⁵ we evaluated both the optimal and a maximum feasible sample size of 2500 patients (reported in Appendix section 1.4 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.03.016). Net benefit obtained with emergency use authorization of treatments while performing further RCTs was determined for the expected number of patients to be hospitalized in the USA while awaiting trial results and their implementation (current patients) over 3 months. The expected VOI was extrapolated to the patient population that could benefit from new trial results; that is, the number of patients expected to be hospitalized after the trial results is available (future patients). The number of patients was calculated as the sum of the number of daily hospitalizations forecasted by the Institute of Health Metrics and Evaluation as of November 1, 2021, until March 1, 2022.⁴⁷

The net value for each strategy was calculated according to the equations in Figure 1A. Potential strategies included were reject (left lower quadrant), approve (right lower quadrant), approve with research (AWR, right upper quadrant), or use the drug OIR settings (left upper quadrant).⁴ Rejection without further research (left lower quadrant) was considered the reference (“default”) strategy (net value = 0). We assumed both the AWR and approve strategies irrecoverable implementation and reversal costs⁴⁸ to be $0 given that treatment protocols for COVID-19 are continuously and expeditiously updated, which does not require major investments, and no fixed capital investments are required for these treatments.

In sensitivity analyses, we investigated main drivers of the results by testing extreme values and consequences of underlying modeling assumptions. In addition to the analyses in the article assuming a WTP threshold of $100 000, the Appendix in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.03.016 provides the cost-effectiveness acceptability curves for each drug and in addition illustrates how the EVPPI results depend on the WTP threshold.

Ethics Approval

Medical ethical review board approval was not required because we performed mathematical modeling and simulation using published data. No data from human participants were collected in this study

Results

Cost-Effectiveness Analysis

Key findings for each treatment strategy are summarized in Table 2 . Incremental costs and effectiveness (in QALYs) per individual are shown in Figure 2 .

Table 2.

Summary results from our analysis.

Item	Hydroxychloroquine	Remdesivir	Casirivimab-imdevimab	Dexamethasone	Baricitinib-remdesivir	Tocilizumab	Interferon beta-1a	Lopinavir-ritonavir
Is treatment cost-effective?	No∗	Yes†	Yes‡	Yes‡	Yes‡	Yes‡	No∗	No∗
Incremental costs ($)	−12 227	−5	696	6856	10 673	35 849	−2538	−1404
Incremental QALYs	−0.263	0.252	0.171	0.614	0.775	0.882	−0.472	−0.091
ICER ($/QALY)	46 427	n/a	4075	11 169	13 772	40 633	5377	15 418
Incremental net monetary benefit ($) (thousand)	−14	25	16	55	67	52	−45	−8
Incremental net health benefit (QALY)	−0.141	0.252	0.164	0.545	0.668	0.524	−0.447	−0.077
EVPPI ($) (million)	375	127	0	0	0	1.3	0	0
Current patients (thousand)	598	598	598	99	598	99	598	598
Future patients (thousand)	220	220	220	36	220	36	220	220
Optimal strategy	OIR	Approve	Approve	Approve	Approve	Approve	Reject	Reject
Net value ($) (million)	198	20 645	13 389	7358	54 642	7069	0	0

Note. Results shown are the mean results from the probabilistic analysis, calculated as the treatment arm versus the care-as-usual arm of each trial at a WTP of $100 000/QALY and the results of the value of information analysis. Future/current patients are based on all expected hospitalized patients (hydroxychloroquine, remdesivir, casirivimab-imdevimab, baricitinib-remdesivir, interferon β-1a, lopinavir-ritonavir) or ICU patients only (dexamethasone, tocilizumab).

EVPPI indicates partial perfect information; ICER, incremental cost-effectiveness ratio; ICU, intensive care unit; n/a, ICER not applicable because of dominance; OIR, only in research; QALY, quality-adjusted life-year; WTP, willingness to pay.

^∗Treatment is cost-saving, but not enough that ICER > WTP (ie, treatment is not decrementally cost-effective).

^†Treatment is dominant.

^‡Treatment is effective and ICER < WTP.

Figure 2.

Incremental cost-effectiveness plane for mean estimates per individual resulting from the PA. Incremental costs in USD and effects are calculated and shown as the treatment group versus the control group within the respective trial, and not in comparison with the other treatments projected. The right side of the WTP threshold line represents cost-effectiveness.

PA indicates probabilistic analysis; USD, US dollar; WTP, willingness to pay.

Our PA results indicate a decreased mortality and increased quality of life (QOL) for baricitinib-remdesivir, dexamethasone, remdesivir, and tocilizumab, whereas hydroxychloroquine, interferon beta-1a, and lopinavir-ritonavir are associated with increased mortality and decreased QOL. Over a lifetime, the average QALY gains per patient were 0.775 (uncertainty interval −0.192 to 1.670) with baricitinib-remdesivir, 0.614 (0.039-1.258) with dexamethasone, 0.252 (−0.141 to 0.670) with remdesivir, 0.171 (−0.082 to 0.440) with casirivimab-imdevimab, and 0.882 (−0.052 to 1.937) for tocilizumab, at an incremental cost of $10 673 (−$3930 to $24 372) for baricitinib-remdesivir, $6856 (−$19 696 to $33 723) for dexamethasone, $695 (−$33 082 to $36 541) for casirivimab-imdevimab, and $35 849 ($20 447-$52 175) for ftocilizumab and with marginal cost savings of −$5 (−$33 318 to $235 724) for remdesivir, making remdesivir dominant, with higher effects and lower costs than usual care. Higher costs for tocilizumab were mainly driven by treatment costs, whereas baricitinib-remdesivir, dexamethasone, and casirivimab-imdevimab costs were driven by the high healthcare costs during remaining LYs in surviving patients. Conversely, lower costs for hydroxychloroquine (−$12 227 [−$32 725 to $7344]), interferon beta-1a (−$2538 [−$14 453 to $8622]), and lopinavir-ritonavir (−$1404 [−$40 584 to $37 062]) were due to decreased survival leading to reduced future healthcare costs.

At a WTP of $100 000/QALY, positive incremental net monetary benefits were found for baricitinib-remdesivir ($66 826 [−$15.895 to $144 126]), dexamethasone ($54 526 [−$10 111 to $120 427]), remdesivir ($25 249 [−$23 881 to $73 206]), casirivimab-imdevimab ($16 375 [−$25 404 to $57 641]), and tocilizumab ($52 378 [−$30 049 to $149 555]), consistent with being cost-effective. The remaining strategies—hydroxychloroquine, interferon beta-1a, and lopinavir-ritonavir—were not cost-effective. The expected values as presented earlier should be used to identify the optimal strategy.⁴⁹ The uncertainty intervals reflect the range of decision uncertainty, and the consequence of this uncertainty should be assessed in VOI analysis.

These PA results, uncertainty intervals, cost-effectiveness planes, and cost-effectiveness acceptability curves for each treatment strategy are provided in the Appendix Supplemental graphs and tables (Appendix Supplemental graphs and Tables 1.2/1.3/2.1-2.3/3.1-3.3/4.1/4.3/5.1-5.3/6.1-6.3/ 7.1-7.3/8.1-8.3/9.1-9.3 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.03.016).

VOI and Optimal Overall Strategy

The population EVPPI was only positive for hydroxychloroquine ($375 million), remdesivir ($127 million), and tocilizumab ($1.4 million) (Table 2, Appendix Supplemental graphs and Tables 1.4/1.5/2.4/3.4/4.4/5.4/6.4/7.4/8.4/9.4 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.03.016), suggesting further RCTs to determine treatment effect more precisely may be worthwhile. Conversely, high certainty surrounding the treatment effect of dexamethasone and baricitinib-remdesivir and the absence of benefit of lopinavir-ritonavir and interferon beta-1a suggest that performing more RCTs will not affect the decision about the use of these treatments in clinical practice (Table 2). This indicates that implementing dexamethasone ($7.4 billion), casirivimab-imdevimab ($13.4 billion), and baricitinib-remdesivir ($54.6 billion) and rejecting lopinavir-ritonavir ($0) and interferon beta-1a ($0) without further trials were implementation strategies with the highest overall value.

To decide whether further research is warranted for hydroxychloroquine, remdesivir, and tocilizumab, the net value of the relevant strategies (Fig. 1B: upper quadrants) was calculated for an optimal trial sample size and then compared with reject or approve as appropriate. The optimal overall strategy for hydroxychloroquine, OIR (Fig. 1B: left upper quadrants, OIR), had an estimated net value of $198 million with an optimal sample size of 4800 patients. For the preset maximum feasible sample size of 2500 patients, this net value is reduced to $174 million. The value of this further research would stem from the investigation of decremental cost-effectiveness, answering the question of whether the cost savings justify the QALYs lost, which has ethical implications.

For remdesivir and tocilizumab, the potential benefit of further trials did not outweigh the costs of research, and the highest net value ($21 billion and $7 billion, respectively) was obtained with approval (Fig. 1B: right lower quadrant, Approve). Our findings suggest that none of the investigated treatments should be granted emergency use authorization (right upper quadrant, AWR).

Sensitivity Analyses

The cost-effectiveness acceptability curves illustrating the cost-effectiveness across WTP thresholds and tables displaying the EVPPI for different WTP thresholds are provided in the Appendix for all treatments (Appendix Supplemental graphs and Tables 2.3/2.4/3.3/3.4/4.3/4.4/5.3/5.4/6.3/6.4/7.3/7.4/8.3/8.4/9.3/9.4 in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.03.016).

The WTP thresholds at which the therapy would be cost-effective was $0 for remdesivir, $10 000 for casirivimab-imdevimab, $20 000 for dexamethasone, $20 000 for baricitinib-remdesivir, and $40 000 for tocilizumab. The WTP threshold at which usual care would be cost-effective was $20 000 for lopinavir-ritonavir, $10 000 for interferon beta-1a, and $50 000 for hydroxychloroquine. At lower WTP thresholds, these treatments would be decrementally cost-effective, because they would save costs through reduced long-term healthcare expenditures because of reduced survival.

Discussion

Summary of Findings

Our results illustrate how VOI can inform policy and practice amidst a pandemic when considering whether to approve therapies, permit emergency use authorization, perform additional research, or simply reject potential therapies in the treatment of hospitalized patients with COVID-19. As of November 2021, our results indicate that, at a WTP of $100 000, treatment with remdesivir, casirivimab-imdevimab, dexamethasone, baricitinib-remdesivir, and tocilizumab leads to positive mean incremental net benefit compared with care as usual, whereas treatment with hydroxychloroquine, lopinavir-ritonavir, and interferon beta-1a does not. Additionally, our results suggest sufficient certainty that decisions about treating patients with dexamethasone, casirivimab-imdevimab, baricitinib-remdesivir, lopinavir-ritonavir, and interferon beta-1a would not change with further RCTs. Further research could needlessly consume resources, expose trial participants to avoidable risks, and preclude them from receiving alternative (potentially) effective treatments. For remdesivir and tocilizumab, the net value of further trials did not outweigh the cost of research, making immediate approval their optimal overall strategies. The net value of further trials for hydroxychloroquine outweighed the cost of research, and therefore, the highest net value for this drug was found in the OIR strategy. Nevertheless, this further research would be conducted to investigate decremental cost-effectiveness (saving costs due to reduced survival), the ethical implications of which should be considered (see Table 3 ^{3 , 8 , 20 ,} 22, 25 ^{, 37 ,} 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71).

Table 3.

Policy implications and discussion per treatment.

Treatment	Overall strategy	Current status	Reflections and other considerations
Hydroxychloroquine	OIR (highest net value) Reject (ethical consideration)	The FDA initially granted it emergency use authorization but later revoked that designation after further scientific data.50 Nevertheless, there are countries that continue to recommend its prescription51,52 based on early studies that later received heavy criticism.53 The WHO announced in June 2020 that the hydroxychloroquine arm of the SOLIDARITY trial was stopped, based on findings from the RECOVERY and SOLIDARITY trials. At least 27 trials investigating hydroxychloroquine have been prematurely terminated and 12 withdrawn before enrollment,3 and some countries have banned the drug in the treatment of COVID-19.54 As of November 2021, still > 100 trials were ongoing or planned to investigate (hydroxy)chloroquine with perhaps over 100 000 participants involving all severity levels (including prophylaxis).3	Our findings support the conclusion that hydroxychloroquine should be rejected, yet they also suggest that there is expected positive net value in investigating hydroxychloroquine in trials (OIR). Nevertheless, the expected value from further trials stems from the drug being close to decrementally cost-effective55: it saves costs through reduced long-term healthcare expenditures because of decreased survival. On average, a healthcare cost of $12 101 per year would be “saved” by premature death.37 Performing further trials to demonstrate cost savings because of reduced survival raises ethical questions to be considered.
Lopinavir-ritonavir	Reject	Treatment guidelines have recommended against the use of lopinavir-ritonavir,25 after the drug showed no effect in hospitalized patients. This recommendation is in line with the findings in our analysis. According to the NMA initiative, 1 trial was reported terminated, 1 withdrawn, and 1 suspended.3 Nevertheless, 44 trials continue to recruit patients.3	Our VOI found the highest net value in the rejection strategy. Conducting further trials may expose patients to unnecessary harms and prevents them from receiving potentially effective treatments.
Interferon beta-1a	Reject	For interferon β-1a some improvements of clinical aspects of COVID-19 have been identified56; nevertheless, when administered in later stages of infection, the drug exacerbates the disease severity because of excessive inflammation and tissue damage56,57; 3 trials were registered as terminated, 1 as withdrawn, 1 as suspended, and 54 continue recruiting.3	Our VOI found the highest net value in the rejection strategy. Conducting further trials may expose patients to unnecessary harms and prevents them from receiving potentially effective treatments.
Remdesivir	Approve	In early findings of remdesivir in the ACTT-I trial, policy makers and regulatory agencies concluded that, given the urgent need for COVID-19 treatment, the 4-day reduction in recovery time was a satisfactory proxy of the drug’s effectiveness. Therefore, the FDA issued emergency use authorization conditional on further research investigating its impact on mortality.58 There are 29 trials investigating remdesivir still recruiting patients as of November 2021.3 In September 2021, a fifth large RCT, the DisCoVeRy trial59 was published. The results have not yet been integrated into the major meta-analyses that we used in our analysis, but this trial also demonstrated a small but statistically nonsignificant beneficial effect of remdesivir on the secondary outcome of mortality. Gyselink et al60 have suggested that differences in findings across trials may be partially explained by the different levels of use of systemic steroids.	Although we found persistent uncertainty surrounding the effectiveness of remdesivir, our model found that the benefits of widespread immediate implementation outweighed the net value of further research. This conclusion is in contrast to the WHO’s recommendation to not treat with remdesivir and to continue to recruit patients. Although both our and the WHO panels’ conclusions are based on the same meta-analysis,20 our findings differ because our model accounts for a lifetime horizon and the potential life-years and QALYs lost, rather than statistical significance of treatment effect during the trial timeframe alone.8 Further analyses that will include latest trials such as the DisCoVeRy trial59 analyze specific subgroups and that building on network meta analyses61 may help to identify the drivers of the different results for remdesivir trials over time.
Baricitinib-remdesivir	Approve	Baricitinib is an FDA-approved treatment for rheumatoid arthritis. In November 2020, an emergency use authorization was issued for the use of baricitinib in combination with remdesivir for hospitalized patients with COVID-19.62 This guidance is based on the ACTT-2 results.23 Nevertheless, the ACTT-2 trial has been criticized for its inability to evaluate the effect of baricitinib in addition to corticosteroids.62 The NIH panel recommended the use of either baricitinib or tocilizumab in combination with dexamethasone alone or dexamethasone plus remdesivir.62 Their recommendations are based on the results of the COV-BARRIER study published in September 2021.62 This study compares baricitinib alone with usual care (including corticosteroids and antivirals). As of November 2021, 11 trials are registered as recruiting patients.3	In our model, baricitinib-remdesivir is compared with remdesivir as usual care, as per the ACTT-2 results.23 This is made explicit, given that all patients from the usual care group received remdesivir, whereas in other trials some did and some did not. Therefore, incremental costs and acquired QALYs should be interpreted as incremental to remdesivir alone.
Tocilizumab	Approve	The treatment effect found in meta-analyses63 for tocilizumab relied heavily on the population that was treated. Tocilizumab’s clinical implementation has been mainly based on the results from the REMAP-CAP24 and RECOVERY63 trials and is recommended only for severe patients already receiving dexamethasone.64 As of November 2021, 28 trials in the WHO register are recruiting patients.3	In our simulation, we use the treatment effect evidence from REMAP-CAP that was solely applied to ICU patients. This is in contrast to the RECOVERY63 meta-analysis, which included patients ranging from nonsevere (EMPACTA65) to severe, given that this reflects clinical practice.
Dexamethasone	Approve	Dexamethasone did not require emergency use authorization, because it is a drug currently in use for patients who require respiratory support.66 In patients with severe infections, the drug can prevent lung injury caused by community-acquired pneumonia by suppressing exuberant systematic inflammation.69 In nonsevere COVID-19, the use of corticosteroids is not recommended.67 There are 29 trials that continue to recruit patients to investigate dexamethasone.3	Similarly to tocilizumab, dexamethasone22 is only given to the ICU population in our simulation, in line with the treatment recommendations68
Casirivimab-imdevimab	Approve	As of September 2021, the WHO issued a conditional recommendation for the use of casirivimab and imdevimab for only patients with seronegative status.70 This recommendation is based on the statistically significant reduction of mortality in the seronegative subgroup but not the seropositive subgroup. There are 10 trials recruiting patients to investigate casirivimab-imdevimab.	Our current analysis included the overall effect of both seropositive and seronegative patients because our underlying population and transition probabilities represented a mixed serostatus group. Performing subgroup analysis while applying the same transition probabilities to both groups is in our opinion not justified because it would bias the results.71 Future analyses investigating subgroup-specific effects with varying control group characteristics should take serostatus into account that will require the collection of subgroup-specific prognostic data.

FDA indicates Food and Drug Administration; ICU, intensive care unit; NIH, National Institutes of Health; NMA, National Medical Association; OIR, only in research; QALY, quality-adjusted life-year; RCT, randomized controlled trial; VOI, value of information; WHO, World Health Organization.

Policy Implications

As further trials unfold, the allocation of drugs in specific strategies should be considered as nonstatic. Food and Drug Administration emergency use authorization has been granted for remdesivir, casirivimab-imdevimab, and baricitinib-remdesivir.^{6 , 72} For hospitalized patients with COVID-19, the Infectious Diseases Society of America’s guidelines recommend against the use of hydroxychloroquine and lopinavir-ritonavir and suggest the use of tocilizumab, remdesivir, and baricitinib-remdesivir. Our findings support the Infectious Diseases Society of America’s guidelines for all investigated treatments. A summary of policy implications for each of the included treatments is provided in Table 3 . ^{3 , 8 , 20 ,} 22, 23, 24, 25 ^{, 37 ,} 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71

Although cost-effectiveness is an important tool to inform drug approval policy, there is little consensus on the WTP threshold,⁷³ let alone during a pandemic. A potential point of discussion is whether a WTP threshold should be higher in pandemic times due to the emergency status or lower because of considerations of affordability when a large number of people are treated. In their assessment of the cost-effectiveness of remdesivir, the Institute for Clinical and Economic Review applied a $50 000 WTP threshold, stating their belief that this threshold is more likely to be policy relevant for consideration of treatments in public health emergencies.⁷⁴ A potential additional consideration when treatment is cost-effective but raises concerns on affordability is to consider the health opportunity costs of overall budget impact of the approval of new drugs.^{75 , 76} The United States has no specified WTP threshold or a single, defined budget for healthcare spending.⁷³ Historically, US-based cost-effectiveness studies have considered incremental cost-effectiveness ratios at thresholds ranging from $50 000 to $300 000 per QALY.^{43 , 73 , 76} Our assumption of a WTP of $100 000 follows recent criticisms in health economics research that a WTP of $50 000 would be relatively low on the basis of increases in healthcare spending and increased per capita annual income.41, 42, 43 ^{, 73} Still, in the Appendix in Supplemental Materials found at https://doi.org/10.1016/j.jval.2022.03.016, we provide the cost-effectiveness acceptability curves across a wide range of WTP thresholds for all investigated treatments to inform the decision-making process. These results show that, at a WTP of $50 000, the conclusions of which drugs were cost-effective (remdesivir, casirivimab-imdevimab, dexamethasone, baricitinib-remdesivir, and tocilizumab) would remain the same.

Clinical Practice

For clinicians, these findings have implications for both prescription practice and participation in further trials. First, our evidence-based approach can identify treatments under investigation that may not only be ineffective but also harmful; for example, hydroxychloroquine is still widely prescribed based on personal beliefs or experience.⁷⁷ With rapidly expanding evidence in the pandemic, it is paramount that clinical guidelines and practice are continuously updated to reverse previously approved therapies when appropriate so that clinicians and patients can be informed about potential benefits and harms of treatments. Furthermore, VOI summarizes existing evidence and can inform clinicians about the potential added information value, costs of further research, and ethical implications that come with it. Performing VOI can aid decision making for policy makers, researchers, clinicians, and patients on whether to initiate or continue enrollment in drug trials, stop such trials, or simply approve and implement.

Strengths and Limitations

To the best of our knowledge, our study is the first to perform a VOI analysis for the treatment of patients with COVID-19. Although trials and meta-analyses often conclude further research is needed⁷ and where current guidelines are based on statistical significance in clinical evidence, VOI results are based on both uncertainty and the potential consequences of making decisions with and without further evidence. Our article expands the potential impact of drug approval by investigating not only drug efficacy and effectiveness but also the overall net benefits. Considering the unprecedented rollout of clinical trials investigating potential treatments for COVID-19, objective research prioritization seems paramount.^{3 , 5} Our model can be updated with further evidence from trials and (cumulative) meta-analyses as they become available to continuously evaluate the optimal overall strategy as the pandemic evolves.

Input parameters were based on best-available evidence. Some parameters, such as the QOL of patients with COVID-19 in 5 years and the costs of research, had to be estimated based on previous studies considering other diseases. Where necessary, we chose conservative approaches to our model parameters and settings. For example, treatment effects were only applied for reported trial duration and not extrapolated beyond and wide distributions were chosen to represent large uncertainty. Additionally, we calculated the net value for the US population alone, whereas globally more patients would benefit from determining the optimal overall strategy.

Nevertheless, unidentified bias in studies may have affected our results. Previous articles have discussed the potential disagreements between meta-analyses and large trials.^{78 , 79} Single trials may not consider heterogeneity that is likely to exist across trials and centers. Nevertheless, an advantage of single trials is the more detailed group-specific information. In the case of tocilizumab, we explicitly decided not to use the results of a meta-analysis because of the clinically relevant differences between hospital ward and ICU patients. Large network meta-analyses, such as the living World Health Organization guideline on COVID-19 treatments,⁶⁷ do not currently provide sufficient details to enable distinctions in treatment effect, and this needs to be considered in future analyses.

With respect to the methods, our article builds on the work by McKenna and Claxton by calculating net value and not only considers the decision for conducting further research versus implementing treatment options,^{7 ,} 80, 81, 82 but also accounts for the effect of immediate implementation on patients who would have “missed out” on treatment while awaiting trial results.⁸² The effect of the strategy decision on current patients is especially important given the pace of hospitalizations during the pandemic. The foregone benefit of delaying approval while awaiting further trial results could be exceptionally high. Although our article focuses on COVID-19, creating an infrastructure to investigate rapidly emerging existing trials and value of additional trials in real-time using evidence synthesis and VOI models in potential future pandemics could form the basis for informing clinical practice, research, and policy decisions going forward.

The results of any VOI depend on the underlying choices and assumptions.⁸³ Therefore, the limitations of our analysis need to be considered. We estimated the EVPPI with a linear-regression meta-model and EVSI with a Gaussian approximation approach as proposed by Jalal and Alarid-Escudero.^{44 , 45} Nevertheless, this is one of several existing estimation methods,^{44 , 45} which differ in approaches but none of which has shown computational or statistical superiority.⁸⁴ The advantage of the chosen approximation method is the computational efficiency without introducing substantial bias.⁴⁶ A potential limitation of the linear-regression meta-model is that this normal approximation of the previous and preposterior distributions for parameters of interest in EVPPI and EVSI computations may introduce bias if severely non-normal.⁴⁶ Given that our analysis contains parameters with sufficiently large sample sizes to approach normal distributions as per the central limit theorem, we did not consider this a cause for an introduction of bias in our analysis. Finally, the current EVSI estimation methods do not consider structural uncertainty,⁴⁵ meaning that even if the true values for all input parameters are known, we are still not certain that the model reflects reality.

The key uncertainty investigated in this analysis is the estimated precision of the treatment effectiveness, because this uncertainty would be reduced by acquiring additional RCT data. Other uncertainties that could be addressed with further research include long-term morbidity and mortality, heterogeneity, adverse events of treatment, recovery time, QOL after recovery from COVID-19, the number of hospitalizations, and costs. Furthermore, our models’ findings are not appropriate for comparing investigated treatments with each other. Our analysis does not aim to prioritize drug treatment by comparing active treatments with each other but rather focuses on the research and approval health policy questions of each studied drug regimen. Through different mechanisms and when applied in different contexts, these treatments may be useful in sequence or in combination. A head-to-head comparison of treatments based on currently available evidence would require the assumption of independent effects and comparability of study populations, which would likely strongly bias the results. Key differences in considered trials include patient populations, for example, only ICU patients for dexamethasone and tocilizumab; the duration for which the treatment effect is applied; and evolving usual care as the pandemic unfolded. For example, when remdesivir and dexamethasone became incorporated in usual care, patients in both treatment and control arms in subsequent trials received these drugs, and accordingly, our model investigated the incremental effect of the newly introduced treatment. Future network meta-analyses that identify head-to-head treatment-specific effects could provide new input for the model to help distill comparative cost-effectiveness.

Our model is based on several assumptions. The unavailability of appropriate US cohort data necessitated the assumption that the large cohort of UK patients was sufficiently representative. It is likely that care for patients with COVID-19 has improved since May 2020. We additionally made assumptions on the utilities of patients recovered from the ICU and hospital ward over their lifetime. Projections of hospitalized patients⁴⁷ may also be altered because of the rollout of vaccinations, new virus variants, (reversal of) lockdown measures, or the effect of treatment on COVID-19 transmission.

Our model did not account for age-, sex-, serostatus-, or comorbidity-specific treatment effects and investigated reductions in mortality, but not severity of disease. Our analysis also assumed treatments were available to all patients in our simulated cohort. This assumption ignores a potential shortage of treatments in certain areas or to specific patient groups when there is a need for prioritization of resources. Nevertheless, the analysis could be repeated to investigate subgroup-specific costs, effects, and strategy recommendations for specific subgroups of interest.

Conclusions

Our study demonstrates that using hydroxychloroquine OIR; approval and implementation of remdesivir, casirivimab-imdevimab, dexamethasone, baricitinib-remdesivir, and tocilizumab; and the rejection of lopinavir-ritonavir and interferon beta-1a provide the highest net value per November 2021. In the case of hydroxychloroquine, this highest net value arises from decremental-cost-effectiveness, and investigating hydroxychloroquine further may be infeasible for ethical reasons. Performing ongoing VOI analyses using updated research results during the pandemic can help define the optimal moment to implement emerging therapies and whether further clinical trials are justified.

Article and Author Information

Author Contributions:Concept and design: Dijk, Krijkamp, Kunst, Hunink

Acquisition of data: Dijk, Krijkamp

Analysis and interpretation of data: Dijk, Krijkamp, Kunst, Gross, Wong, Hunink

Drafting of the manuscript: Dijk, Krijkamp, Gross, Wong, Hunink

Critical revision of the paper for important intellectual content: Dijk, Krijkamp, Kunst, Gross, Wong, Hunink

Statistical analysis: Dijk, Krijkamp, Kunst

Obtainingfunding: Dijk, Krijkamp, Kunst, Hunink

Supervision: Hunink

Conflict of Interest Disclosures: Dr Dijk reported receiving grants from the Gordon and Betty Moore Foundation during the conduct of the study and reported receiving grants from German Innovation Fund outside the submitted work. Ms Krijkamp reported receiving grants and personal fees from the Society for Medical Decision Making fellowship through a grant by the Gordon and Betty Moore Foundation (GBMF7853) outside the submitted work. Dr Gross reported receiving grants from Johnson & Johnson, the National Comprehensive Cancer Network Foundation (Pfizer/AstraZeneca), and Genentech outside the submitted work. Dr Hunink reported receiving grants from the Gordon and Betty Moore Foundation during the conduct of the study, reported receiving reimbursement for travel expenses from European Society of Radiology and from the European Institute for Biomedical Imaging Research, reported receiving royalties for a textbook on medical decision making from Cambridge University Press, and reported receiving grants from the American Diabetes Association, The Netherlands Organization for Health Research and Development, the German Innovation Fund, and The Netherlands Educational Grant (“Studie Voorschot Middelen”) outside the submitted work. No other disclosures were reported.

Funding/Support: This research was funded by the Gordon and Betty Moore Foundation through grant GBMF9634 to Johns Hopkins University to support the work of the Society for Medical Decision Making COVID-19 Decision Modeling Initiative.

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.