Coronavirus Disease 2019: Considerations for Health Technology Assessment From the National Centre for Pharmacoeconomics Review Group

Abstract

It is expected that the coronavirus disease 2019 (COVID-19) pandemic will leave large deficits in the budgets of many jurisdictions. Funding for other treatments, in particular new treatments, may become more constrained than previously expected. Therefore, a robust health technology assessment (HTA) system is vital. Many clinical trials carried out during the pandemic may have been temporarily halted, while others may have had to change their protocols. Even trials that continue as normal may experience external changes as other aspects of the healthcare service may not be available to the patients in the trial, or the patients themselves may contract COVID-19. Consequently, many limitations are likely to arise in the provision of robust HTAs, which could have profound consequences on the availability of new treatments. Therefore, the National Centre for Pharmacoeconomics Review Group wishes to discuss these issues and make recommendations for applicants submitting to HTA agencies, in ample time for these HTAs to be prepared and assessed. We discuss how the pandemic may affect the estimation of the treatment effect, costs, life-years, utilities, discontinuation rates, and methods of evidence synthesis and extrapolation. In particular, we note that trials conducted during the pandemic will be subject to a higher degree of uncertainty than before. It is vital that applicants clearly identify any parameters that may be affected by the pandemic. These parameters will require considerably more scenario and sensitivity analyses to account for this increase in uncertainty.

Highlights

• •The coronavirus disease 2019 pandemic may affect many ongoing clinical trials and leave large deficits in the budgets of many jurisdictions.
• •This article discusses potential difficulties with data from these trials, such as excess deaths; issues with extrapolation and evidence synthesis; and lack of generalizability of cost, utility, and discontinuation data.
• •We conclude with recommendations for those submitting health technology assessments.

Introduction

Consequences secondary to the coronavirus disease 2019 (COVID-19) pandemic will be seen in the healthcare service for many years. The immediate interventions required for the COVID-19 pandemic may leave large deficits in the budgets of many jurisdictions. Therefore, funds for other treatments, in particular new treatments, may become more constrained than previously expected. Hence, a robust health technology assessment (HTA) system is vital. The National Centre for Pharmacoeconomics (NCPE) in Ireland is a national HTA agency, responsible for assessing the cost-effectiveness and budget impact of new medicines submitted by applicant pharmaceutical companies (herein the applicant) for potential reimbursement by the state health payer (Health Service Executive).

Unfortunately, owing to the pandemic, many limitations are likely to arise in the provision of robust HTAs, increasing the time taken to develop and assess cost-effectiveness and budget impact models. This may also increase the uncertainty in the outputs from these models and lead to delayed and less certain recommendations being communicated to the decision maker, which could have profound consequences on the availability of new treatments. Both manufacturers and decision makers face more complex and uncertain HTA assessments, compounded by budget deficits, which may affect willingness-to-pay thresholds. For example, historically the cost-effectiveness threshold in Ireland has varied between €20 000 and €45 000 per quality-adjusted life-year.¹ Currently, treatments below the upper threshold are considered cost-effective; however, it is unknown if that may change as the healthcare budget becomes more constrained. Therefore, it is imperative that these issues are discussed now and planned for accordingly with ample time for these HTAs to be prepared and assessed.

Among others, the European Medicines Agency² and the U.S. Food and Drug Administration³ have developed guidance for handling deviations to the trial protocol. Cro et al⁴ and Meyer et al⁵ also discuss statistical issues and missing data arising in trials conducted during the COVID-19 pandemic. Nevertheless, there are additional issues that will affect HTA, over and above those considered by regulators. Clinical trial investigators and regulators use statistical models to infer treatment effects. HTA models use this inference and build on it to predict patient outcomes, costs, and quality of life. Lorgelly and Adler⁶ identified a number of issues regarding the impact of a pandemic on HTA. We wish to further discuss what we consider to be the most pertinent concerns, identify additional issues relating to the HTA process, and make some recommendations for applicants.

Treatment Effect

Uncertainty in estimated treatment effects, deemed tolerable for marketing authorization, may be more consequential to a later reimbursement decision. In some respects, regulatory agencies develop guidance with the intent of ensuring that there is at least some treatment benefit, whereas recommendations put forward by HTA agencies are dependent on the magnitude of said benefit. Methods used to assess or correct potential biases and uncertainty in clinical trial data for the purposes of determining safety and efficacy may not be appropriate for building cost-effectiveness and budget impact models.

Trial Arms May Be Affected Differently

It might be argued that any treatment effect observed in a randomized controlled trial should not be affected by COVID-19 because both arms will be affected equally. Nevertheless, this may not always be the case. For example, where a self-administered treatment is compared with a healthcare professional–administered treatment, the increased contact with healthcare professionals and medical environments potentially increases the COVID-19 risk in one arm.

Additionally, the sudden emergence of the pandemic may have a greater impact on one arm than the other if it occurred when the proportions of patients at risk differed between arms (eg, owing to increased survival).

Furthermore, owing to the nature of treatment, patients in one arm may be more vulnerable to COVID-19. For example, Gougis et al⁷ note various classes of immunosuppressive drugs indicated for the treatment of cancer(s). These patients may be more at risk of COVID-19 than patients who are receiving nonimmunosuppressive comparators. Additionally, Gougis et al highlight recognized interactions between drugs used for the treatment of cancer and antivirals now considered for COVID-19. The authors note a range of interactions and the potential resultant increased risk of severe toxicities or decreased efficacy associated with the combination of these treatments.

Single-Arm Studies

Similar to other HTA agencies, the number of HTAs informed by single-arm trials submitted to the NCPE has increased in recent years. Single-arm evidence is a well-documented source of significant uncertainty and is susceptible to bias.^{8 , 9} In particular, patients in single-arm trials may not be comparable to patients in different settings and time frames. The NCPE Review Group together with the Scottish Medicines Consortium have previously published guidance in this area.¹⁰ A pandemic highlights particular vulnerabilities with this type of evidence because the world in which the study was conducted may not be the same as the world going forward. This means that the outcomes observed in a trial conducted during the pandemic may not be the same as expected in the future. Nevertheless, any adjustment to the absolute effects in these trials should be approached with caution owing to the inherent high uncertainty in these trials. Table 1 details some recommendations for applicants when making a submission to the NCPE.

Table 1.

Recommendations to applicants when making a submission to the NCPE.

Issue	Recommendation	Cost-effectiveness model	Budget impact model
General	Clearly document protocol deviations or changes to statistical analysis as a result of COVID-19. Given the widespread impact of the pandemic, where no adjustments have been made, this should be stated within the submission.
	Methods for missing data should consider mechanisms causing missingness and present results under a range of plausible assumptions.
	More emphasis should be placed on structural uncertainty within the model. For example, probabilistic sensitivity analyses could account for additional data sources not affected by the pandemic by appropriate weighting.
Treatment effect/extrapolation	Present results of pre–COVID-19 data cut in a scenario analysis in addition to latest data cut.
	Any adjustments made to account for the impact of COVID-19 should be explored through sensitivity and scenario analyses.
Evidence synthesis	Perform sensitivity analyses to exclude trials affected by pandemic where other trials of similar quality are available.
	Investigate potential effect modifiers and differences in outcomes where trial results are available both before and during the pandemic.
Quality of life	Provide scenario analyses using non-affected sources of quality-of-life data.
	Consider including a covariate for COVID-19 in the statistical analysis of trial EQ-5D data.
Resource utilization/treatment discontinuation	Provide scenario analyses using non-affected sources of data.
System change/resource availability	Updated post-pandemic outbreak cost sources should be consulted.
	Model impact of potential delays in diagnosis and initiation of treatment on expected patient numbers, costs, and outcomes.
	Provide scenario analyses that assume constrained provision of resources.
	Analyze using varying willingness-to-pay thresholds, because threshold may be reduced in times of fiscal constraints.

COVID-19 indicates coronavirus disease 2019; NCPE, National Centre for Pharmacoeconomics.

Evidence Synthesis

Evidence synthesis allows for the comparison of treatments not studied in the same trial. This is a key area of concern in HTA, as relevant comparators are often not included in the pivotal trials, and evidence synthesis can introduce additional uncertainty within a cost-effectiveness model. Combining trials carried out during the pandemic with those carried out before or after the pandemic will require caution because the settings of these trials may be very different.

Estimation of Outcomes for HTA

Extrapolation

Generally, extrapolation of patient-level Kaplan-Meier curves is required to usefully incorporate survival data into cost-effectiveness models. This is often a key challenge. Now, health outcomes associated with COVID-19 could fundamentally change the shape of curves such as overall survival. A parametric distribution that fits the trial data well may not be appropriate under non-pandemic circumstances and hence may not be appropriate to use for extrapolation. Moreover, it may appear appropriate to split trial data into piecewise curves or to use mixture cure models, when this may be more a function of the pandemic than of the treatments under investigation. The risk of overfitting to COVID-19–related features is amplified where more flexible models are used.

As standard, outcomes such as disease progression will be assessed at regular predefined intervals. Although lag time in detecting progression (ie, interval censoring) is a limitation of many trials, this may be amplified during the pandemic. Assessments requiring attendance with healthcare providers may be delayed or avoided. Events recorded later than planned may affect many types of models. It may affect the calculation of transition probabilities in Markov models and the shape of fitted curves in partitioned survival models. This is concerning for cost-effectiveness models because biases in estimated treatment effects may be compounded when extrapolated beyond the trial time horizon.

Quality-of-Life Data

The pandemic is a source of significant distress and anxiety for many.¹¹ Economic concerns, predicted or real impacts on health outcomes, and restrictions on movement and social contact may negatively affect physical health and mental well-being. These considerations may affect health-related quality-of-life measures such as EQ-5D collected during the pandemic. This may limit the generalizability of utility values derived from these measures.

Excess Deaths

Certain subgroups of the population (including older people and people with particular underlying health conditions) are at higher risk of mortality and complications arising from COVID-19.¹² Therefore, overall survival may be affected in certain trials. Difficulties with access to testing, issues with testing accuracy, and the presence of comorbidities mean that it may be difficult to know which of these deaths can be attributed to COVID-19. Hence, both costs and life-years may be underestimated. In a hypothetical randomized controlled trial in which COVID-19 presents an increase in baseline hazard of death, the treatment effect of an intervention (measured as a hazard ratio) would be unchanged; however, the life-years gained would be reduced. We note also that general population mortality tables collected during the pandemic may be affected by excess deaths.

Estimation of Costs for HTA

Healthcare Resource Utilization Data

There is evidence that fewer patients are attending emergency departments or primary care providers, and nonurgent hospital appointments have been delayed or cancelled. For example, there was a 23% reduction in emergency department attendance in March 2020 versus March 2019 in England.¹³ Conversely, there may be an increase in costs and resource use if some patients in a trial develop COVID-19. Therefore, the generalizability of data on healthcare resource utilization collected during trials at this time may not be reflective of future patients’ activities.

Treatment Discontinuation

Participants in clinical trials may experience high rates of non-treatment-related discontinuation owing to the pandemic, which may not reflect future practice. Factors such as the potential for the treatment to cause immunosuppression and the requirement for hospital administration and follow-up may influence the decision to continue treatment. Consequently, time to treatment discontinuation may be subject to a high degree of uncertainty. Similarly, higher than anticipated rates of study withdrawal may affect the detection of statistically significant treatment effects or adverse events or may introduce attrition bias.

Healthcare System Change and Cost Uncertainties

The pandemic has disrupted the normal delivery of healthcare services. For example, the necessity to minimize social and physical contact prompted more telephone-based and remote health consultations. The pandemic may bring about change in the delivery of healthcare going forward, leading to different treatment delivery methods, times, and costs. Using costs and the health service payer perspective assumptions based on pre–COVID-19 healthcare delivery practices may no longer best reflect the likely budget impact and cost utilization of new treatments.

Most cost-effectiveness and budget impact models assume unlimited resources. Nevertheless, during peak times of the pandemic, not all patients will be able to access healthcare in a timely fashion. Cancer diagnosis is likely to be delayed owing to decreased screening, thus delaying treatment¹⁴ and potentially leading to worse outcomes. This may also affect estimated patient numbers in budget impact models, because more patients would be diagnosed with advanced or metastatic disease than in normal times, affecting eligibility for certain treatments. Furthermore, cancer treatments may not be given or may be given at reduced doses.¹⁵ Budget impact models, in particular, should take into account that less resources may be used than are actually needed. Nevertheless, it is important to recognize that this could lead to further costs and complications in the future if healthcare is delayed.

Article and Author Information

Accepted for Publication: September 3, 2020

Published Online: April 18, 2020

doi: https://doi.org/10.1016/j.jval.2020.09.003