Associations between uncertainties identified by the European Medicines Agency and national decision making on reimbursement by HTA agencies

Abstract

We aimed to determine whether uncertainties identified by the European Medicines Agency (EMA) were associated with negative relative effectiveness assessments (REAs) and negative overall reimbursement recommendations by national health technology assessment (HTA) agencies. Therefore, we identified all HTA reports from Haute Autorité de Santé (HAS; France), National Institute for Health and Care Excellence (NICE; England/Wales), Scottish Medicine Consortium (SMC; Scotland), and Zorginstituut Nederland (ZIN; The Netherlands) for a cohort of innovative medicines that the EMA had approved in 2009 to 2010 (excluding vaccines). Uncertainty regarding pivotal trial methodology, clinical outcomes, and their clinical relevance were combined to reflect a low, medium, or high level of uncertainty. We assessed associations by calculating risk ratios (RRs) and 95% confidence intervals (CIs), and agreement between REA and overall reimbursement recommendation outcomes. We identified 36 medicines for which 121 reimbursement recommendations had been issued by the HTA agencies between September 2009 and July 2018. High versus low uncertainty was associated with an increased risk for negative REAs and negative overall reimbursement recommendations: RRs 1.9 (95% CI 0.9–3.9) and 1.6 (95% CI 0.7–3.5), respectively, which was supported by further sensitivity analyses. We identified a lack of agreement between 33 (27%) REA and overall reimbursement recommendation outcomes, which were mostly restricted recommendations that followed on negative REAs in case of low or medium uncertainty. In conclusion, high uncertainty identified by the EMA was associated with negative REAs and negative overall reimbursement recommendations. To reduce uncertainty and ultimately facilitate efficient patient access, regulators, HTA agencies, and other stakeholders should discuss how uncertainties should be weighed and addressed early in the drug life cycle of innovative treatments.

Study Highlights.

• WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?

Several studies have assessed whether uncertainty in regulatory decision making is associated with health technology assessment (HTA) outcomes, with varying results. However, these studies focused on one specific disease‐related, clinical, or regulatory factor that represented regulatory uncertainty.

• WHAT QUESTION DID THIS STUDY ADDRESS?

Is a comprehensive measure of uncertainties identified by the European Medicines Agency (EMA) regarding (i) pivotal trial methodology, (ii) clinical outcomes, and (iii) their clinical relevance, associated with negative relative effectiveness assessments (REAs), and negative overall reimbursement recommendations by national HTA agencies?

• WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE?

A high level of uncertainty in regulatory decision making seemed to be associated with negative REAs and negative overall reimbursement recommendations.

• HOW MIGHT THIS CHANGE CLINICAL PHARMACOLOGY OR TRANSLATIONAL SCIENCE?

To enable efficient patient access to new innovative medicines, it is imperative to reduce uncertainty through early multistakeholder discussions about relevant uncertainties and how they should be weighed and addressed. Currently, ongoing and future initiatives that facilitate such dialogues are therefore of utmost importance.

INTRODUCTION

In Europe, patient access to costly innovative medicines often requires a positive reimbursement recommendation by a national health technology assessment (HTA) agency. ¹ , ² HTA agencies provide recommendations based on a relative effectiveness assessment (REA), and, depending on the agency, other considerations, such as a cost‐effectiveness assessment (CEA) and budget impact analysis (BIA). ³ For these assessments, HTA agencies depend on evidence submitted to them by medicine manufacturers. However, because HTA decision making on reimbursement is preceded by regulatory decision making on market approval in the drug life cycle, evidentiary standards set by regulators, such as the European Medicines Agency (EMA), influence the amount and type of data and (un)certainty available to downstream decision makers, such as HTA agencies. ⁴

When deciding on market approval, medicine regulators assess the benefit‐risk balance of a medicine based on its quality, safety, and efficacy. ⁵ These decisions always involve some level of uncertainty, which is inherent to the underlying data and the relative weights that are, explicitly or implicitly, given to different safety and efficacy outcomes. ⁵ , ⁶ Moreover, when considering patient access to medicines, regulators weigh the need for more data against potential risks associated with remaining uncertainties and, in some cases, against an unmet medical need in the studied patient population. ⁷

Whether factors that represent a higher level of uncertainty in regulatory decisions are associated with HTA outcomes has recently been studied, with varying results. Some studies assessed the association between such factors and REA outcomes, ⁸ whereas most focused on overall reimbursement recommendation outcomes. ⁸ , ⁹ , ¹⁰ , ¹¹ , ¹² , ¹³ With regard to REAs, negative outcomes may be expected when regulatory benefit‐risk decisions are made with a high level of uncertainty because these decisions are informed by largely the same clinical data. ¹⁴ In contrast, such uncertainty may have less of an impact on overall reimbursement recommendations because, depending on the HTA agency, these recommendations may also be informed by additional assessments, such as CEA and BIA. ³ , ¹⁵ HTA agencies may then weigh the uncertainties associated with clinical assessment outcomes against CEA and BIA outcomes and considerations, such as unmet medical need. Thus, to better understand the role of upstream uncertainty in the HTA decision making process, it is important to study its impact on both REA and overall reimbursement recommendation outcomes.

Moreover, previous studies focused on one specific disease‐related, clinical, or regulatory factor that represented uncertainty in regulatory decision making (i.e., presence of [ultra‐]orphan status for medicines, ⁹ , ¹⁰ uncontrolled clinical trials supporting regulatory approval, ⁸ , ¹¹ and use of early access pathways). ¹² , ¹³ However, a more diverse set of regulatory uncertainty aspects may be more in line with the HTA perspective on relative effectiveness (e.g., uncertainty regarding the methodology of pivotal clinical trials, uncertainty regarding the clinical outcome demonstrated by these trials, and uncertainty regarding the clinical relevance of these outcomes).

For the current study, we hypothesized that a higher level of these uncertainty aspects identified by the EMA during regulatory assessment would be associated with negative REAs because the data underlying these assessments are roughly similar. However, we expected that the level of uncertainty would be less strongly associated with overall reimbursement recommendation outcomes, because also other aspects are taken into account. Therefore, the aim of this study was to determine whether a higher level of uncertainty identified by the EMA was associated with negative REAs and negative overall reimbursement recommendations by national HTA agencies.

METHODS

Study design and inclusion criteria for medicines and HTA agencies

We performed a retrospective cohort study consisting of all innovative medicines (i.e., products containing new active substances, that were approved by the EMA between January 1, 2009, and December 31, 2010). This cohort was chosen for two reasons: (i) for these medicines, confidential, nonpublicly available data on the EMA’s uncertainty regarding pivotal clinical trial data (methods, clinical outcome, and clinical relevance) had previously been obtained through a Memorandum of Understanding with the Dutch Medicines Evaluation Board, ¹⁶ which formed a unique opportunity to study this association, and (ii) substantial follow‐up time was considered necessary to allow for the availability of the HTA decision making outcomes. We excluded vaccines because their product and clinical use characteristics require HTA assessment processes that are substantially different from the assessment processes for other medicines.

Consecutively, we determined whether the following four HTA agencies had assessed the initially approved indications of the remaining medicines: the Haute Autorité de Santé (HAS, responsible for France), the National Institute for Health and Care Excellence (NICE, responsible for England and Wales in the United Kingdom), the Scottish Medicine Consortium (SMC, responsible for Scotland in the United Kingdom), and the Zorginstituut Nederland (ZIN, responsible for the Netherlands). These agencies were selected based on five criteria that we also used in previous studies ⁸ , ¹² : (i) the agency had to be responsible for making reimbursement recommendations in a European jurisdiction during the study period, (ii) recommendation reports had to be publicly available, (iii) recommendations had to play an official role in the final reimbursement decision making process, (iv) the agency had to be the primary institute with legal capacity in making reimbursement recommendations within the jurisdiction, and (v) the report had to be in a language understood by the researchers (authors L.T.B., R.A.V., and N.W.L.P.; Dutch, English, French, or German. We excluded medicines that had not been assessed by any of the above agencies.

Data extraction: EMA uncertainty aspects and HTA reimbursement recommendations

For the included medicines, we first assessed the level of uncertainty identified by the EMA during the regulatory assessment, based on three uncertainty aspects. First, uncertainty regarding the methodology of pivotal clinical trials was considered present when so‐called “major objections” concerning the study design, choice of end points, patient population studied, trial duration, and statistical analyses had been expressed during the pre‐approval review process. ¹⁷ Second, uncertainty regarding the clinical outcome demonstrated by pivotal clinical trials was considered present when uncertainty regarding the statistical significance of the primary outcome remained at the time of approval and/or serious safety concerns had been raised. Third, uncertainty regarding the clinical relevance of the clinical outcomes was considered present when none of the following applied at the time of approval: a large effect size, important medical need, and compelling clinical benefit. These data were previously extracted from public and confidential EMA assessment reports and assessed, with substantial agreement reached between the primary data collection and a blinded independent review of a randomly selected sample. ¹⁶ The level of composite uncertainty was scored as low when none of these uncertainty aspects were considered present, medium when one aspect was considered present, and high when two or three aspects were considered present.

Second, we identified the first reimbursement recommendation report for each medicines’ initial EMA approved indication(s) (“medicine‐indication combination”). This was done for all four HTA agencies noted above and up to November 30, 2020. We disregarded recommendation reports that were not based on data (“non‐submissions”) and excluded re‐assessments. When HTA agencies had split the EMA approved indications into subindications for which they issued separate reimbursement recommendations, we regarded these as unique medicine‐indication combinations. From the included reports, we extracted the date of recommendation, REA outcome, and overall reimbursement recommendation outcome for each relevant medicine‐indication combination. We assessed REA outcomes as positive or negative, and overall reimbursement recommendation outcomes as unrestricted positive, restricted positive, or negative, in line with previous research. ⁸ , ¹² Relative effectiveness that was higher than or comparable to a comparator was considered a positive REA outcome, whereas lower effectiveness—including in case of a lack of data—was considered a negative REA outcome. Overall reimbursement recommendations were considered restricted in case of reimbursement for a smaller indication than initially approved by the EMA or lower reimbursement than the price requested by the company. ⁸ , ¹² Data extraction was performed by N.W.L.P. for the full cohort and validated by L.T.B. for a random 10% sample of medicines, based on which we calculated the percentage of agreement and Cohen’s kappa for interrater agreement. ¹⁸ Data that did not correspond were discussed until consensus was reached.

Data analysis

We initially characterized the cohort using descriptive statistics. We then performed two main analyses to assess associations between a higher level of composite uncertainty (medium vs. low and high vs. low) identified by the EMA and HTA outcomes, by calculating risk ratios (RRs) and Wald 95% confidence intervals (CIs). First, we assessed the association with negative REAs. Second, we assessed the association with negative overall reimbursement recommendations. For the latter analysis, restricted positive and unrestricted positive overall reimbursement recommendations were aggregated. The analyses were performed irrespective of the HTA agency that issued the recommendations. However, to provide insight in agency‐specific associations, we visualized the overall and agency‐specific distributions of outcomes. In addition, we performed sensitivity analyses by restricting the two main analyses to medicine‐indication combinations for which all agencies issued reimbursement recommendations. This was done to avoid that the analyses would be affected by variation due to differences between HTA agencies in medicine‐indication combinations they assessed.

Furthermore, to provide insight in the most important uncertainty aspects driving potential associations, we performed six ancillary analyses to assess associations between each individual uncertainty aspect and negative REAs and negative overall reimbursement recommendations. For these, we also performed sensitivity analyses as described above.

Additionally, we performed sensitivity analyses to substantiate our assumption that pre‐approval major objections concerning the methodological robustness of pivotal clinical trials would reflect remaining methodological uncertainty. We therefore reviewed the major objections and how these were addressed by the companies, and considered whether a higher level of methodological uncertainty in line with the major objections remained at the time of approval. In doing so, we followed a conservative approach and only considered the level of uncertainty to remain higher if companies were unable to submit the requested data pre‐approval and thus committed to submit further data postapproval. If an indication was restricted pending the submission of data postapproval, we considered that the level of methodological uncertainty was lowered. We then recategorized the level of composite uncertainty and replicated both the main analyses and the ancillary analyses involving methodological uncertainty to assess whether any changes in the categorization of uncertainty affected the results.

Last, we assessed the proportion of medicine‐indication combinations for which the REA and overall reimbursement recommendation outcomes did not correspond. That is, when an unrestricted or restricted positive overall reimbursement recommendation was issued while the REA was negative, or a negative overall reimbursement recommendation while the REA was positive. We also assessed whether this proportion differed depending on the level of uncertainty identified by the EMA.

RESULTS

Cohort characteristics: Medicines, HTA agencies, and reimbursement recommendations

Between January 1, 2009, and December 31, 2010, 45 innovative medicines were approved by the EMA. Of these, we excluded nine medicines: eight vaccines and one medicine, for which initial indication had not been assessed by any included HTA agency (rilonacept, brand name Rilonacept Regeneron). We included the remaining 36 medicines (see Table 1 for some summary characteristics). A detailed overview of the included medicines and their indications as initially approved by the EMA is available in Table S1. We identified uncertainty regarding the methodology of pivotal clinical trials for 22 medicines, uncertainty regarding the clinical outcome for 6, and uncertainty regarding clinical relevance for 10. ¹⁶

TABLE 1.

Characteristics of the included medicines (n = 36) approved by EMA in 2009–2010

Characteristic	Level of composite uncertainty
	Low (n = 9)	Medium (n = 18)	High (n = 9)
Biological or ATMP	5 (56%)	8 (44%)	0 (0%)
Initial approved indication
Cancer treatment	1 (11%)	3 (17%)	2 (22%)
Cardiovascular treatment	2 (22%)	1 (6%)	1 (11%)
Immunosuppressive treatment	1 (11%)	3 (17%)	0 (0%)
Musculo‐skeletal disorder treatment	2 (22%)	1 (6%)	0 (0%)
Other treatment a	3 (33%)	10 (56%)	6 (67%)
Type of market approval
Regular approval	7 (78%)	15 (83%)	9 (100%)
Conditional approval b	0 (0%)	3 (17%)	0 (0%)
Exceptional approval b	2 (22%)	0 (0%)	0 (0%)
Orphan status at approval	3 (33%)	4 (22%)	1 (6%)
At least one indication assessed by
HAS (France)	9 (100%)	18 (100%)	9 (100%)
NICE (England and Wales)	4 (44%)	8 (44%)	4 (44%)
SMC (Scotland)	6 (67%)	13 (72%)	7 (78%)
ZIN (the Netherlands)	7 (78%)	11 (61%)	9 (100%)
Time between EMA approval and HTA recommendation (median, range)
HAS (France)	240 days (86–567)	242 days (104–1631)	380 days (62–1210)
NICE (England and Wales)	774 days (154–2340)	529 days (146–1854)	412 days (272–569)
SMC (Scotland)	332 days (126–926)	245 days (81–2808)	186 days (32–386)
ZIN (the Netherlands)	395 days (171–1387)	399 days (130–1653)	200 days (130–1146)

Abbreviations: ATMP, advanced therapeutic medicinal product; EMA, European Medicines Agency; HAS, Haute Autorité de Santé; HTA, Health Technology Assessment; NICE, National Institute for Health and Care Excellence; SMC, Scottish Medicines Consortium; ZIN, Zorginstituut Nederland.

^a For example, antidiabetics, blood‐related treatment, diagnostic agents, psychopharmacological treatment, respiratory treatment, sex hormones, and related treatment. All ≤3 medicines in total.

^b Conditional approval and approval under exceptional circumstances are regulatory pathways that aim at providing (early) access to medicines that address a high unmet medical need. ¹⁹

The 36 medicines were approved by the EMA with one or more initial indication(s)—40 in total—and some were further split by HTA agencies into 2 subindications. In total, this led to 45 unique medicine‐indication combinations for which HTA agencies could have issued reimbursement recommendations. However, not all agencies assessed all medicine‐indication combinations, and we therefore ultimately included 121 reimbursement recommendations that had been issued between September 2009 and July 2018. The process of identification of medicines, HTA agencies, and reimbursement recommendations is shown in Figure 1. The data validation yielded a 93% agreement rate with a Kappa of 0.88, indicating excellent agreement.

FIGURE 1.

Flowchart of medicines included in the study cohort and medicine‐indication combinations for which health technology assessment (HTA) outcomes were extracted. EMA, European Medicines Agency; HAS, Haute Autorité de Santé; HTA, health technology assessment; NICE, National Institute for Health and Care Excellence; SMC, Scottish Medicines Consortium; ZIN, Zorginstituut Nederland. (1) Epoetin theta (Eporatio) and denosumab (Prolia) were initially approved for two indications, golimumab (Simponi) was initially approved for three indications (see Table S1). (2) HAS further split one EMA approved indication into two subindications, which resulted in two separate recommendations. (3) NICE further split two EMA approved indications into two subindications each, which resulted in four separate recommendations. (4) ZIN further split two EMA approved indications into two subindications each, which resulted in four separate recommendations

Relative effectiveness assessment

Of the 121 REAs, 48 (40%) were negative and 73 (60%) were positive. The distribution of these outcomes is presented in Figure 2a; separately for medicine‐indication combinations associated with a low, medium, and high level of composite uncertainty identified by the EMA, and both overall as well as for each individual HTA agency. RRs for a negative REA were 1.7 (95% CI 0.9–3.5; medium vs. low uncertainty) and 1.9 (95% CI 0.9–3.9; high vs. low uncertainty; Table 2), which, given the relatively small sample, is suggestive of an association between the level of uncertainty and decision making on REAs by HTA agencies. The sensitivity analysis that was restricted to medicine‐indication combinations for which all four HTA agencies issued reimbursement recommendations (see Figure 3a) supported the existence of an increased RR for high versus low uncertainty: RR 2.1 (95% CI 0.9–5.0). This result indicates a slightly more pronounced association given the higher point estimate and higher lower bound of the CI. However, it did not support the existence of an increased RR for medium versus low uncertainty: 1.3 (95% CI 0.5–3.1; Table 2). The most important uncertainty aspect driving the association seemed to be uncertainty regarding the methodology of pivotal clinical trials: 1.6 (95% CI 1.0–2.7; see Figure S1a and Table S2).

FIGURE 2.

Relative effectiveness assessment (REA) (a) and overall reimbursement recommendation (b) outcomes for all medicine‐indication combinations (n = 121) stratified by level of composite uncertainty, overall, and per HTA agency. HAS, Haute Autorité de Santé (France); NICE, National Institute for Health and Care Excellence (England and Wales, United Kingdom); SMC, Scottish Medicines Consortium (Scotland, United Kingdom); ZIN, Zorginstituut Nederland (the Netherlands)

TABLE 2.

Associations between level of composite uncertainty and negative REAs and overall reimbursement recommendations

Level of composite uncertainty	Primary analysis (n = 121)	RR (95% CI)	Sensitivity analysis (n = 68) a	RR (95% CI)
Negative REA
Low	7/28 (25%)	Ref.	4/12 (33%)	Ref.
Medium	27/63 (43%)	1.7 (0.9–3.5)	17/39 (44%)	1.3 (0.5–3.1)
High	14/30 (47%)	1.9 (0.9–3.9)	12/17 (71%)	2.1 (0.9–5.0)
Negative overall reimbursement recommendation b
Low	7/28 (25%)	Ref.	3/12 (25%)	Ref.
Medium	16/63 (25%)	1.0 (0.5–2.2)	8/39 (21%)	0.8 (0.3–2.6)
High	12/30 (40%)	1.6 (0.7–3.5)	10/17 (59%)	2.4 (0.8–6.8)

Abbreviations: CI, confidence interval; REA, relative effectiveness assessment; RR, risk ratio.

^a Restricted to medicine‐indication combinations for which all HTA agencies issued reimbursement recommendations.

^b For this analysis, the alternative non‐negative outcome consisted of restricted and unrestricted positive overall reimbursement recommendations.

FIGURE 3.

Relative effectiveness assessment (REA) (a) and overall reimbursement recommendation (b) outcomes for medicine‐indication combinations for which all four HTA agencies issued reimbursement recommendations (n = 68), stratified by level of composite uncertainty.

Overall reimbursement recommendations

Of the 121 overall reimbursement recommendations, 35 (29%) were negative, 71 were positive but restricted (59%), and 15 (12%) were positive and unrestricted. The distribution of these outcomes is presented in Figure 2b; separately for medicine‐indication combinations associated with a low, medium, and high level of composite uncertainty identified by the EMA, and both overall as well as for each individual HTA agency. RRs for a negative overall reimbursement recommendation were 1.0 (95% CI 0.5–2.2; medium vs. low uncertainty) and 1.6 (95% CI 0.7–3.5; high vs. low uncertainty; Table 2), which suggests a potential association only for a high versus low level of composite uncertainty. These findings were both supported by the sensitivity analysis (see Figure 3b): RR 0.8 (95% CI 0.3–2.6), indicating no association for medium versus low uncertainty, and RR 2.4 (95% CI 0.8–6.8), indicating that a high level of uncertainty led to more negative overall recommendations by HTA agencies (Table 2). The most important uncertainty aspect driving the potential association seemed to be uncertainty regarding the clinical outcome: 1.7 (95% CI 1.0–3.0; see Figure S1b and Table S2).

Review of major objections and sensitivity analyses

We considered that a higher level of methodological uncertainty remained for at least 11 of the 22 medicines for which major objections had been expressed during the pre‐approval review process, because of commitments to provide additional data postapproval. For nine of these medicines, all data were to be obtained from new or ongoing studies that had not been part of the approval dossier. For one, only preliminary data of one of two requested studies had been part of the approval dossier. For another, requested long‐term efficacy and safety data of the pivotal trial had to be provided postapproval. The analyses based on this alternative categorization supported the main and ancillary analyses, indicating the same trends and no substantial changes in point estimates considering the relatively broad CIs (Table S3).

For the other 11 medicines, major objections had been resolved through (a combination of) restricted indications, labelling, additional analyses, or narrative justifications. However, also for these medicines, we often noted that, at approval, the EMA had flagged important remaining limitations in the data that resolved major objections, which may affect HTA decision making. These included nonpreferred comparators, the uncontrolled nature of additional studies, indirect comparisons, and inability to demonstrate noninferiority.

Discrepancies between REA and corresponding overall reimbursement recommendation outcomes per medicine‐indication combination

REA and overall reimbursement recommendation outcomes did not correspond for 33 of the 121 medicine‐indication combinations (27%). This occurred most frequently for medicine‐indication combinations with a negative REA: 23 of 48 negative REAs (48%) were followed by a positive overall reimbursement recommendation. Of these, 22 (96%) were restricted positive overall reimbursement recommendations. In case of a negative REA, medicine‐indication combinations with a high level of composite uncertainty seemed less likely than those with a low or medium level to receive a (restricted) positive overall reimbursement recommendation: 4 of 14 (29%) versus 3 of 7 (43%) and 16 of 27 (59%), respectively. In contrast, only 10 of 73 positive REAs (14%) were followed by a negative overall reimbursement recommendation and this occurred equally often for low (3/21, 14%), medium (5/36, 14%), and high 2/16, 13%) level of composite uncertainty.

DISCUSSION

Our study suggests that a high versus a low level of composite uncertainty identified by the EMA was associated with a 1.9‐fold increased risk of negative REAs and 1.6‐fold increased risk of negative overall reimbursement recommendations by HTA agencies. Our sensitivity analysis restricted to medicine‐indication combinations for which all agencies issued reimbursement recommendations showed stronger associations and strengthened our main findings.

These associations for medicine‐indication combinations with a high level of composite uncertainty may at least be partly explained by similarities in clinical data that inform benefit‐risk assessments and REAs. ¹⁴ In addition, similarities in how regulators and HTA agencies assess relevant uncertainties in these data may also play a role. ²⁰ HTA agencies may obtain information on relevant uncertainties either indirectly through the regulator’s assessment—as evidenced by the many references to the EMA’s public assessment reports that we identified in HTA reports and the fact that some HTA agencies explicitly require these reports to be submitted ²¹ —or by performing their own assessment of the data. However, although regulators may decide to grant approval and address remaining uncertainties through requests for further postapproval evidence generation, HTA agencies have to come to a decision based on the then available data, including uncertainties. Moreover, regulators are potentially more inclined to do so in case of uncertainties that are of less relevance to them as they are to HTA agencies—such as use of a nonpreferred comparator ²⁰ , ²² or surrogate rather than clinical outcomes in clinical trials—which may result in negative REAs, as we show in our study.

In contrast, we identified a weaker association between a medium level of composite uncertainty and negative REAs that largely disappeared in the sensitivity analysis and no association with negative overall reimbursement recommendations. One of the reasons for this was that a large proportion of negative REAs was translated into a positive overall reimbursement recommendation—of which most (96%) were restricted. This occurred most often for medicine‐indication combinations with a medium level of composite uncertainty; more than twice as often as for those with a high level of composite uncertainty and 1.4 times as often as for those with a low level. These clinical and/or economic restrictions may be one way for HTA agencies to address a remaining—but acceptable—level of uncertainty while allowing access to medicines.

The lack of an association with negative HTA outcomes for medicine‐indication combinations with a medium level of composite uncertainty could further be explained by other factors that may be taken into account during reimbursement decision making, such as unmet medical need ²³ and price‐related aspects, such as CEA and BIA. ³ These may cause a medium level of composite uncertainty to be weighed differently and considered acceptable, whereas a high level of uncertainty is not. The importance of unmet medical need in HTA decision making has been highlighted by others that studied uncertainty associated with medicines that had been approved based on data from uncontrolled trials ¹¹ , ²⁴ or through early access pathways. ¹² Both uncontrolled trials and approval through early access pathways are typical characteristics of medicines that address an unmet medical need, ²⁵ , ²⁶ , ²⁷ and may also have played a role in our study. Although only few medicines had been approved through early access pathways (14%), all three that were conditionally approved—indicating that uncertainties had to be addressed postapproval—were associated with a medium level of uncertainty and mostly received positive (but restricted) overall reimbursement recommendations. In addition, most medicines indicated for cancer treatment—which often address a high unmet medical need and may be approved based on uncontrolled trials ²⁸ —were associated with a higher level of uncertainty identified by the EMA. However, also other indications may be associated with an unmet medical need. For example, dronedarone (Multaq) was associated with one of the highest levels of uncertainty—scoring negative on all uncertainty aspects—and all HTA agencies considered that its relative effectiveness in preventing atrial fibrillation recurrence was negative. Nonetheless, NICE and SMC issued a positive, but restricted, reimbursement recommendation to allow for the availability of a treatment option with a better side‐effect profile, which was regarded an unmet medical need by patients and health care providers. ²⁹ , ³⁰

Importantly, whereas the different HTA agencies request broadly similar evidence for their REAs, ³¹ they differ in the extent to which they take aspects, such as CEA, BIA, and unmet medical need into account. ¹⁵ , ³² , ³³ , ³⁴ Differences in the content and the processes of these assessments between agencies may explain discrepancies in reimbursement recommendation outcomes between them that have previously been reported. ⁸ , ¹³ , ³³ , ³⁴ , ³⁵ In our current study, agency‐specific distributions of overall reimbursement recommendation outcomes indicate an association between a higher level of uncertainty and negative outcomes for HAS and ZIN, but not for NICE and SMC. A potential explanation may be the extent to which CEA is taken into account by agencies. NICE and SMC perform a comprehensive CEA for every recommendation and may perform pricing negotiations prior to issuing a reimbursement recommendation. In contrast, HAS does not perform CEAs in most cases and ZIN applies a risk‐based approach to considering CEAs, whereas pricing negotiations fall outside their mandate. ¹⁵ Moreover, NICE’s assessment process is very extensive and includes a review of the company submission as well as additional data—for which they are known to sometimes wait—by an external “Evidence Review Group”. ³⁶ This may reduce uncertainty and thus lead to less negative outcomes; also of their REAs, as evidenced by the agency‐specific data. Conversely, HTA outcomes for NICE constitute final reimbursement decisions whereas HTA outcomes of other organizations can comprise recommendations to a subsequently deciding authority that may still wish to negotiate prices, for example, the Minister of Health in the Netherlands. ³ These differences between agencies may also explain the differences in time from market approval to issue of reimbursement recommendation that we observed.

To prevent that uncertainties adversely impact patient access to innovative medicines, it is imperative to reduce overall uncertainty through multistakeholder discussions about relevant uncertainties and how they should be weighed and addressed. In addition, these may also stimulate further alignment on specific evidence needs for decision making between regulators and HTA agencies. Currently, ongoing initiatives that facilitate such dialogues—often early in the drug life cycle—are therefore of great importance. These include, for example, collaboration between the EMA and the European Network for Health Technology Assessment (EUnetHTA), ³⁷ , ³⁸ the EMA PRIority MEdicines (PRIME) scheme, ³⁹ and other (inter)national initiatives. ⁴⁰ , ⁴¹ , ⁴² , ⁴³ These are of great relevance to overcome the current barriers to efficient patient access to new innovative medicines, including the impact of remaining uncertainties after regulatory approval. ⁴⁴

An important strength of our study is that it studied associations between a comprehensive measure of uncertainties identified during regulatory assessment and subsequent HTA decision making outcomes. Moreover, we substantiated our assumption that major objections would reflect remaining methodological uncertainty because (i) the results of the sensitivity analyses based on a conservative assessment of remaining methodological uncertainty were in line with our other findings, and (ii) we flagged important caveats that may affect HTA decision making for many of the other medicines for which major objections had been expressed. Furthermore, our study provided insights in HTA agency‐specific associations for such uncertainty that appeared in line with known differences in activities and mandates between agencies. However, it also has several limitations. First, although the major objections reflect a diverse set of methodological aspects of the regulatory assessment of clinical data that is largely in line with the HTA REA, they may not always capture the uncertainty aspects that are relevant to HTA agencies (e.g., choice of comparator and noninferiority rather than superiority study designs). ²⁰ , ²² We can thus not exclude the role of any other methodological uncertainties. Second, we assessed a cohort of medicines that was approved by the EMA several years ago. However, the broad type of medicines and indications were largely similar to those currently approved, ⁴⁵ consisting of a fair share of biologicals and even one cell‐based therapy and with cancer treatment already being the major indication area. Nevertheless, recent approvals are likely associated with even more uncertainty (e.g., because they are more often based on single‐arm studies that include small numbers of patients). ⁴⁶ Therefore, if anything, a more negative impact on HTA decision making outcomes can be expected. Third, differences in the type of medicines assessed by each HTA agency as well as differences in assessment methods, responsibilities, and mandates may have caused variation in assessment outcomes between HTA agencies that affected our results. However, we have addressed this by performing sensitivity analyses restricted to medicine‐indication combinations that had been assessed by all agencies and these strengthened our main analyses by indicating even more pronounced results. Fourth, because of the small number of medicine‐indication combinations per agency, we were not able to estimate with sufficient precision agency‐specific associations and discrepancies between REA and corresponding overall reimbursement recommendation outcomes. In addition, due to the relatively small sample of recommendations, we may not have been able to identify associations that actually exist. We have tried to lower the impact of this limitation by performing several sensitivity analyses on a restricted cohort and assessing and discussing any resulting shift in point estimates. Of note, the fact that our results consistently suggest a “dose‐dependent” association between uncertainty and negative HTA outcomes (i.e., the highest uncertainty was associated with the highest risk of negative outcomes), further support our findings. Fifth, we only included data from four HTA agencies, mostly because of a lack of publicly available HTA recommendation reports from other agencies. Considering the organizational and mandate‐related differences between agencies, this limits the generalizability of our findings to HTA decision making in Europe in general.

CONCLUSIONS

A high level of composite uncertainty identified by the EMA seemed to be associated with negative REAs and negative overall reimbursement recommendations by HTA agencies in Europe. To reduce uncertainty, current and future initiatives for multistakeholder interaction early in the drug life cycle must include discussions about relevant uncertainties and how they should be weighed and addressed. Ultimately, this will facilitate efficient patient access to new innovative treatments.

CONFLICT OF INTEREST

H.G.M.L. reports that he is a member of the Lygature Leadership Team. All other authors declared no competing interests for this work.

AUTHOR CONTRIBUTIONS

L.B., R.V., J.H., M.E., H.L., O.K., W.G., and A.M. wrote the manuscript. L.B., R.V., N.P., J.H., and A.M. designed the research. L.B., R.V., and N.P. performed the research. L.B., R.V., N.P., J.H., M.E., H.L., O.K., W.G., and A.M. analyzed the data.

DISCLAIMER

The views expressed in this article are the personal views of the authors and must not be understood or quoted as being made on behalf of or reflecting the position of the organizations with which the authors are affiliated.

Supporting information

Supplementary Material