Weighing of Evidence by Health Technology Assessment Bodies: Retrospective Study of Reimbursement Recommendations for Conditionally Approved Drugs

Abstract

This study assessed whether five Health Technology Assessment (HTA) bodies in Europe were more negative about drugs with a Conditional Marketing Authorization (CMA) that are approved without controlled studies compared to CMA drugs that are approved based on controlled studies. The HTA recommendations were categorized into positive, restricted, and negative. A total of 92 HTA recommendations were available for 27 drugs. Thirty of 62 (48%) and 17 of 30 (57%) of the recommendations were negative for drugs with and without controlled studies, respectively. Overall, only 12 (13%) recommendations were positive. In all jurisdictions, recommendations between drugs with and drugs without controlled data were comparable, which suggests that the presence of controlled data is not decisive in HTA evaluations. The small proportion of unrestricted positive recommendations highlights difficulties with recommending the drugs in this cohort, which may be caused by scientific uncertainty or other factors. Earlier collaboration between stakeholders is advised in order to improve patient access.

Study Highlights.

• WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?

☑ CMA is increasingly being provided to drugs. The quantitatively more limited evidence package of CMA drugs can include uncontrolled and controlled studies. HTA bodies need to assess relative effectiveness and may, thus, have a problem with accepting drugs based on solely uncontrolled studies.

• WHAT QUESTION DID THIS STUDY ADDRESS?

☑ This study investigated whether HTA bodies throughout Europe were less likely to recommend drugs with a CMA that are approved without controlled studies compared with CMA drugs that are approved based on controlled studies.

• WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE?

☑ The presence or absence of controlled data is not decisive in HTA evaluations of CMA drugs. HTA bodies have rarely given unrestricted positive recommendations for CMA products.

• HOW MIGHT THIS CHANGE CLINICAL PHARMACOLOGY OR TRANSLATIONAL SCIENCE?

☑ To ensure adequate patient access to novel drugs, manufacturers, HTA bodies, and regulators should collaborate more efficiently. This can improve the suitability of evidence for making decisions that are sensitive to quality of evidence and to the added value of (CMA) drugs.

Pharmaceutical regulation needs to protect patients from harm due to ineffective or unsafe drugs, without unnecessarily delaying patient access to safe and effective medicines.1, 2 To balance these interests, the European Medicines Agency (EMA) has implemented the Conditional Marketing Authorization (CMA) in 2006 to speed up patient access to medicines. A CMA can be provided for medicines that fulfill unmet medical needs for patients, in which the benefit of immediate availability outweighs the risks of increased uncertainty, and when it is likely that more comprehensive data will become available at a later stage.3 The EMA will require the manufacturer to commit to performing postmarketing studies, but analyses of obligatory studies showed that more than half were completed with a substantial delay or not at all.4, 5, 6, 7, 8, 9, 10 An evaluation of the EMA oncology approvals in 2009–2013 demonstrated that, at a minimum of 3.3 years after market entry, there was still no solid evidence that these drugs improved or extended life.11 Therefore, it can be questioned whether the more limited evidence package for medicines approved via expedited regulatory pathways is resolved through postauthorization studies. This emphasizes the importance of investigating downstream effects of expedited approvals.

Common evidence limitations for conditional approval drugs are the use of surrogate end points and uncontrolled studies. Although the impact of using surrogate end points for approvals has been extensively studied,12, 13, 14, 15 less is known about approvals based on uncontrolled evidence only, even though they represent a significant proportion of drug approvals, including CMAs.16 An analysis of the EMA oncology approvals between 1995 and 2004 showed that for 48% of indications, no phase III comparative clinical trial was available.17 During 1999–2014, the EMA approved 44 indications solely on the basis of uncontrolled studies, whereas 9 applications with only uncontrolled data were rejected. During the same period, the US Food and Drug Administration (FDA) approved 60 indications based on solely uncontrolled studies, rejecting only one.18 Although approvals with solely uncontrolled studies represent a substantial proportion of regulatory approvals, little is known about their downstream impact.

Although timely patient access to needed drugs is one of the aims of expedited regulatory pathways, in practice, most patients in Europe will not have access to a drug until a positive recommendation for its use in a publicly funded healthcare system is made by Health Technology Assessment (HTA) bodies. The HTA bodies evaluate drugs relative to the national standard of care in order to balance individual patient needs with societal affordability. Therefore, the lack of randomized controlled trials might lead to more negative HTA recommendations.19 HTA bodies are not necessarily more negative about drugs receiving early conditional approval, but they might be reluctant to recommend drugs for which no comparative evidence is available.20, 21 Others have indicated that invoking a barrier for the use of therapies for which comparative effectiveness evidence is lacking is an important and positive feature of HTA bodies and payers in Europe.22 HTA bodies and payers suggested that the increased uncertainty accompanying accelerated approvals should be reflected in a lower drug price.21, 23 However, a recent study found that the Canadian (Canadian Agency for Drugs and Technologies in Health (CADTH)), German (Institute for Quality and Efficiency in Health Care (IQWIG)), and English (National Institute for Health and Care Excellence (NICE)) HTA organizations were willing to accept noncomparative evidence in certain situations in which treatment benefit could still be demonstrated.24 Thus, the CMA cohort is particularly interesting from an HTA perspective, as it represents drugs in which greater uncertainty is implicit. The balance between increased evidence and earlier access has been the subject of many discussions between regulators and HTA body representatives. Research has shown that a lack of alignment on evidentiary standards was found to be one of the greatest barriers to the implementation of adaptive licensing and facilitated regulatory pathways.25, 26 However, insight into the effects of differing evidentiary standards is limited. There is no systematic effort to investigate if HTA recommendations of European HTA bodies for drugs that received expedited approval based on solely uncontrolled studies have differed from those for drugs for which controlled studies are available is lacking.

Therefore, the objective of this study was to assess whether HTA bodies throughout Europe gave more negative recommendations for conditionally authorized drugs that were approved based on uncontrolled studies only.

Results

Product characteristics

Of the 30 CMA drugs approved by the EMA until June 2016, 27 were included in the final analysis. Two drugs were excluded because the CMA was withdrawn by the marketing authorization holder and one product was excluded because it was not assessed by any HTA organization. Nine CMAs (33%) were granted for drugs based on solely uncontrolled studies. For the remaining 18 drugs, at least one controlled trial was available. Two of those had trials with active controls, whereas the rest had a placebo as comparator. Indications for the 27 CMAs included 17 oncology drugs, 6 infectious disease drugs, 3 Central Nervous System drugs, and 1 ophthalmologic drug. Thirteen drugs had their CMA converted to standard Marketing Authorization at the cutoff period for data collection. Twelve CMAs were for orphan drugs.

HTA dossier inclusion

From the five included HTA jurisdictions (England: NICE; France: HAS; Germany: IQWIG; Scotland: SMC; and The Netherlands: ZIN), not all jurisdictions performed HTAs of all newly approved drugs. The number of assessed drugs varied from seven (Germany) to 26 (France) (Figure 1). The low number of recommendations for IQWIG is partly explained by the fact that IQWIG does not assess orphan drugs with a budget impact lower than 50 million Euros per year. A total of 92 HTA recommendations for 34 individual indications were available for the 27 drugs included in this study.

Figure 1.

Flowchart of included Health Technology Assessment recommendations. CEA, cost‐effectiveness assessment; CMA, conditional marketing authorization; HTA, health technology assessment; REA, relative effectiveness assessment.

Recommendation outcomes

The majority of HTA recommendations (N = 47; 51%) for the 27 CMA drugs were negative. Seventeen of 30 (57%) HTA recommendations for drugs without controlled studies were negative as compared to 30 of 62 (48%) HTA recommendations for drugs with controlled studies (Figure 2). The relative risk for receiving a negative HTA recommendation was 1.2 (95% confidence interval (CI): 0.8–1.8) for drugs without controlled studies vs. drugs with controlled studies within this cohort. There were no significant differences in negative recommendations between products with and without controlled trials in any jurisdiction. Twelve of 92 (13%) recommendations were positive. Only two of the drugs that received a CMA based on uncontrolled studies received a positive HTA recommendation. A total of 33 recommendations were positive with restrictions (36%). Twenty‐eight recommendations included economic restrictions (e.g., discounts) and five included economic as well as clinical restrictions (e.g., indication narrowing). Twenty‐one economic restrictions were for drugs with controlled studies and 12 for drugs without, corresponding to 34% and 40% of recommendations, respectively. There were no major differences in the distribution of negative recommendations for orphan products between drugs with (5/13 recommendations negative; 38.5%) and without (7/15 recommendations negative; 46.7%) controlled studies.

Figure 2.

Overall recommendations per Health Technology Assessment jurisdiction and for all combined.

Reasons for negative recommendations are shown in Figure 3. Reasons for giving a negative reimbursement recommendation varied considerably between jurisdictions, reflecting the different nature of HTA processes. Reasons for negative recommendations were subcategorized into economic (i.e., price), clinical, and organizational reasons (i.e., if a company does not make a submission, SMC advice is always negative). Only Scotland had organizational reasons. In France, Germany, and The Netherlands only clinical reasons resulted in negative recommendations, whereas in England and Scotland, both clinical and economic reasons led to negative recommendations. There are no clear differences present between drugs with and without controlled studies in the reasons for giving a negative HTA recommendation, but it should be noted that these reasons are an addition of multiple interacting factors, which we clustered in a limited number of categories.

Figure 3.

Reasons for negative recommendations per jurisdiction and for all jurisdictions combined.

The sensitivity analysis showed that using a different categorization did not lead to different results in Germany, because all recommendations were negative. In France, it changed two overall recommendations from positive to economically restricted (sunitinib for gastrointestinal stromal tumor and everolimus for subependymal giant cell astrocytoma associated with tuberous sclerosis complex). This change did not have an impact on the comparison of negative recommendations.

There were 86 HTA recommendations that included a relative effectiveness assessment (REA), whereas six did not (Figure 4). These six recommendations were all made in Scotland for assessments in which the company did not provide an evidence submission. No REA is performed in such cases. Restricted REAs included two in England (pazopanib and osimertinib) and two in France (raltegravir and darunavir). No significant differences were present in negative REAs for products with or without controlled studies in any country or for all countries combined. The relative risk for receiving a negative REA was 1.4 (95% CI: 0.7–2.7) for drugs without controlled studies vs. drugs with controlled studies.

Figure 4.

Relative effectiveness assessment (REA) outcomes and cost‐effectiveness assessment (CEA) outcomes for all jurisdictions combined.

A total of 33 cost‐effectiveness assessments (CEAs) were performed (Figure 4), no CEAs for the drugs in this study were found within France and Germany. Excluding France and Germany, 12 of 13 (92%) REAs for drugs without controlled studies also had a CEA, whereas this was the case for 21 of 33 (64%) REAs for drugs with controlled studies. For CEAs, all positive recommendations were for drugs that had controlled studies. The relative risk for receiving a negative CEA was 1.2 (95% CI: 0.7–2.0) for drugs without controlled studies vs. drugs with controlled studies.

Discussion

Summary of findings

This study shows that the lack of controlled data was not decisive for the outcome of HTA evaluations in England, Scotland, France, Germany, and the Netherlands of drugs licensed under the European Union (EU) Conditional Marketing Authorization scheme. Furthermore, the majority (87%) of HTAs performed for CMA drugs, irrespective of study type, did not result in an unrestricted positive recommendation. This highlights the difficulties with recommending the drugs in this cohort, which may be caused by scientific uncertainty due to the more preliminary nature of clinical data in CMA drugs or by other factors, such as budget impact. The difference in positive recommendations between REAs and CEAs highlights that these difficulties often originate in the economic evaluations. Because REAs are less granular than CEAs (REAs are either found positive/negative or are qualified in up to five categories, whereas CEAs necessitate precise quantification), this either indicates uncertainty in the precise quantification of the effects (scientific uncertainty) or unacceptable costs in relation to the benefits.

Comparison with other studies

The EMA has approved multiple drugs based on uncontrolled data only. Research has shown that between 1999 and 2014, the EMA approved 44 indications on the basis of only uncontrolled studies, whereas only 9 applications with only uncontrolled data were rejected.18 Another study showed that of 11 recommendations by European HTA organizations for drugs without controlled studies, 7 were negative (64%).24 Although they did not specifically include conditionally approved drugs, this rate of negative results is roughly in line with our findings. Another study compared HTA recommendations for oncology drugs with a CMA vs. oncology drugs with standard marketing authorization (SMA). The authors found no differences in positive recommendations between CMA (7/35; 20%) and SMA (17/79; 22%) drugs.20 Our proportion of positive decisions for CMA drugs is lower (12/92; 13%), most likely due to the classification of French and German recommendations. Our proportion of negative recommendations is in line with their findings (51% vs. 46%, respectively).

Policy considerations

Pharmaceutical companies increasingly use expedited access pathways for new drug indications,16 intended to facilitate early access to promising drugs. However, most healthcare systems in Europe will use the HTA to assess the added value of these drugs before patients will have access. Inherent to early access is a more limited evidence base at conditional approval, and, thus, more uncertainty about relative treatment effects. One of the most fundamental concepts in grading treatment effects is the availability of controlled data and a lack of controlled data leads to downgrading of evidence quality in any grading system.27, 28

Our results show that negative HTA recommendations or (economic) restrictions do not apply more often for conditionally approved drugs without controlled evidence. This means that in HTA, the use of uncontrolled studies for CMA products itself is not a decisive factor to come to a negative or a restricted recommendation. It should be noted that many considerations underlie an HTA recommendation. Other studies have tried to clarify decisive, causative factors for getting positive or negative reimbursement recommendations.29 Our study provides retrospective insight into the way CMA drugs were handled by the HTA institutions. A detailed multivariate analysis of HTA recommendations of CMA drugs could provide useful insight in which (combination of) factors determine negative or restricted recommendations by HTA bodies. The low proportion of unrestricted positive recommendations among all CMA drugs—not just those with solely uncontrolled studies—suggests that HTA bodies struggled with assessing the added value of these CMA drugs. This might be partly explained by factors that may differ between CMA and non‐CMA drugs, such as an over‐representation of orphan products or a lack of comparator treatments, rather than a difference in the proportion approved based on uncontrolled data, as our results indicate. Nevertheless, our results indicate that expedited regulatory pathways may not always lead to earlier patient access. This suggests that the effectiveness of expedited regulatory pathways in balancing more comprehensive evidence with earlier patient access could be improved. Manufacturers, regulators, and HTA bodies should collaborate early during drug development in order to improve the alignment of evidence generation strategies to satisfy both regulatory and HTA evidence requirements. The new joint process for parallel consultations, in which manufacturers can engage in scientific advice from the EMA and multiple HTA institutions in parallel, may provide such a platform that would help to ensure a better aligned vision on which data are necessary for market authorization and reimbursement.30

Limitations

This study has several limitations. All data was extracted manually from HTA reports from different organizations and interpretation of data may have resulted in wrong categorization of HTA recommendations. However, we have validated our data extraction, found a high inter‐rater agreement, and have performed a sensitivity analysis to minimize this limitation. Substantial differences in processes, methods, and procedures for HTA between jurisdictions exist and not all countries assessed all CMA drugs included in this study, which might have influenced the results. For example, in the Netherlands (ZIN) only one product with solely uncontrolled trials was assessed but ZIN recommendations were relatively more often positive for the CMA drugs they assessed. Our selection criteria excluded many HTA institutions in Europe, which differ vastly from the included institutes. Thus, our results cannot be extrapolated to all HTA institutions in Europe. Additionally, we report aggregate results, which are not corrected in a multivariate analysis. The correction of HTA recommendations for multiple covariates is complicated, because known covariates, such as orphan status, already available treatments, (lack of) an assessment of cost‐effectiveness in some jurisdictions, and other factors are mixed with unknown covariates that may impact HTA decision making. Our inherently limited sample size of CMA drugs prohibits us from producing reliable multivariate results. Variation in reimbursement recommendations due to other unknown factors might also result in not observing differences. Thus, it is important to stress that our effort should not be interpreted as an argument for (a lack of) causation between nature of evidence and negative HTA recommendations.

Additionally, our results apply specifically to CMA products and should not be interpreted as applicable to any drug approved based on uncontrolled data. Finally, we could only include five jurisdictions due to the lack of publicly available HTA reports or summaries in most European countries. Systematically making HTA assessment reports public is a prerequisite for studying HTA recommendations for CMA drugs based on solely uncontrolled studies in other European countries and is highly recommended.

Conclusions

The lack of difference between HTA recommendations for CMA drugs with and without uncontrolled studies suggests that HTA bodies treat CMA drugs that do not have controlled data similar to those that do. The lack of unrestricted positive recommendations suggests that there is great uncertainty in HTA bodies considering CMA products, which may be a result of multiple known and unknown factors, among them the greater scientific uncertainty that accompanies CMA drugs. Thus, further collaboration to align regulatory and HTA evidence requirements and improve evidence generation strategies through early advice procedures is advised in order to improve patient access.

Methods

Drugs and jurisdictions

We conducted a retrospective analysis of available HTA reports for all 30 CMA drugs approved between January 2006 and June 2016 that were included in a recent EMA report.31 HTA jurisdictions were selected based on six criteria: (i) the HTA jurisdiction had to be linked to an EU jurisdiction; (ii) reports had to systematically be publicly available; (iii) reports had to include background information on included studies; (iv) the HTA recommendation had to have an official role in the decision‐making process considering reimbursement; (v) the HTA body involved is the primary institute with legal remits within the jurisdiction; and (vi) the report had to be in a language understood by the assessors (i.e., English, French, German, or Dutch). Of 29 jurisdictions within the EU (England and Scotland have separate HTA jurisdictions), 22 did not systematically publish reports with background information. Two jurisdictions reported in a language not understood by the assessors. The final inclusion consisted of HTA reports from England (NICE), France (HAS), Germany (IQWIG), Scotland (SMC), and The Netherlands (ZIN). HTA reports were included up until June 30, 2017.

Data collection

For all CMA drugs, we matched HTA reports based on the first approved indication in the CMA provided by the EMA. When products are approved for multiple indications or for an indication for which multiple comparator treatments exist, some HTA bodies split these into separate recommendations, either within the same dossier (France and Germany) or in a separate dossier (England and Scotland). In these cases, all recommendations were included separately. Basic product information was gathered from the EMA website, including therapeutic category, dates of CMA and possible SMA, orphan status, and type and amount of pivotal studies. For each HTA jurisdiction, data were collected on the overall reimbursement recommendations, outcomes of REAs, and outcomes of CEAs. Reasons for negative decisions or restrictions were collected, as well as decision and publication dates. A standardized data extraction form was developed by the first author (R.A.V.) to assure data quality. The extraction form was improved by a second investigator (L.T.B.) and, consequently, a third investigator (J.C.B.) validated the data collection through assessing a random selection of drugs in each jurisdiction. Interrater agreement was 90.7%, with an unweighted Kappa of 0.867, indicating excellent agreement.32, 33 Inconsistencies were discussed until consensus was reached.

Data categorization and analysis

Studies were categorized as uncontrolled when they did not include an active comparator or placebo arm (e.g., single‐arm trials and dose‐ranging studies). In line with previous research we categorized HTA recommendations into positive, positive with restrictions (called restricted), or negative.19 Furthermore, we collected the outcomes of the REA, the CEA for those countries that include CEA, and the overall recommendation separately. In France and Germany, REA assessments will result in one of multiple categories that have different consequences for price setting and negotiations, depending on the level of added benefit that the drug provides. We used sensitivity analysis to assess whether a different categorization mechanism would have led to different conclusions (primary categorization is provided in Figure S1 and categorization for sensitivity analyses in Figure S2). Positive decisions with restrictions were further subcategorized into economic (e.g., discounts) and clinical (e.g., only for specific patient subgroups) restrictions. Reasons for negative recommendations were subcategorized into economic (i.e., price), clinical, and organizational reasons (e.g., if a company does not make a submission, SMC advice is always negative). Relative risks for negative HTA recommendations for drugs approved with and without controlled studies were calculated. Data analysis was performed in Excel (Microsoft, Redmond, WA).

Funding

No funding was received for this work.

Conflict of Interest

All authors declare no support from any organization for the submitted work; A.M.H. reports unrestricted grants from GlaxoSmithKline, outside the submitted work, the other authors declare no other relationships or activities that could appear to have influenced the submitted work.

Author Contributions

All authors wrote the manuscript. R.A.V., J.C.B., H.G.M.L., and W.G.G. designed the research. R.A.V., L.T.B., and J.C.B. performed the research. R.A.V., A.M.H., A.K.M.‐T., H.G.M.L., and W.G.G. analyzed the data.

Disclaimer

The views expressed in this article are the personal views of the authors and must not be understood or quoted as being made on behalf of or reflecting the position of the Dutch National Healthcare Institute, the National Institute of Health and Care Excellence, or the Dutch Medicines Evaluation Board.

Supporting information

Figure S1. Categorization of health technology assessment recommendations.

Figure S2. Categorization of health technology assessment recommendations in the sensitivity analysis.