Biometrical issues in the analysis of adverse events within the benefit assessment of drugs

Abstract

The analysis of adverse events plays an important role in the benefit assessment of drugs. Consequently, results on adverse events are an integral part of reimbursement dossiers submitted by pharmaceutical companies to health policy decision‐makers. Methods applied in the analysis of adverse events commonly include simple standard methods for contingency tables. However, the results produced may be misleading if observations are censored at the time of discontinuation due to treatment switching or noncompliance, resulting in unequal follow‐up periods. In this paper, we present examples to show that the application of inadequate methods for the analysis of adverse events in the reimbursement dossier can lead to a downgrading of the evidence on a drug's benefit in the subsequent assessment, as greater harm from the drug cannot be excluded with sufficient certainty. Legal regulations on the benefit assessment of drugs in Germany are presented, in particular, with regard to the analysis of adverse events. Differences in safety considerations between the drug approval process and the benefit assessment are discussed. We show that the naive application of simple proportions in reimbursement dossiers frequently leads to uninterpretable results if observations are censored and the average follow‐up periods differ between treatment groups. Likewise, the application of incidence rates may be misleading in the case of recurrent events and unequal follow‐up periods. To allow for an appropriate benefit assessment of drugs, adequate survival time methods accounting for time dependencies and duration of follow‐up are required, not only for time‐to‐event efficacy endpoints but also for adverse events. © 2016 The Authors. Pharmaceutical Statistics published by John Wiley & Sons Ltd.

1. Introduction

The analysis of adverse events plays an important role in the regulatory process of drug approval 1, 2, as well as in the benefit assessment of drugs after approval 3. Despite the importance of adequate reporting of safety data, statistical approaches to analyze and summarize adverse events are frequently restricted to descriptive methods or oversimplified standard methods for contingency tables 4, 5. The improvement of statistical methods to analyze safety data from randomized controlled trials has been repeatedly recommended 2, 6; however, the methodological advances in this field are limited 5, 7.

As noted in regulatory guidelines 8, 9, the adequate quantification of risks associated with adverse events may require the application of statistical methods for time‐to‐event data to account for the duration of follow‐up and censoring. The application of appropriate survival time methods is important, especially for the early benefit assessment of new drugs in Germany, as according to German law, any statement on safety must be based on a quantitative comparison of adverse events between the new drug and the so‐called appropriate comparator therapy (see succeeding paragraphs) 3.

In this paper, we present legal regulations on the benefit assessment of drugs in Germany, in particular, with regard to the analysis of adverse events. We also discuss differences in safety considerations between the drug approval process and the benefit assessment. In addition, we present examples to show that the application of inadequate methods for the analysis of adverse events in the reimbursement dossier can lead to a downgrading of the evidence on a drug's benefit in the subsequent assessment.

2. Requirements for the Analysis of Adverse Events within the Benefit Assessment

In the drug approval process, a new drug is granted approval for a specific therapeutic indication and patient population if sufficient evidence on efficacy and safety is available from high‐quality clinical trials and if the benefits outweigh the risks. While efficacy requirements are well‐defined, owing to continuous advances in the statistical methods for efficacy evaluation, only limited advances have been made in the analysis of safety data 7, 10, 11. Within the framework of drug regulation, assessments of drug safety are performed during both the premarketing and postmarketing stages; for both stages, recommendations are given for safety analyses, including planning, data collection, evaluation, and reporting 12. In this paper, we consider only the premarketing stage to discuss different requirements for analyses of adverse events within the drug approval process and the benefit assessment.

Adverse events associated with drug exposure vary with regard to seriousness, frequency, and statistical properties. Several measures and statistical methods are available to describe and analyze adverse events. Whether their use is appropriate depends on the situation considered 11. It has long been recognized that the use of naive proportions is inappropriate in situations where patients are not treated and followed up for the same period of time 11. Several regulatory guidelines underline the fact that the adequate quantification of risks associated with adverse events may require the application of statistical methods for time‐to‐event data to account for the duration of follow‐up and censoring 8, 9. It has also long been known that further challenges exist because of competing risks and recurrent events 11.

The use of appropriate measures and statistical methods to describe and analyze adverse events is especially important for quantitative benefit‐risk evaluations. However, in clinical trials, much more attention is paid to the estimation of efficacy than to harm 13. Although attempts have been made to perform quantitative benefit‐risk evaluations 14, 15, the traditional evaluations of efficacy and safety in the drug approval process are performed separately and the conclusion on the benefit‐risk ratio is largely based on an informal qualitative approach 4. Several regulatory agencies, research organizations, and pharmaceutical companies have been developing formal quantitative approaches for benefit‐risk evaluations to improve the consistency, transparency, and communication of these evaluations within the drug approval process 4. However, it is not expected that quantitative approaches will replace the approach currently used; that is, the qualitative evaluation of efficacy and safety will remain a standard component of the drug approval process 14, 16.

The use of simplified methods to detect safety signals and evaluate them qualitatively may be sufficient within the drug approval process, as avoidance of unacceptable risks is one of the primary purposes. Hence, no causality judgement is made for adverse events, and even strong bias in favor of the control group is considered acceptable. After drug approval, the occurrence of unacceptable risks is considered to be highly unlikely. Therefore, in the benefit assessment of drugs after approval, a reconfirming benefit‐risk evaluation based on safety signals is not required. However, this does not mean that any harm from the drug can be excluded. In the benefit assessment of drugs, a purely qualitative consideration of safety signals is insufficient, as the goal of such an assessment is a causality judgement and not only signal detection. In the benefit assessment, all patient‐relevant endpoints play a role, which means that on the safety side, particular consideration is given to serious and severe adverse events as well as treatment discontinuations. In addition, adverse events that are of special importance within the context of the disease or drug class considered may play a role. According to the German Act on the Reform of the Market for Medicinal Products (Gesetz zur Neuordnung des Arzneimittelmarktes, AMNOG), introduced in 2011, the basis for the pricing of a new drug is given by the extent and probability of the drug's added benefit 3. The Federal Joint Committee (G‐BA), the main decision‐making body in the German healthcare system, supervises the benefit assessments conducted pursuant to AMNOG. It usually commissions the Institute for Quality and Efficiency in Health Care (IQWiG) to conduct the assessments, which focus on the patient population for whom the drug is approved according to the summary of product characteristics (SPC). IQWiG uses the following approach to assess the extent and probability of the added benefit of a new drug compared with the appropriate comparator therapy specified by the G‐BA.

Conclusions on the probability of (added) benefit are made depending on the number of available studies, the certainty of the study results, and the direction and statistical significance of the corresponding estimated treatment effects 3. The four categories, (1) ‘proof’; (2)‘indication’; (3) ‘hint’; and (4) ‘none of the other 3 categories’, are used for grading the probability of (added) benefit. More details can be found in IQWiG's methods paper 3. To describe the extent of added benefit or harm, the estimated effects on all patient‐relevant endpoints are graded into the following six categories: (1) ‘major’, (2) ‘considerable’, (3) ‘minor’, (4) ‘non‐quantifiable’, (5) ‘no added benefit’, or (6) ‘less benefit’ 3, 17. These categories are defined verbally in the German Regulation for Early Benefit Assessment of New Pharmaceuticals (Arzneimittel‐Nutzenbewertungsverordnung, ANV) 18; that is, their use is mandatory.

In the ANV, it is clearly stated that the benefit of a drug is determined on the basis of the patient‐relevant therapeutic effect on the following endpoints: improvement of health, shortening of duration of the disease, prolongation of survival, reduction in adverse events, or improvement of quality of life. An ordinal grading approach therefore has to be used in the benefit assessment in order to classify the effect size of treatment on all patient‐relevant endpoints, including safety endpoints. The naive application of simple proportions in reimbursement dossiers frequently fails to produce interpretable effect sizes if observations are censored and/or the average follow‐up periods differ between groups. Likewise, the application of incidence rates may be misleading in the case of recurrent events and unequal follow‐up periods. Therefore, the use of adequate survival time methods accounting for time dependencies and duration of follow‐up is required for the benefit assessment; this also applies to safety endpoints.

3. Examples

For illustration, we present two examples of results on patient‐relevant endpoints presented in the dossiers submitted by the pharmaceutical companies as well as the subsequent assessments performed by IQWiG. In the present paper, we focus on safety data.

3.1. Vandetanib in thyroid cancer

Vandetanib has been approved in Germany since February 2012 for adult patients suffering from aggressive and symptomatic medullary thyroid carcinoma with unresectable, locally advanced or metastatic disease. Pursuant to AMNOG, IQWiG examined the added benefit of vandetanib compared with the appropriate comparator therapy ‘best supportive care’ (BSC) on the basis of two dossiers and subsequent data provided by the pharmaceutical company 19, 20, 21. IQWIG concluded that there was a hint of a minor added benefit of vandetanib in patients aged younger than 65 years but a hint of greater harm (lesser benefit) in older patients.

In the original dossier, the company presented no data on patients for whom the drug is approved according to the SPC. Consequently, no added benefit could be proven, as the dossier was incomplete 19. The company applied for a reassessment of vandetanib within a transition period and submitted a new dossier to the G‐BA. In the second dossier, the company presented data from the study submitted in the approval process (approval study) for patients who had been treated according to the SPC.

The risk of bias in the approval study was high, particularly because patients could switch treatment after disease progression. About two‐thirds of them switched from the control group to open treatment with vandetanib. Consequently, the median duration of treatment in the vandetanib group was considerably longer than in the BSC group (88.6 weeks vs. 37.1 weeks). Regardless of this difference, for serious adverse events, naive proportions based on a simple 2 × 2 table with a sample size of 185 patients were presented in the dossier, leading to an estimated relative risk (RR) of 1.87 with 95% confidence interval (CI) [1.01, 3.48]. This statistically significant result to the disadvantage of vandetanib was interpreted by the company as proof of greater harm from vandetanib. However, as the analysis did not adequately consider the markedly longer duration of treatment in the vandetanib group, the risk of bias in this group was assessed as too high, and no final conclusion on harm was drawn by IQWiG 20, meaning that greater harm from vandetanib was neither proven nor could it be excluded. Because of the great uncertainty regarding harm, it could also not be excluded that negative effects outweighed the positive ones. Consequently, no added benefit of vandetanib could be proven on the basis of the data from the second dossier.

The company submitted additional analyses in the commenting procedure conducted by the G‐BA. The G‐BA subsequently commissioned IQWiG to prepare an addendum to the dossier assessment 21. Besides additional data on pain, the company provided survival analyses based on Cox proportional hazards models for adverse events to account for the duration of follow‐up. On the basis of the additional data on pain, IQWiG concluded a hint of an added benefit of vandetanib, which was rated as ‘considerable’ in patients aged younger than 65 years. However, in older patients, there was no difference in pain progression between vandetanib and BSC. Regarding safety, it was shown that for serious adverse events, the statistically significant effect in the analysis using naive proportions disappeared when survival time methods were used and the hazard ratio (HR) was estimated (sample size: 185 patients); that is, a statistically non‐significant effect was found (HR=1.4, 95% CI: [0.74, 2.63]). However, the effect was statistically significant (HR=2.27, 95% CI: [1.47, 3.52]) with regard to severe adverse events of Grade 3 severity or higher according to the Common Terminology Criteria for Adverse Events. With regard to this endpoint, IQWiG rated the extent of harm from vandetanib as ‘major’.

In consequence, IQWiG concluded that treatment with vandetanib resulted in both positive and negative effects in patients aged younger than 65 years and downgraded the extent of added benefit from ‘considerable’ to ‘minor’. In older patients, only negative effects were identified. Overall, IQWiG concluded that there was a hint of a minor added benefit of vandetanib in patients aged younger than 65 years but a hint of lesser benefit in older patients.

3.2. Abiraterone in prostate cancer

Abiraterone acetate (abiraterone for short) has been approved in Germany since December 2012 for men with metastatic prostate cancer that is not responsive to conventional androgen deprivation therapy (ADT). Further criteria are that patients have no or at most mild symptoms and that chemotherapy is not yet clinically indicated. IQWiG assessed the added benefit of abiraterone compared with watchful waiting (while maintaining ADT) on the basis of a dossier and subsequent data provided by the company 22, 23. IQWiG concluded that in comparison with watchful waiting, abiraterone can prolong overall survival and delay the occurrence of severe pain. However, because of the poor safety data available, it could not be excluded with certainty that abiraterone also causes greater harm. Overall, IQWiG derived a hint of a considerable added benefit.

The assessment was based on the approval study of abiraterone in which patients received either abiraterone and prednisone or placebo and prednisone. Almost all patients (94%) in both treatment groups also received a hormone‐blocking drug. In both groups, treatment was continued until disease progression. Unfortunately, the monitoring of adverse events was stopped after treatment discontinuation. In the abiraterone group, this was the case after a median of 13.8 months, whereas in the control group, the median time to disease progression was 8.3 months. This means that the duration of treatment and follow‐up differed greatly between the two groups. Nevertheless, the study results showed that abiraterone offered advantages for mortality (overall survival) and morbidity (severe pain). IQWiG thus concluded an indication of an added benefit for both endpoints. The extent of added benefit was rated as ‘minor’ for mortality and ‘considerable’ for morbidity 22.

Similarly to the vandetanib example, most safety data presented by the company were not analyzed appropriately. For example, for serious adverse events, naive proportions based on a simple 2 × 2 table with a sample size of 1082 patients were presented, leading to an estimated RR of 1.28 (95% CI: [1.07, 1.54], p=0.007). This statistically significant result to the disadvantage of abiraterone was interpreted by the company as greater harm from abiraterone of a minor extent. However, these data could not be used because of the considerable difference in the duration of treatment between the two groups (13.8 vs. 8.3 months), which was not properly considered by the company. The analysis included in the approval documents of severe adverse events occurring during the first 3 months of treatment could be used. However, at this early stage, where most patients were probably still being treated with abiraterone or placebo, the difference between the two groups was not statistically significant.

Naive proportions, also based on a simple 2 × 2 table with a sample size of 1082 patients, were reported for the overall rate of adverse events, resulting in a statistically significant difference to the disadvantage of abiraterone (RR=1.02, 95% CI: [1.01, 1.04], p=0.007). In addition, the number of adverse events per 100 patient years (1156 in the abiraterone group vs. 1264 in the control group) was provided and – in contrast to the aforementioned conclusions – interpreted as a trend towards fewer adverse events per patient year in the abiraterone group. This analysis was also inappropriate, as multiple events of a single patient are counted in the same way as the same number of patients with only one adverse event.

Greater or lesser harm from abiraterone was therefore not proven but could not be excluded with certainty either. On the basis of the available data, only positive effects remained, namely indications of a minor added benefit regarding mortality (overall survival) and a considerable added benefit regarding morbidity (severe pain). However, because of the uncertainty regarding harm, overall IQWiG did not derive an indication of added benefit and downgraded the benefit statement to a hint of a considerable added benefit of abiraterone 22.

The company submitted additional analyses in the commenting procedure conducted by the G‐BA. The G‐BA subsequently commissioned IQWiG to prepare an addendum to the dossier assessment 23. Regarding safety, survival analyses with estimated HRs were provided by the company for adverse events and serious adverse events. However, the new survival analyses contained discrepancies to the dossier, as well as to the clinical study report, regarding the sample size and the number of patients with such events. For example, the underlying sample size for the overall rate of adverse events was 1049, that is, 33 patients fewer than before. These discrepancies could not be resolved. The uncertainty regarding harm therefore remained, and the conclusion of the original dossier assessment 22 was not changed 23.

A useful additional analysis in this example would be the application of survival time methods for recurrent events to make use of the full information in the case of multiple events in single patients 24, 25. In general, the choice of appropriate statistical methods for recurrent events depends on many factors, including the research question, the type of event, the biological process, the number of events, the relationship between subsequent events, and the distribution of events 24. However, as a first step, an adequate analysis of the time to the first event should be provided, which was not the case in this example.

4. Discussion

Simple standard methods for contingency tables are frequently used in the analysis of adverse events in reimbursement dossiers on new drugs. However, the results produced may be misleading if observations are censored at the time of discontinuation due to treatment switching or noncompliance, as the resulting unequal follow‐up periods are not taken into account. By means of two examples, we showed that the application of inadequate methods for the analysis of adverse events can lead to a downgrading of benefit statements on a drug in the assessment of reimbursement dossiers. In both examples, censoring by a competing event (treatment discontinuation at the time of progression) occurred, leading to unequal follow‐up periods between the treatment groups. If endpoints are not monitored over the whole follow‐up period, it is not possible to analyze treatment effects adequately and the corresponding results thus do not represent a fair comparison of treatment groups. In both examples, the neglect of these differences was unfavorable for the new drug.

This can be discussed in more detail by means of the abiraterone example. At the time point for the primary analysis, 46% of the patients in the abiraterone group were receiving chemotherapy versus 59% in the control group. Chemotherapy is usually initiated when progression occurs, and according to the protocol, the study medication is discontinued at this time. Given the aforementioned difference in the number of patients receiving chemotherapy, it can be concluded that the use of abiraterone could reduce the need for, or at least delay the start of, chemotherapy. Because chemotherapy is associated with a considerable risk of (severe or serious) adverse events, the use of abiraterone could also prevent, or at least delay, their occurrence. However, this hypothesis in favor of abiraterone cannot be proven if the monitoring of adverse events is stopped after the study medication is discontinued.

This clearly illustrates an important difference between the approval of a new drug and the assessment of its benefit after approval. While regulatory authorities are primarily interested in direct associations between treatment with the new drug and the occurrence of adverse events, a benefit assessment focuses primarily on a treatment strategy (e.g., starting treatment with a certain drug and then potentially switching or adding treatments) and the long‐term effects of this strategy. In approval studies, it might therefore be reasonable to stop the observation of study participants shortly after the discontinuation of the study medication. This approach was also discussed from a regulatory perspective with a recommendation for major improvements in the documentation and analysis of safety endpoints in clinical trials to enable appropriate benefit‐risk analyses 13. For the benefit assessment of drugs, it is definitely inappropriate to stop the follow‐up of study participants shortly after discontinuation of the study medication, as secondary or indirect effects of a treatment strategy can then no longer be proven. The current practice of stopping the documentation of adverse events when the study medication is discontinued should be changed to enable a fair comparison of treatment strategies. Methodological challenges in the analysis of safety data for the purpose of a benefit assessment include adequate consideration of competing risks 26, 27 and recurrent events 25, 26.

With regard to safety assessments, another important difference between the approval of a new drug and the assessment of its benefit after approval is that regulatory authorities make no causality judgements for adverse events, as a priori hypotheses and endpoints for safety are less well defined than those for efficacy; the sample size estimation in clinical trials is performed for efficacy and not for safety endpoints, and multiplicity problems are unavoidable 11. On the other hand, the basis of a benefit assessment is the demonstration of causality for all patient‐relevant endpoints, including adverse events. In systematic reviews, the assessment of harm is more difficult than the assessment of efficacy endpoints 3. It is therefore even more important that adverse events in clinical trials are completely documented to enable a fair comparison of treatment groups, also for safety endpoints.

In summary, for an appropriate benefit assessment of drugs, adequate survival time methods accounting for time dependencies and duration of follow‐up are required, not only for time‐to‐event efficacy endpoints of clinical trials but also for adverse events.

Bender, R. , Beckmann, L. , and Lange, S. (2016) Biometrical issues in the analysis of adverse events within the benefit assessment of drugs. Pharmaceut. Statist., 15: 292–296. doi: 10.1002/pst.1740.