Assessing the benefit:risk ratio of a drug - randomized and naturalistic evidence

Abstract

Randomized evidence from clinical trials and naturalistic evidence collected from pharmacoepidemiology and pharmacovigilance activities both contribute to the initial and continuous assessment of the benefits and risks of a drug, ie, the balance between therapeutic efficacy and safety risks. Benefit-risk assessment (BRA) mainly relies on a qualitative assessment of quantitative data. Current attempts to quantify BRA are reviewed and discussed, along with the expectations of regulatory authorities such as the Food and Drug Administration and the European Medicines Agency. No method provides a fully satisfactory solution regarding BRA, because it is difficult to reduce its multidimensional aspect to simple metrics, in a context where other therapeutic alternatives play a role. Consistency and transparency are key in this assessment, which is performed throughout the whole drug life cycle. BRA is mainly based on randomized clinical studies during clinical development, and it is continued and consolidated by naturalistic data once the drug is on the market.

Drugs provide therapeutic benefits, ie, curing a disease, slowing its evolution, or alleviating its symptoms, but drugs also carry the risks of adverse drug reactions (ADRs), which can span from frequent and minor symptoms, such as nausea or headache, to rare but severe events, such as anaphylaxis, liver failure, or cancer. This dual aspect of therapeutic interventions is seen beyond pharmacology, for example in surgery with the risk of complications such as hemorrhages or infections, and even in psychotherapy, as psychotherapeutic interventions sometimes induce aggravation of psychiatric symptoms.

The review of the benefits and the risks associated with a drug is called benefit:risk assessment (BRA), or benefit-risk balance, or benefitrisk ratio evaluation. BRA is basically an evaluation of two dimensions. The dimension of benefits is measured primarily in terms of therapeutic efficacy, ie, the successful treatment of the condition lor which the drug is indicated. There are other types of benefits, such as improvement of quality of life or pharmacoeconomic aspects, that are of interest in a period where the costs of medicine are closely scrutinized. The dimension of risks includes the safety profile observed in the form of the sum of all ADRs, but also includes the potential risk of unobserved ADRs anticipated on the basis of the mechanism of action.

The evaluation of the benefit:risk ratio of a drug is essential throughout the whole life cycle of a drug. During the discovery phase, the analysis of the biological targets as well as medical chemistry will allow selection of lead molecules with the best BRA potential over hundreds of candidate molecules.^1,2 During the preclinical development of the drug, the evidence obtained from animal models of the disease is compared with the preclinical safety data obtained from toxicological studies, and the preclinical BRA will determine whether a candidate drug will be administered lor the first time in humans.³ The BRA is not a static process, and it evolves during the clinical development, the registration process, and the marketing period, when the drug is administered to patients. However, at all times, BRA remains a major and complex concept. In general, the dynamic aspects of BRA are due to new findings that better characterize the safety profile of a drug and sometimes uncover side effects, making the safety profile of the drug less favorable. Drugs which have been on the market lor years can be withdrawn because the revised safety evaluation confronted with the efficacy findings, no longer supports a favorable BRA, even for drugs with “blockbuster” status.⁴

A revision of the BRA can be justified by the introduction of risk management measures such as a restriction of the indication or monitoring measures. For example, the multiple sclerosis monoclonal antibody natalizumab was registered with significant restrictions in the target patient population following suspension of clinical trials due to some cases of severe infections.

Exceptionally, there are examples where unfavorable BRAs have turned positive, for example when the discovery of a new indication for an old drug increases the positive aspects of its BRA: the relaunch of thalidomide in the indications of multiple myeloma and erythema nodosum leprosum is an example.⁵

The above comments indicate that the BRA of a drug is not an isolated exercise, since it occurs in a global medical and pharmaceutical context. The type of indication for which the drug is planned is critical in this assessment. A drug with a safety profile including risks of severe ADRs, potentially lethal, may be accepted in oncology, but it should not be introduced for the treatment of less severe disorders.

In absolute terms, the BRA of a drug is independent of the existence of alternative therapeutics, but it is clear that when other therapeutics are available in a given indication, regulatory authorities and prescribers will prefer the drug with the more favorable BRA. Economic considerations intervene here as well, and may influence this value scale. In this review, we discuss the relative value for the BRA based on evidence collected by randomized clinical trials versus naturalistic studies. We adopt the following definition of a naturalistic study: “a study in which the researcher carefully observes and records some behavior or phenomenon, sometimes over a prolonged period, in its natural setting, while interfering as little as possible with the subjects or phenomena.”⁶ The naturalistic approach represents essentially all types of observations which are not obtained in randomized clinical trials, but which are obtained during the activities of pharmacovigilance and pharmacoepidemiology. Based on the different approaches used to create the BRA during the life cycle of a drug and in the framework of drug regulations, we discuss how both settings are of interest in this assessment.

Naturalistic versus randomized evidence

During the first half of the 20th century, the evidence lor the therapeutic efficacy of new drugs, in particular anti-infectious drugs, was often so obvious that the naturalistic observations of therapeutic successes in treated patients were sufficient to demonstrate efficacy. However, soon the demonstration of the therapeutic efficacy of new drugs became less obvious, and the need to implement a methodology to demonstrate efficacy appeared necessary.⁷ The demonstration of drug efficacyis essentially a comparative exercise in which a new drug is evaluated versus a comparator, a placebo, or a reference active drug. The clinical efficacy of a treatment is assessed by clinical trials, the methodology of which has been developed and perfected since the early experiments of Sir Austin Bradford Hill in the 1950s⁸; the cornerstone ol clinical trials is the randomization process which ensures that groups ol patients receiving the different treatments are similar. From a statistical viewpoint, the demonstration of efficacy is based on the rejection of the null hypothesis, ie, that there is no difference between the experimental and the comparator treatments.

Several clinical trial designs are used during drug development and generally a couple of randomized controlled trials should provide a demonstration of the statistically significant superiority of the experimental treatment over the comparator. For example, the US Food and Drug Administration (FDA) requires at least two phase III pivotal trials with positive results to allow registration of a new drug.⁹ Regulatory authorities such as the European Committee for Medicinal Product for Human Use (CHMP) from the European Medicines Agency (EMA) regularly publish guidelines on how to evaluate and demonstrate the efficacy and safety of drugs in different therapeutic indications, for example more than 20 CHMP guidelines set the framework for clinical development and clinical trial methodology lor neuropsychiatrie drugs in Europe. Regulatory agencies rely essentially on randomized controlled trials to support the efficacy evidence. The establishment ol efficacy is achieved at the end of Phase III, when the results of the pivotal trials which are key to support the registration process are available. The efficacy conclusion for a given indication will not be modified after this stage, although replication of therapeutic efficacy studies sometimes leads to disappointing results in comparison with those from the pivotal trials.¹⁰ Any new observation of efficacy in subgroups of patients by serendipity will need to be confirmed by randomized evidence obtained in Phase Illb trials, in order to obtain an extension of the indication on the drug label. In the perspective of drug efficacy demonstration, the naturalistic studies represent a weaker design in terms of clinical and statistical quality and power. Although a comparison between an active treatment and a comparator can still be done in a naturalistic setting, such a setting does not permit control for all sources of bias in the estimation of efficacy because of the absence of randomization. The randomized evidence is the support lor demonstrating the benefits expected in BRA for the majority of drugs. There are rare exceptions to this rule, either due to the scarcity of cases or the terminal stage of an incurable illness, or because of an imminent medical threat to the population due to infectious agents,¹¹ which could justify omitting proper clinical trials. In cases of threat of a pandemic infectious disease, it could be necessary to market drugs or vaccines despite limited information from randomized clinical trials; in such cases, there would also be little to no information based on naturalistic observations, and the decisions to administer the therapy in an emergency would be based on surrogate outcomes. Another situation where naturalistic observations might influence the BRA would be when the efficacy of a drug, as demonstrated in randomized clinical trials, did not seem to be maintained in the clinical setting; for example, the clinical benefit Irom psychotropic drugs seems to have declined over the last decades.

The evaluation ol the safety profile of a drug is more complex than the demonstration of its efficacy. Clinical trials are designed and powered to demonstrate the efficacy of the drug; although a lot of safety information is collected during randomized trials, this information covers essentially frequent ADRs, or more exactly the frequent adverse events rather than drug reactions (as the causal relationship between events and the taking of the drug is not yet established). A minority of trials are designed specifically for the assessment of safety, such as trials which assess ECG changes due to drugs expected to affect cardiac electrical conduction.¹² Indeed, the clinical development is limited in terms of patient exposure and duration of exposure: only a few thousand patients receive the drug during the clinical development, most ol these during a relatively short period. Common ADRs can be identified during the clinical development, but rare reactions, with frequency less than 0.1%, are generally not identified. It will require the exposure of 10 000 patients or more in order to detect rare serious ADRs.¹³ The randomized controlled trials performed during Phases I to III do not have the power to properly assess the full safety profile of a drug, and the safety profile of the drug and its BRA established at the time of registration remain limited by this difficulty to capture rare or/and delayed ADR. During the post-registration period, pharmacovigilance through spontaneous reports is critical to consolidate the safety profile of the drug. However, the rarity of spontaneous declarations by prescribers and the complexity of assessing the causality of adverse events lead to the idea that pharmacovigilance is insufficient to fully characterize the BRA during the post-marketing period.¹⁴

This can be complemented by pharmacoepidemiology studies such as observational cohort studies, also called post-approval safety studies in Europe,¹⁵ where patients are prescribed the drug of interest on purely medical grounds, without any randomization. The pharmacovigilance surveillance and the observational pharmacoepidemiology studies offer a naturalistic observational setting which is essential to build the more comprehensive safety profile post-registration and to confirm the preregistration BRA; the naturalistic setting plays a critical role lor marketed drugs.

Quantitative methods for drug benefit-risk assessment

There is a growing interest in quantitative estimates of the BRA,¹⁶ and we review several quantitative and semi-quantitative methods developed with this goal. Each of these methods presents advantages and limitations, meaning that so far none has received unanimous approval nor is systematically used by regulatory authorities or by pharmaceutical industries. The methods presented provide an average BRA for a population of patients, ie, they are not intended for a benefit-risk estimation in individual patients.

Number needed to treat

Number needed to treat (NNT) and number needed to harm (NNH) are simple methods which are useful for assessing the BRA in a single clinical trial.¹⁷ The NNT is the number of patients who need to be treated with the drug in order to achieve one more occurrence of efficacious treatment of the disease targeted by the drug. It is not an absolute value - the NNT depends on the conditions compared: experimental drug versus no treatment, or a more or less efficacious alternative. Hie NNH means the number of patients who need to be treated before one more patient will experience an ADR. The NNH:NNT ratio¹⁸ is a simple tool to measure the increase in the number ol therapeutic successes achieved for each additional ADR incurred from using the drug of interest rather than the reference treatment; it is a simple tool to assess the benefit:risk ratio. If NNILNNT is greater than 1, fewer patients need to be treated to observe a benefit from the drug than to have one additional occurrence of an ADR; in other words the BRA is positive, at least numerically. This metric is widely used by clinicians as well as reimbursement agencies or health insurances because of the simplicity of the concept and its easy calculation. The advantage of this approach is to use similar metrics for efficacy and safety. However, it is questionable to compare directly one therapeutic success versus one ADR. Indeed, a therapeutic success of a drug can have a low clinical impact, for example the decrease of the intensity of symptoms, whereas an ADR can be severe. Therefore, simply counting the chance of one success versus the risk of one ADR is simplistic. In addition, the risk profile of a drug concerns several different ADRs, and such complexity is difficult to summarize by just one single NNH value.

In an attempt to account for the differences in the safetyprofile, it is possible to take into account the patient utility value ol the outcome, ie, to consider patients' preference and quantify the preference lor avoiding the disease ol interest or a specific ADR.^19,20 These utility scores can be included in the NNH:NNT calculation. This makes the calculation more complex and the relative utility scores include some subjectivity.

Quality-Adjusted Time Without Symptoms and Toxicity

Quality-Adjusted Time Without Symptoms and Toxicity is a method where the time lost due to an ADR is subtracted from the time gained from the treatment. In this calculation, one can also use quality-adjusted life years (QALYs), a measure of both the quality and the quantity of life. Benefit is measured by drug-attributed gain in QALYs, and the cumulative risks and disease progression are calculated to obtain drug-attributed loss of QALYs.²¹ This approach allows direct comparison of the gain (benefit) with the loss (risk) into a single metrics. For an individual patient, this estimation can be valid, but for a population of patients the attribution of the value of a year of life gained or lost is more difficult, as individual opinions diverge about this value. A somewhat similar approach is the incremental net health benefit (INHB), where the method described above is used in a comparative manner between two drugs.²²

Multi-Criteria Decision Analysis

Multi-Criteria Decision Analysis (MCDA) is a tool to support decision-making where several benefits and risks can be taken into account. This method had been developed initially to support decision-making in the domains of business and administration. In drug-related MCDA, several risks measured by ADR, treatment discontinuations, drug/drug or drug/disease interactions can be considered, while several benefits can be represented, such as biochemical or clinical efficacy end points and quality of life end points.^23,24 The method is based on hierarchical decision trees that include defined options with different probabilities of occurrence. Différent expected performance scores are obtained, and the different weighted scores for each option can be calculated. Uncertainty parameters and sensitivity analyses can also be computed in MCDA. This approach is promising as it identifies which areas (risks or benefits) are more influential and need more scrutiny, allowing a more explicit decision process. However, the model can be quite complex and statistically tricky, and the assigned weights can bring bias of subjectivity into the model.

Other multidimensional approaches

Other approaches have been proposed. In one of these,²⁵ a rectangle is formed by multiplying the strength of the benefit (such as the magnitude of the positive efficacyresponse) by the response rate. The rectangle is then multiplied by the dimension (quantification) of evidence to form a tridimensional efficacy cuboid. For a given ADR, severity, frequency, and strength of evidence are the three dimensions to construct the safety cuboid. The positive benefit:risk ratio is demonstrated when the volume of the cuboid for benefits outbalances the sum of all cuboids for the different ADRs. The advantage is that different ADRs can be considered together. However, if the concept is theoretically interesting, there is no practical way of comparing the benefit and risk cuboids, and it is not certain that the volume represented by the sum of ADRs can be geometrically compared with a volume measuring the benefit of a drug.

The methods mentioned above, despite their complexity, still do not allow determination, in a simple way, of the relative importance of the benefits and the risks of a given drug in a specific indication. So far, they have not replaced qualitative judgments by experts.

Regulatory authority views

The position of regulatory authorities on the BRA question is instructive, because these authorities have the dual objective of encouraging pharmaceutical therapeutic progress, while protecting public health. Regulatory authorities rely essentially on qualitative assessments and expert opinions. Quantitative methods such as those presented above play only a supportive role in the registration or drug surveillance process. Relying on qualitative assessment and expert opinions makes it necessary to ensure that the regulatory process is valid, consistent, and transparent.²² We present here some aspects ol the US and European regulatory authorities' approaches.

The FDA does not use a quantitative assessment of the BRA, and relies on a qualitative assessment of the quantitative data collected during drug development. For the FDA, the drug benefit derives from the efficacy end points of clinical trials, and risks are based on adverse events reported in trials and, once the drug is on the market, on spontaneous safety data.²⁶ The assessment is based on a judgment where, in addition to the benefit and risks. other factors enter into account such as the notion of unmet medical need or the risk management plan proposed to mitigate the potential safety risks of the drugs. An important element in the BRA performed by the FDA is the opinion given by the Advisory Committees before drug registration, where different specialists independent of the FDA, and sometimes also representatives from patient groups, assess the drug dossier, and take a decision by a vote. The committee decision is indicative, the final decision being made by the FDA. The FDA qualitative assessment can be guided by a framework,²⁷ in a way similar to that of the EMA. This framework supports and formalizes the BRA judgment. It allows in particular standardization of the consistency and transparency in the BRA process and decision, which is essential for the prescribes, the patients, and the pharmaceutical industry.

In Europe, the EMA published in 2008 a paper entitled Reflection Paper on Benefit-Risk Assessment Methods in the context of the Evaluation of Marketing Authorisation Applications of Medicinal Products for Human Use: ²⁸ The Agency explored approaches to improving the methodology for this assessment and the consistency and transparency of the evaluations. For the EMA, as for the FDA, assessments by experts are essential in BRA, and quantitative approaches do not yet replace this qualitative assessment. Two main conclusions emerge from the EMA paper. First, the Agency proposes the use of a specific template for the benefit-risk section of the drug dossier, with specific guidance for the assessors. Hiis guidance allows summarization ol the main data about benefits and risks of the evaluated drug in a structured manner. In particular, the BRA must be performed considering the therapeutic context of the assessed drug. The reflection paper also emphasizes the uncertainties and variability of these estimations and their impact on the decision. Second, it contains an acknowledgment of the need to support research in the development of quantitative or semiquantitative BRA methodologies. The recently created European Network of Centres for Pharmacoepidemiology and Pharmacovigilance is part of this initiative.

Clearly, the FDA and the EMA still rely on expert opinions and qualitative assessment, and not yet on quantitative methods, to summarize the evidence obtained in clinical trials to construct the BRA prior to registration of new drugs. But both agencies encourage the use of frameworks to structure these assessments in order to ensure consistency in the evaluation and decisions.

Discussion

Contrary to the drug efficacy, for which statistical tests can be used to demonstrate superiority of an experimental drug over a comparator in a controlled study design, the methodology to demonstrate in a definitive way the safety ol a treatment is less straightforward and cannot be fully captured by randomized controlled studydesign. For a given safety issue, the risk can be estimated on the basis of incidences of ADR compared between active and reference treatments; however, the safety profile of a drug includes numerous safety issues, and it is difficult to summarize this configuration into a onedimensional concept. Moreover, once the safety risks are identified, in a benefit-risk perspective, one needs to define the acceptance level for each of the risks. What is the tolerated threshold incidence for a given severe ADR such as a drug-induced hepatic failure: should one accept an incidence of one case per 10 000 treated patients, or one case per 100 000, or even less? The response depends on the indication and efficacy of the drug. To add to the difficulty, the efficacy of a drug is well measured in randomized trials, while the risk of a specific ADR can only be assessed once this ADR has been observed: as long as this is not the case, the ADR remains hypothetical, based on some supposed biological mechanism, or even ignored when the ADR is idiosyncratic. For example, the risk ol agranulocytosis with clozapine became obvious when the first case series were recorded,²⁹ not at the time of registration. The potential for a given risk based on the known mechanism of action of the drug (or on that of the pharmacological class of the drug) also enters into the balance, and this potential risk can only be quantified with much uncertainty.

The dimension of time is central to the evaluation of risks, and the BRA of a drug starts during the preclinical development, to continue during the clinical development and the marketing phase. Once on the market, the first years are critical for a drug BRA, as the exposure to the new drug increases considerably in terms of number of patients, of duration of exposure and of heterogeneity of patients compared with the selected patient population included in the clinical trials. However, even the first few years on the market are sometimes not enough to establish a full BRA: the longterm exposure can be critical, as certain ADR may be observed only alter an exposure ol several years, such as cancers or chronic organ toxicity. Immunodepression-related lymphoprolilerative disorders take about 5 years to appear,³⁰ and liver cirrhosis may appear only alter decades of treatment with methotrexate,³¹ Delayed toxicity can be observed in the offspring of patients exposed to a drug, as seen with vaginal adenocarcinomas in daughters of women who had taken diethylstilbestrol during pregnancy.³² The information gathered from randomized studies done during the clinical development corresponds to a drug exposure of limited duration: at this stage of development, the long-term exposure to the drug (1 year or more) is restricted to a limited number ol patients - a lew hundred. The International Conference on Harmonisation (ICH) guideline Ela on the long-term safety requires only 100 patients followed up for 1 year in a registration dossier.³³ Only the naturalistic observations of large-scale and long-duration post-marketing exposure will bring the information on rare and/or delayed ADR. The BRA, based on randomized evidence during the initial clinical development phase, becomes mainly based on naturalistic evidence during the post-marketing period, ie, on evidence from pharmacoepidemiological observational studies and the pharmacovigilance system.

The BRA remains mainly a qualitative exercise. An important limitation inherent to all quantitative BRA estimations is the level of subjectivity in estimating the impact of the ADR, or of the safety risk. Certain BRA quantitative methods use utility scores or patient preferences in their computation. Such approaches are limited, as patient preferences are not available for all conditions. Also patients suffering from a life-threatening disorder such as cancer might not assess a given ADR the same way as patients suffering from a less severe disorder such as depression. It is unclear to what extent one could compare the utility-based approaches with a subjective and individualized assessment to the evidencebased appraisal of drugs. Another limitation of quantitative BRA methods is the risk of oversimplification of the parameters of the benefitrisk ratio; the NNH:NNT ratio is an example of a mathematical tool too simple to capture the complexity of the problem.

This review focuses on the public health perspective, ie, the BRA for the population of potential patients: this is the view of the regulatory authorities and that of the pharmaceutical industry. The BRA based on average values represents what one could expect for the population of patients (in clinical trials or pharmacoepidemiological studies). For a given patient, the efficacy and safety of a given treatment can differ, and often does, from what was concluded on the basis of a population of patients: a patient can be a responder or a nonresponder to a drug, and the efficacy of a medically recognized drug can be null at the individual level. Similarly, the individual safety profile often differs from the average safety profile seen in a population. The subjective perception ol an ADR also plays a role: a given ADR can be tolerated by certain patients, but may be unacceptable to others. In this sense, the MCDA technique reduced to the individual level is very similar to decision analysis, an economic technique whose use has been suggested in medicine since the 1960s³⁴: the BRA integrates probabilities in its computation and one can use it to determine lor a given patient what the best alternative is. In conclusion, the BRA is a dynamic process that evolves according to the cumulated knowledge acquired on drugs, mainly on their safety, as well as on more general conditions such as acceptance of risks in public health terms, or the existence of therapeutic alternatives. Clearly, the evidence from randomized clinical trials is critical to furnish the bases for the BRA before registration of the drug, but collecting information on the patients exposed once the drug is marketed is critical to pursue the BRA process during the life cycle of the drugs.?