Evaluation of Pre-marketing Factors to Predict Post-marketing Boxed Warnings and Safety Withdrawals

Abstract

Introduction

An important goal in drug regulation is understanding serious safety issues with new drugs as soon as possible. Achieving this goal requires us to understand whether information provided during the Food and Drug Administration (FDA) drug review can predict serious safety issues that are usually identified after the product is approved. However, research on this topic remains under-studied. In this paper, we examine whether any pre-marketing drug characteristics are associated with serious post-marketing safety actions.

Methods

We study this question using an internal FDA database containing every new small molecule drug submitted to the FDA’s Center for Drug Evaluation and Research (CDER) on or after November 21, 1997, and approved and commercially launched before December 31, 2009. Serious post-marketing safety actions include whether these drugs ever experienced either a post-marketing boxed warning or a withdrawal from the market due to safety concerns. A random effects logistic regression model was used to test whether any pre-marketing characteristics were associated with either post-marketing safety action.

Results

A total of 219 new molecular entities were analyzed. Among these drugs, 11 experienced a safety withdrawal and 30 received boxed warnings by July 31, 2016. Contrary to prevailing hypotheses, we find that neither clinical trial sample sizes nor review time windows are associated with the addition of a post-marketing boxed warning or safety withdrawal. However, we do find that new drugs approved with either a boxed warning or priority review are more likely to experience post-marketing boxed warnings. Furthermore, drugs approved with boxed warnings tend to receive post-marketing boxed warnings resulting from new safety information that are unrelated to the original warning. Drugs approved with a boxed warning are 3.88 times more likely to receive a post-marketing boxed warning, while drugs approved with a priority review are 3.51 times more likely to receive a post-marketing boxed warning.

Conclusion

Although drugs approved with a boxed warning or priority review are more likely to experience serious post-marketing safety events, other information provided during the FDA drug review that is easy to quantify is generally not associated with post-marketing safety events. It appears that these post-marketing events are not discernible during a pre-marketing review and therefore might not be avoidable using current review data.

1. Introduction

Understanding a medicine’s safety profile is important to the public health. Although a drug’s common side effects are generally well understood by the time the Food and Drug Administration (FDA) approves it, less common serious safety issues can emerge once products are marketed. The identification of post-marketing safety issues not encountered in pre-marketing clinical trials is usually attributed to the following post-marketing factors: (1) a substantially larger patient population, which allows for the detection of rare side effects; (2) a broader range of patients with underlying conditions and concomitant medications, which allows the detection of drug–disease and drug–drug interactions; (3) patients who use the medicine for conditions for which it was not studied, which allows for the detection of adverse reactions in the off-label setting; (4) medication errors; and (5) patient misuse of drugs.

While these post-marketing factors that allow identification of new drug safety issues are well known, the pre-marketing factors that might be correlated with serious adverse drug reactions initially identified in the post-marketing period are less well understood. Potential pre-marketing factors may include the following:

1. The number of subjects tested in the clinical trials [1]. Some have argued that studying more patients prior to approval would detect more adverse events [2].

2. The time to approve an application or the time window between a drug’s review goal date and the date the drug is actually approved [3–9]. Other studies have hypothesized that drugs that were approved under an arbitrarily specified deadline might cause reviewers to overlook safety information that, had it not been overlooked, would have allowed the reviewer to identify a drug’s serious safety issues during the review process, potentially preventing future post-marketing safety events [8, 9].

3. The manufacturer’s experience developing drugs. Experienced manufacturers tend to be larger in size and may therefore have the resources for more sophisticated systems for identifying new post-marketing safety information.

4. Various other drug characteristics, such as a drug’s intended therapeutic category or route of administration [3–9].

Research to date on this topic has focused on the relationship between review time characteristics and post-marketing safety events due to the implementation of the Prescription Drug User Fee Act (PDUFA) in 1992, which reduced review times by instituting review timelines and associated review goal dates. In a series of studies, Olson examined drugs approved before and after the implementation of PDUFA, and found that drugs that received faster approval times were associated with an increased number of serious post-marketing adverse event reports [3–5]. By contrast, Grabowski and Wang found no relationship between serious post-marketing adverse event reports and approval times for drugs approved after PDUFA and, Berndt et al. showed that the proportion of post-marketing drug safety withdrawals remained unchanged in the periods before and after the implementation of PDUFA [6, 7].

More recent studies examined whether serious post-marketing safety actions were associated with the time window between a drug’s review goal date and the date it was approved (which we will refer to as the “time window”). Carpenter et al. examined drug applications submitted to the FDA after the implementation of PDUFA, and found that drugs approved closer to their first-cycle review goal dates—in the 2-month time window before their first-cycle PDUFA goal dates—were more likely to experience post-marketing safety withdrawals, safety alerts, dosage discontinuations, and boxed warnings than were drugs approved either after their goal dates or more than 2 months before their goal dates [8, 9].

To address these issues, we first examine whether post-marketing safety outcomes are associated with pre-marketing characteristics that previous studies did not examine, such as the number of patients tested in the clinical trials. The number of persons exposed to a drug during the clinical trial period is generally dictated by the sample size requirements of efficacy trials (though there are international guidelines on the extent of exposure, including long-term exposure, for drugs intended to treat chronic illnesses), as well as the size of the potential patient population (i.e., small trial sizes for drugs for rare or ultra-rare diseases). Some researchers argue that too few patients are studied to uncover potential adverse safety outcomes, and thus recommend expanding preapproval clinical trial sizes [1, 2].

Second, we study a more recent time period than previous studies. Most papers in this literature examined drugs approved in the early 1990s to early 2000s, when the PDUFA program was in its early stages. Although studying this early PDUFA time period provides important insights on the initial relationship between serious post-marketing safety actions and either review times or time windows, the new drug review process has evolved and progressed as science, policy, and technology have improved over time. Thus, studying a more recent time period provides insights that are relevant to current review practices.

2. Methods

2.1. Data Analysis

We use an internal FDA dataset that contains every application for a new small-molecule drug that was initially submitted to the FDA’s Center for Drug Evaluation and Research (CDER) on or after November 21, 1997, the date of the first reauthorization of PDUFA, and that was approved and commercially launched before December 31, 2009. For this study, new small-molecule drugs include every new chemical entity drug except those approved exclusively for diagnostic use. Hence, we have excluded diagnostic agents, therapeutic biologics, vaccines, and other non-therapeutic biologic compounds.

We investigate the relationship between pre-marketing factors and post-marketing safety actions [expressed as odds ratios (ORs) and 95% confidence intervals (CIs)] using a random effects logistic model, following the previous literature [8, 9]. Using this model enables us to compare our results to those published in past papers, and provides evidence that any differences in results are not attributable to model selection. Using alternative models, such as a simple logit, simple probit and random effects probit regression, produced qualitatively similar results. All analyses were conducted using STATA software, version 12.

We use two measures of serious post-marketing safety actions as our dependent variables of interest. One measure is a binary variable that equals one if the drug meets the following criteria: (1) it was withdrawn from the market due to safety concerns; or (2) its sale was suspended due to safety concerns; or (3) it had significant restrictions to its indicated use due to safety concerns that changed the benefit–risk considerations for the drug’s initial approved indication. The first two criteria are technically referred to as safety withdrawals, while the third criterion is referred to as a restricted indication. Although restricted indications do not result in a drug’s full withdrawal, we group restricted indications with safety withdrawals when these restrictions reflect a substantial, safety-driven revision of a drug’s benefit–risk profile.

The other measure is a binary variable that equals one if a drug received a post-marketing boxed warning on its label due to a safety concern. We define a post-marketing boxed warning to include new boxed warnings, where new warnings include either (1) the inclusion of a boxed warning on a drug that was approved without a boxed warning on the label initially, (2) a substantive modification of an existing risk, or (3) an additional risk added to an existing boxed warning during the marketing phase of a drug that resulted from new safety information (e.g., new adverse event reports, voluntary clinical trials, or other scientific studies) unrelated to the original warning. We do not consider new boxed warnings to include editorial changes, revised terminology, clarifying language, non-substantive expansions of the original warning, or labeling changes that improve labeling consistency across related drugs with respect to an already existing boxed warning. Boxed warnings that are the result of a new indication supplement where the identified risk is specific to the newly approved indication are also excluded from our definition. Both post-marketing variables were measured with the methods used in previous studies, and were observed through to July 31, 2016 in order for us to observe the post-marketing safety profile of every drug in our sample for at least 6 years [8, 9].

We have two pre-marketing factors as our main independent variables of interest: (1) the log number of individuals exposed to the study drug during the pre-marketing clinical trials, which we use to assess whether examining more individuals in clinical trials improves the ability to uncover adverse safety risks and thus reduce the number of post-marketing serious safety actions; and (2) a binary variable that equals one if a drug’s last-cycle review was completed 1 week before its last-cycle PDUFA goal date, where the last-cycle review represents the review cycle resulting in the drug’s approval.

In addition to these variables, our analysis also includes other independent variables of drug characteristics that are potentially associated with post-marketing safety. These other characteristics include the log number of prior approvals received by the drug’s manufacturer, whether the drug was subjected to a priority review versus the standard drug review [10], whether the drug was approved with a boxed warning, the drug’s route of administration, the drug’s novelty (measured by its order of entry into its drug class), the number of years the drug was commercially available on the market during our study period, and the log number of cumulative units sold for a drug during the drug’s first 2 years after market launch (a detailed description of our construction of these variables is provided in Electronic Supplementary Material 1). Our random effects captures the drug’s primary therapeutic indication [8, 9].

We expand upon previous analyses by introducing two novel drug characteristics: (1) the drug’s post-marketing exposure; and (2) whether a drug was approved with a boxed warning, which captures whether the drug was approved with a known serious safety risk sufficient to warrant a boxed warning. Previous studies indicate that clinical trial sizes and review windows are associated with various confounding factors that are correlated with post-marketing safety actions, and that two particularly important confounding factors are a drug’s post-marketing exposure and known health risks [3–7]. Hence, controlling for the above covariates is expected to address an important limitation in the literature.

2.2. Sensitivity Analysis

In addition to the above analysis, we conducted multiple sensitivity analyses to test the robustness of our results. Previous studies examined the relationship between post-marketing safety outcomes associated with drugs approved 2 months before their first review cycle goal dates [8, 9]. To test whether our results are robust to other time windows and review cycles, we examined the relationship between post-marketing safety outcomes associated with drugs approved 1 week before their first review cycle goal dates, drugs approved 2 months before their first review cycle goal dates, and drugs approved 2 months before the drug’s last review cycle.

3. Results

CDER received and subsequently approved 225 new-chemical-entity drug applications between November 21, 1997, and December 31, 2009. We excluded six applications whose post-marketing utilization data were unavailable. Table 1 reports the summary statistics for the remaining 219 new chemical entity drugs. By July 31, 2016, at least one serious post-marketing safety action was taken for 34 drugs. Thirty drugs (14%) received post-marketing boxed warnings attributed to safety labeling changes, while 11 drugs (5%) were either withdrawn from the market or had a restriction imposed on their indication due to safety concerns; for seven of these drugs, there was either an antecedent or concurrent addition of a boxed warning to the drug label. One-hundred fifty-three drugs (69%) were approved less than 1 week before the drug’s last-cycle review goal date, 102 drugs (46%) were approved with a priority review designation, and 50 drugs (23%) were approved with a boxed warning.

Table 1.

Descriptive statistics

	Mean/proportion	Count
New drug applications		219
Post-marketing safety outcomes
Safety withdrawal	5%	11
Boxed warning	14%	30
General drug characteristics
Pre-marketing boxed warning	23%	50
Drug primarily approved with an injectable route of administration	30%	66
Natural log (post-marketing volume sales in standardized units)	8.38
Natural log (number individuals studied in clinical trials)	6.84
Number of years the drug is commercially marketed	11.25
Review/designation
Priority	46%	102
Last review cycle deadline characteristics
Review completed 1 week before PDUFA goal date determining approval	69%	153
Review characteristics
Log (drug’s order of entry)	1.5
Log (manufacturer’s accumulated number of drug approvals)	1.7

To measure post-marketing volume sales, we used data from IMS Health, IMS National Sales Perspectives^™, January 1997–December 2015, Retail and Non-Retail Channels, data extracted December 2015

PDUFA Prescription Drug User Fee Act

Electronic Supplementary Material 2 lists the characteristics associated with the 34 drugs with serious post-marketing safety actions. The time on market prior to receiving a safety action varies across drugs. The median time to first action was 3.9 years, with some actions occurring less than 1 year after approval and others occurring up to 12 years after approval. A wide range of safety issues gave rise to the serious post-marketing safety actions. Slightly more than 20% of safety issues were related to rare adverse events that are difficult to detect in preapproval clinical trials (e.g., hepatotoxicity, hypersensitivity, and serious skin reactions), whereas a separate 20% of issues were related to broader class warnings.

Table 2 presents the ORs associated with each drug’s pre-marketing factors on each separate post-marketing safety outcome. The results indicate that there is not a significant relationship between either post-marketing safety outcome and clinical trial size. Drugs with larger clinical trials are modestly more likely to experience a post-marketing safety withdrawal (OR 1.12) or post-marketing boxed warning (OR 1.24), but the result is not statistically significant.

Table 2.

The relationship between adverse post-marketing safety outcomes and time to review goal date (1 week)

Post-marketing safety outcome	Drug withdrawn after approval due to safety concerns			Drug ever received a post-marketing boxed warning
Explanatory variables	Odds ratio	95% CI	p value	Odds ratio	95% CI	p value
Log (number individuals exposed to drug in clinical trials)	1.12	[0.35, 3.552]	0.85	1.24	[0.726, 2.127]	0.43
Review completed 1 week before last-cycle review goal date	0.26	[0.033, 2.098]	0.21	0.89	[0.323, 2.449]	0.82
Drug underwent the priority review designation	1.19	[0.116, 12.187]	0.88	3.51**	[1.02, 12.106]	0.05
Log (post-marketing volume sales in standardized units)	1.31	[0.796, 2.155]	0.29	1.22*	[0.961, 1.555]	0.10
Log (drug’s order of entry)	0.98	[0.159, 6.044]	0.99	0.96	[0.422, 2.197]	0.93
Log (manufacturer’s accumulated number of drug approvals)	2.82	[0.715, 11.113]	0.14	1.40	[0.82, 2.378]	0.22
Route of administration injectable	4.03	[0.192, 84.177]	0.37	0.43	[0.085, 2.16]	0.31
Drug approved with boxed warning	0.74	[0.04, 13.614]	0.84	3.88***	[1.32, 11.399]	0.01
Number of years the drug is commercially marketed	1.51**	[1.008, 2.265]	0.05	1.04	[0.892, 1.2]	0.65
N	219			219

^***Statistically significant at the p<0.01 level;

^**Statistically significant at the p<0.05 level;

^*Statistically significant at the p<0.10 level (all tests are two-tailed)

There is no significant relationship between review time windows and post-marketing safety outcomes in our sample period. Drugs approved within 1 week before their last-cycle review goal dates were less likely to be withdrawn from the market due to safety concerns than drugs approved either after their last-cycle review goal dates or whose approval occurred more than 1 week before their last-cycle review goal dates. The OR is 0.26, and it is not statistically significant. Drugs approved within 1 week before their last-cycle review goal dates were also less likely to receive post-marketing boxed warnings than drugs approved at some other time. The OR is 0.89, but is not statistically significant.

Two pre-marketing factors are strongly associated with the addition of a post-marketing boxed warning: whether the drug was initially approved with a boxed warning (OR 3.88, 95% CI 1.32–11.40), and whether the drug underwent a priority review versus a standard review (OR 3.51, 95% CI 1.02–12.11). The latter result is consistent with findings published in previous papers [8, 9].

There is also a moderate relationship between the number of years a drug is commercially available and whether it experiences either post-marketing safety outcome. Sensitivity analyses in which the interval between approval and the PDUFA goal was changed from 1 week to 2 months and in which the last-cycle review was used instead of the first-cycle review were conducted and did not change the findings (see Electronic Supplementary Material 3).

4. Discussion

We have found that neither clinical trial sample sizes nor review time windows are associated with either the addition of a post-marketing boxed warning or the withdrawal or restriction in indication of a new-chemical-entity drug. While some have argued that increasing the size of clinical trial populations may enable detection of more serious safety risks prior to approval [2], the results in Supplementary Table 1 indicate that many of the safety issues that gave rise to serious post-marketing safety actions were rare and usually below the limit of detection in most clinical development programs. For example, multiple drugs were issued safety actions due to rare adverse drug reactions that are generally not identifiable in preapproval clinical trials (e.g., hepatotoxicity, hypersensitivity, and serious skin reactions). Other drugs received class-wide warnings. Unlike individual warnings that result from safety data associated with an individual drug, class-wide warnings can arise from data associated with other drugs in the same class, or from a pooled analysis of multiple drugs in the same class. That said, it is important to note that it is possible that clinical trial size is important for the detection of other important adverse drug reactions that do not lead to boxed warnings, safety-related withdrawals, or restricted indications.

Our finding that there is no relationship between review time windows and serious post-marketing safety actions differs from some other studies [8, 9]. One possible explanation is that our study used internal FDA data to measure review time windows, whereas previous studies relied on publically available data, and that these public data sources are more difficult to use to measure time windows. We also studied a more recent time period, during which time review practices reflect the improvements made to the review process over time. The FDA’s current review practices, as outlined under the CDER twenty-first Century Review process, suggest that the organization of reviews and the time each member of the review team has to complete a review, have become more standardized over time [11]. Using the standard review process as an example, the CDER twenty-first Century Review process requires that primary reviewers complete their review more than 1 month before the PDUFA goal date. Therefore, assessing whether a drug approval occurred at or very near to the PDUFA goal in more recent times may not measure time constraints on the review staff, as the drug review has become more highly managed.

Two pre-marketing characteristics are strongly associated with the addition of a post-marketing boxed warning: whether the drug was initially approved with a boxed warning or via the priority review pathway. We attribute these results to the underlying characteristics of drugs approved with either boxed warnings or via the priority review pathway. For example, the priority review is reserved for drugs that are intended to treat a serious condition and that are expected to provide a significant improvement in safety or effectiveness. Many priority review drugs are intended to treat cancers, serious infectious diseases, or rare diseases. In these situations the benefits of the drug can outweigh serious safety risks. The median time from approval to the addition of a post-marketing boxed warning was similar for drugs that underwent priority review as well as for drugs that underwent standard review (3.8 and 3.9 years, respectively), a finding that suggests that the priority review process itself is no more or less likely to result in a serious post-marketing safety outcome. The finding that priority review is not associated with safety-related withdrawals or restricted indications supports this notion.

A similar reasoning probably contributes to the result associated with drugs approved with boxed warnings. These drugs also tend to treat serious conditions with the expectation of providing significant improvements in effectiveness. Drugs approved with a boxed warning might also receive more post-marketing scrutiny and vigilance, and, as a result, individuals might be more likely to report adverse events for drugs approved with boxed warnings, and to attribute them to these drugs.

That said, the reason underlying the association between a pre-marketing boxed warning and a post-marketing boxed warning is not clear. While we excluded as serious safety actions those boxed warning revisions that were limited to editorial changes, revised terminology, clarifying language, or labeling changes that improve labeling consistency across related drugs with respect to an already existing boxed warning, we did include as serious safety actions revisions to boxed warnings that substantively modified an existing warning (see Electronic Supplementary Material 2). However, reveals that this explanation does not account in large part for our findings.

Like previous studies, we observed a moderate relationship between the number of years a drug is commercially available and whether it experiences either post-marketing safety outcome. Lester et al. indicates that drugs require changes to their safety labels throughout their life span [12].

Our results remain subject to the limitations of our empirical measures. Like other post-marketing safety measures, neither safety withdrawals nor boxed warnings capture all the important and serious safety information associated with each drug. Further, the FDA uses multiple sources of information to determine whether to withdraw a drug or require a boxed warning. Because these sources vary with respect to quality, some post-marketing safety outcome decisions are based on higher quality information than others, and thus might better reflect actual safety issues. Previous papers acknowledge these limitations [5–9]. A possible solution to this limitation is to study the relationship between various pre-marketing factors and every type of safety labeling change, or to construct new measures that more fully capture the range of safety issues that make up a drug’s safety profile.

An additional limitation is that we did not account for other important pre-marketing and post-marketing information. Important pre-marketing information includes the nature of the population studied during the clinical trials and which the drug is intended to treat, and various comorbidity and drug safety information reported in the pre-clinical, pharmacology, and toxicology databases. Important post-marketing information includes the extent of comorbidities and polypharmacy in the population using the medicine and the nature of the safety data that were used to support serious safety actions (e.g., for post-marketing safety actions resulting from adverse event reports or other types of data, such as observational studies or clinical trials). Such factors, which would need to be studied at the level of the individual medicine, may be more important than those we measured.

5. Conclusion

Our study revealed that, on a system-wide level, a new drug’s clinical trial sizes and review time windows are unassociated with whether new drugs ever required a post-marketing boxed warning or market withdrawal. A more direct approach involving individual examination of the relationship between each drug that eventually required a safety withdrawal or boxed warning and its pre-marketing development program and review processes may identify areas for further improvement in those programs and processes.

Key Points.

Clinical trial sizes, review pathways, and review time windows are unassociated with whether a new drug required a post-marketing boxed warning or marketing withdrawal.

Drugs approved with a pre-marketing boxed warning are 3.88 times more likely to receive a post-marketing boxed warning than drugs approved without a boxed warning.