Existence of Notoriety Bias in FDA Adverse Event Reporting System Database and Its Impact on Signal Strength

Abstract

Background: Notoriety bias is defined as “a selection bias in which a case has a greater chance of being reported if the subject is exposed to the studied factor known to cause, thought to cause, or likely to cause the event of interest.” This study aimed to determine the existence of notoriety bias in the FDA Adverse Event Reporting System (FAERS) database and estimate the impact of potential notoriety bias induced by safety alerts on signal estimation using disproportionality analysis. Methods: Publicly available FAERS data were downloaded and used for analysis. Thirty-one drugs which had label change/safety alert issued by FDA from 2009 to 2013 were considered. These drugs were reviewed 4 quarters before and after the safety alert notification for the existence of notoriety bias. The impact of notoriety bias induced by safety alerts was analyzed by comparing the signal strength using reporting odds ratio (ROR) and proportional reporting ratio (PRR), 2 years before and after the safety alert. Wilcoxon signed rank test was used to determine whether there were a statistically significant difference before and after the safety alert. Results: There was increased reporting for 11 drugs after the safety alert/label change by the FDA. The reporting of 20 drugs decreased or remained unchanged after the safety alert/label change by the FDA. Wilcoxon signed rank test showed that there is no statistically significant difference with respect to the number of reports before and after the safety alert (P = .330, Z = −0.974). Fourteen (45.16%) drugs had an increase in ROR, while 17 (54.83%) drugs had a decrease in ROR after safety alert issued by FDA (P = .953, Z = −0.059). Fourteen (45.16%) drugs had an increase in PRR, while 17 (54.83%) drugs had a decrease in PRR after safety alert issued by the FDA (P = .914, Z = −0.108). Conclusion: Although few FDA safety alert/warnings had a strong and immediate impact, many had no impact on reporting of AE and signal strength. This study found that overreporting due to notoriety bias does not exist in the FAERS database and the overall disproportionality in signal estimates is not altered by the safety alert.

Introduction

Pharmacovigilance systems worldwide rely on spontaneous reporting for the identification of suspected adverse drug reactions (ADRs) and risk assessment because of the limitations of clinical trials (restricted sample size, reduced follow-up, appraisal of surrogate markers) in establishing the safety profile of a drug in the premarketing setting.¹

The unfolding of adverse event (AE) profiles of numerous drugs (e.g., cerivastatin, sparfloxacin, thalidomide, sibutramine, tolcapone)^2-4 after they were approved by the US Food and Drug Administration (FDA) led to the blooming of postmarketing surveillance. The analysis of spontaneous reports reflects the real world scenario, in which patients experienced the primary outcome in a complex pharmacological context.⁵ However, there is a delay in broadcasting the relevant postmarketing AE information which is of significant concern. Such delays, combined with the aforesaid limitations of premarketing clinical trial, emphasize the need for meticulous postmarketing vigilance.

The FDA Adverse Event Reporting System (FAERS) database is designed to support the FDA’s postmarketing surveillance of all approved drugs and therapeutic biologic products. The FAERS encompasses voluntary AE reports from health care professionals and consumers, which are reported either through manufacturers or directly to the FDA.⁶ The database encounters many challenges including underreporting, Weber effect,⁷ and notoriety bias.⁸

With respect to underreporting, it reduces the sensitivity because it underestimates the frequency and thereby impact of the problem. In addition, it makes the system more vulnerable to selective reporting, which may introduce a serious bias.⁹ Various initiatives have been introduced in recent years by FDA, global regulatory agencies, and the healthcare industry to encourage and facilitate the reporting of AEs, such as greater accessibility to the spontaneous reporting database through electronic and online reporting. With regard to Weber effect, it is a substantial increase in the spontaneous reporting of ADRs, particularly during the first 2 years after the initial approval of the drug, which then plateaus and eventually drops.¹⁰ However, the recent study suggests that it may not be a significant factor in the analysis of postmarketing AE data.⁷

Healthcare providers depend heavily on safety information from drug label “inserts” which mainly contain premarketing clinical trial results as FAERS data are not readily accessible.¹¹ To keep healthcare professionals abreast with evolving postmarketing safety risks and help guide prescribing decisions, the FDA issues warnings and safety alerts.¹² The dissemination of an alert can alter the reporting of an event. This phenomenon is called notoriety bias and has been defined as a “selection bias in which a case has a greater chance of being reported if the subject is exposed to the studied factor known to cause, thought to cause, or likely to cause the event of interest.”⁸

Many studies have previously evaluated the impact of FDA alerts on drug use,^13-17 but very few have studied the impact of FDA alerts on AE reporting. The abovementioned studies have either considered a single drug or single class of drug for analysis. This study aimed to determine the existence of notoriety bias in FAERS database and estimate the impact of potential notoriety bias induced by safety alerts on signal estimates using disproportionality analysis.

Methodology

Database

The FAERS is a database that comprises AE reports, medication error reports, and product quality complaints resulting in AEs that are submitted to FDA. The database is primarily designed to assist the FDA’s postmarketing safety surveillance program for drug and therapeutic biologic products. It is in compliance with the International Conference on Harmonization (ICH) issued safety reporting guidance (ICH E2B). The reports in FAERS are evaluated by clinical reviewers, in the Center for Drug Evaluation and Research and the Center for Biologics Evaluation and Research. Reports are submitted to the FAERS database from healthcare professionals, consumers, and manufacturers. The database consists of the following components: Demo file (patient demographics and administrative information), Drug file (details about reported drugs), Reaction file (reactions coded in MedDRA terminology), Outcome file (patient outcomes for the event), Therapy file (drug therapy start dates and end dates), and Indication file (coded in MedDRA for indication of reported drug).

Study Procedure

Publicly available FAERS data were downloaded from FAERS website (https://open.fda.gov/data/faers/). Text file was extracted into excel worksheet for further analysis. “New safety information” which is given by USFDA was considered as the source of safety signal. Thirty-one drugs which had label change/safety alert issued by FDA from 2009 to 2013 were considered in this study. These drugs were reviewed 4 quarters before and after the safety alert notification for the existence of notoriety bias. Normalization of reports was done for ease of graphical representation. The highest count of the report of any of the 8 quarters of the drug was considered as 100 and remaining count of reports of that particular drug was normalized accordingly. The safety alert for biologicals and for drugs which had unspecific safety alert (eg, psychiatric events, skin reactions) were eliminated from the study.

Disproportionality Analysis

The impact of notoriety bias induced by safety alerts was analyzed using reporting odds ratio (ROR) and proportional reporting ratio (PRR).⁵ The ROR is the ratio of the odds of reporting of one specific event versus all other events for a given drug compared with the reporting odds for all other drugs present in the database. The higher the value, the stronger the disproportion appears to be. The PRR involves the calculation of the rate of reporting of one specific event among all events for a given drug, the comparator being this reporting rate for all drugs present in the database (including the drug of interest). The computational formula for calculating ROR and PRR^18,19 is given in Tables 1 and 2. ROR and PRR for the drugs were calculated 2 years before and after the safety alert were issued. Wilcoxon signed rank test was used to determine whether there were statistical differences in signal strength before and after the safety alert.

Table 1.

Method of Calculation of the Signals.

	Drug of interest	Other drugs
ADR of interest	A	B
Other ADR	C	D

Note. ADR = adverse drug reaction; A = the number of reports containing both suspected drug and suspected ADRs; B = the number of reports containing drug of choice but with other ADRs; C = the number of reports containing the event of interest but with other medications; D = the number of reports concerning other medications and other ADRs.

Table 2.

Formula for Computation of Signals.

Measure	Computation
ROR	ROR = (A/B)/(C/D) SE = $\sqrt{\frac{1}{A}+\frac{1}{B}+\frac{1}{C}+\frac{1}{D}}$
PRR	PRR = A(A+C)/B(B+D) SE = $\sqrt{\left(\frac{1}{A}-\frac{1}{A+C}+\frac{1}{B}+\frac{1}{B+D}\right)}$

Note. ROR = reporting odds ratio; PRR = proportional reporting ratio.

Results

The analysis was performed for 31 drugs which had safety alert/label change notification by FDA between 2009 and 2013 (Table 3). The reporting trend between 4 quarters before and after the safety alert, along with the moving average is shown in Figure 1. The moving average was not significantly altered before and after the alert which indicates that the drugs are not affected by the safety alert notification.

Table 3.

Drugs Which Had Safety Alert/Label Change Notification by FDA Between 2009 and 2013 and Difference in Number of Reports, ROR and PRR Before and After Safety Alert.

Drug	Safety alert	Date of alert	Number of reports before	Number of reports after	ROR before	ROR after	PRR before	PRR after
Zoledronic acid	Renal impairment	March 13, 2009	61	91	0.44	0.49	0.44	0.49
Ceftriaxone	Hemolytic anemia	June 7, 2009	5	12	6.45	12.46	6.48	12.74
Varenicline	Visual impairment	July 01, 2009	6	11	1.08	1.12	1.09	1.13
Montelukast	Suicidal behavior	August 19, 2009	322	274	8.77	3.35	9.60	3.41
Didanosine	Portal hypertension	January 25, 2010	21	6	341.80	525.86	470.53	607.94
Trazodone	Prolongation of the electrocardiogram QT interval	February 02, 2010	6	23	4.08	4.32	4.06	4.31
Propylthiouracil	Hepatotoxicity	April 1, 2010	5	1	13.23	38.41	13.04	37.14
Orlistat	Hepatotoxicity	May 25, 2010	104	58	0.50	1.23	0.50	1.22
Solifenacin	Angioedema	July 12, 2010	0	2	0.77	0.71	0.77	0.71
Minocycline	DRESS	September 14, 2010	26	0	45.45	6.96	43.59	6.92
Tapentadol hydrochloride	Serotonin syndrome	November 1, 2010	9	9	19.70	40.31	19.33	39.47
Oxybutynin chloride	Angioedema	January 31, 2011	4	3	1.28	1.70	1.28	1.70
Doxycycline hyclate	Stevens Johnson Syndrome	March 21, 2011	3	3	2.72	6.45	2.73	6.49
Finasteride	Male breast cancer	April 13, 2011	3	5	77.45	37.60	93.67	41.80
Lanthanum carbonate	Intestinal obstruction	April 27, 2011	3	12	0.983	0.52	0.98	0.53
Simvastatin	Muscle injury	June 8, 2011	19	10	9.87	9.01	13.15	10.90
Dutasteride	Male breast cancer	June 9, 2011	4	3	70.33	146.97	83.19	192.56
Leuprolide acetate	Osteopenia	June 17, 2011	14	13	0.59	0.21	0.59	0.21
Dronedarone hydrochloride	Pulmonary toxicity	June 21, 2011	8	5	10.89	29.86	11.17	30.80
Bevacizumab	Osteonecrosis of jaw	September 30, 2011	14	18	4.43	2.52	4.51	2.56
Infliximab	Sarcoidosis	October 26, 2011	20	17	5.67	2.04	6.12	2.09
Voriconazole	Periostitis	November 16, 2011	11	1	224.86	195.93	362.26	301.37
Adalimumab	Optic neuritis	December 26, 2011	19	16	1.72	0.74	1.77	0.74
Solifenacin succinate	Somnolence	January 17, 2012	4	8	0.19	0.28	0.19	0.28
Drospirenone	Pancreatitis	February 13, 2012	6	1	3.73	3.36	3.86	3.44
Pegloticase	Anaphylaxis	April 16, 2012	46	7	42.80	74.69	38.92	65.03
Trospium chloride	Somnolence	July 23, 2012	1	4	0.03	0.01	0.04	0.01
Tolterodine tartrate	Somnolence	August 1, 2012	2	4	1.48	1.18	1.48	1.18
Fesoterodine fumarate	Somnolence	August 1, 2012	18	12	2.55	1.78	2.52	1.77
Dalfampridine	Anaphylaxis	January 22, 2013	3	2	0.55	0.27	0.56	0.27
Lacosamide	Neutropenia	April 17, 2013	3	1	0.79	0.69	0.79	0.69

Note. FDA = US Food and Drug Administration; ROR = reporting odds ratio; PRR = proportional reporting ratio; DRESS = drug reaction with eosinophilia and systemic symptoms.

Figure 1.

Reporting trend of 31 drugs, 4 quarters before and after the safety alert.

Note. Each color represents the reporting trend of individual drug. Pink zone indicates the period during which the safety alert was issued. Thick black line indicates the moving average of reporting trends of all drugs. Q = quadrant; moving average of 31 drugs AE reports; BQ = before the safety alert; AQ = after the safety alert.

Drugs Whose Reporting Increased After Safety Alert

There was increased reporting for 11 drugs after the safety alert/label change by FDA. Drugs included Zoledronic acid (number of reports before: 61; number of reports after: 91), Ceftriaxone (5; 12), Varenicline (6; 11), Trazodone (6; 23), Solifenacin (0; 2), Finasteride (3; 5), Lanthanum carbonate (3; 12), Bevacizumab (14; 18), Solifenacin succinate (4; 8), Trospium chloride (1; 4), and Tolterodine tartrate (2; 4) (Table 3). An average increase of 17.18 ± 31.63 was observed for the abovementioned drugs.

Drugs Whose Reporting Decreased or Remained Unchanged After Safety Alert

The reporting of 20 drugs decreased or remained unchanged after the safety alert/label change by FDA. Drugs included Montelukast (number of reports before: 322; number of reports after: 274), Didanosine (21; 6), Propylthiouracil (5; 1), Orlistat (104; 58), Minocycline (26; 0), Tapentadol hydrochloride (9; 9), Oxybutynin chloride (4; 3), Doxycycline hyclate (3; 3), Simvastatin (19; 10), Dutasteride (4; 3), Leuprolide acetate (14; 13), Dronedarone HCl (8; 5), Infliximab (20; 17), Voriconazole (11; 1), Adalimumab (19; 16), Drospirenone (6; 1), Pegloticase (46; 7), Fesoterodine fumarate (18; 12), Dalfampridine (3; 2), and Lacosamide (3; 1) (Table 3).

Wilcoxon signed rank test was performed to identify the impact of safety alert on reporting trend. It was observed that there is no statistically significant difference with respect to the number of reports before and after the safety alert (P = .330, Z = −0.974) (Tables 4 and 5).

Table 4.

Descriptive Statistics.

	n	Mean	Std. Deviation
Number of reports
Before	31	26.9355	59.53371
After		25.8387	57.98684
Signal strength using ROR
Before	31	29.2003	72.29670
After		37.1373	100.85342
Signal strength using PRR
Before	31	38.6842	104.31360
After		44.5129	121.97943

Note. ROR = reporting odds ratio; PRR = proportional reporting ratio.

Table 5.

Wilcoxon Signed Rank Analysis.

	n	Mean rank	Sum of ranks	Z value	P value
Number of reports
Negative ranks	18a	14.58	262.50	−0.974b	.330
Positive ranks	11c	15.68	172.50
Ties	2d
Signal strength using ROR
Negative ranks	17a	14.76	251.00	−0.059b	.953
Positive ranks	14c	17.50	245.00
Ties	0d
Signal strength using PRR
Negative ranks	17a	14.91	253.50	−0.108b	.914
Positive ranks	14c	17.32	242.50
Ties	0d

Note. ROR = reporting odds ratio; PRR = proportional reporting ratio.

^aAfter < before.

^bBased on positive ranks.

^cAfter > before.

^dAfter = before.

Impact of Safety Alert on Signal Strength (ROR)

Fourteen (45.16%) drugs had increase in ROR after safety alert, while 17 (54.83%) drugs had decrease in ROR after safety alert issued by FDA. Zoledronic acid (ROR before: 0.446; ROR after: 0.499), Ceftriaxone (6.45; 12.64), Varenicline (1.08; 1.12), Montelukast (8.77; 3.35), Didanosine (341.8; 525.86); Trazodone (4.08; 4.32), Propylthiouracil (13.23; 38.41), Orlistat (0.5; 1.23), Solifenacin (0.77; 0.71), Minocycline (45.45; 6.96), Tapentadol hydrochloride (19.7; 40.31), Oxybutynin chloride (1.28; 1.70), Doxycycline hyclate (2.72; 6.45), Finasteride (77.45; 37.6), Lanthanum carbonate (0.983; 0.52), Simvastatin (9.87; 9.01), Dutasteride (70.33; 146.97), Leuprolide acetate (0.59; 0.21), Dronedarone HCl (10.89; 29.86), Bevacizumab (4.43; 2.52), Infliximab (5.67; 2.04), Voriconazole (224.86; 195.93), Adalimumab (1.72; 0.74), Solifenacin succinate (0.19; 0.28), Drospirenone (3.73; 3.36), Pegloticase (42.8; 74.69), Trospium chloride (0.03; 0.01), Tolterodine tartrate (1.48; 1.18), Fesoterodine fumarate (2.55; 1.78), Dalfampridine (0.55; 0.27), Lacosamide (0.79; 0.69) (Table 3). There was no significant difference in the ROR before and after safety alert (P = .953, Z = −0.059) (Tables 4 and 5).

Impact of Safety Alert on Signal Strength (PRR)

Fourteen (45.16%) drugs had increase in PRR after safety alert, while 17 (54.83%) drugs had decrease in PRR after safety alert issued by FDA. Zoledronic acid (PRR before: 0.44; PRR after: 0.49), Ceftriaxone (6.48; 12.74), Varenicline (1.09; 1.13), Montelukast (9.6; 3.41), Didanosine (470.53; 607.94), Trazodone (4.06; 4.31), Propylthiouracil (13.04; 37.14), Orlistat (0.5; 1.22), Solifenacin (0.77; 0.71), Minocycline (43.59; 6.92), Tapentadol hydrochloride (19.33; 39.47), Oxybutynin chloride (1.28; 1.70), Doxycycline hyclate (2.73; 6.49), Finasteride (93.67; 41.8), Lanthanum carbonate (0.983; 0.53), Simvastatin (13.15; 10.9), Dutasteride (83.19; 192.56), Leuprolide acetate (0.59; 0.21), Dronedarone HCl (11.17; 30.8); Bevacizumab (4.51; 2.56), Infliximab (6.12; 2.09), Voriconazole (362.26; 301.37), Adalimumab (1.77; 0.74), Solifenacin succinate (0.19; 0.28), Drospirenone (3.86; 3.44), Pegloticase (38.92; 65.03), Trospium chloride (0.04; 0.01), Tolterodine tartrate (1.48; 1.18), Fesoterodine fumarate (2.52; 1.77), Dalfampridine (0.56; 0.27), Lacosamide (0.79; 0.69) (Table 3). Significant difference was not observed in the PRR before and after safety alert (P = .914, Z = −0.108) (Tables 4 and 5).

Discussion

Our study aimed to determine the existence of notoriety bias in FAERS database and estimate the impact of potential notoriety bias induced by safety alerts on signal estimates using disproportionality analysis. We found that overreporting due to notoriety bias does not exist in the FAERS database and the overall disproportionality in signal estimates is not altered by safety alert. Although few drugs demonstrated increased AE reporting following safety alert, there was no significant increase in the reports except for visual impairment due to Varenicline which had more than 50% increase in the number of reports after the FDA alert. Nevertheless, the overall reporting in the database was not significantly affected.

When we compared the number of reports 4 quarters before and after the safety alert, there was an increase in reporting for few and the others remained unaffected by the FDA alert. Wilcoxon signed rank analysis showed no significant difference between the number of reports before and after the safety alert. This was consistent with the study conducted by Hoffman KB et al, which suggests that modern day AE reporting trends do not appear to be substantially affected by FDA alerts.²⁰ The decreased reporting of AE after FDA alert can be attributed to substantial decline in the use or prescribing of the drugs or increase in the use of other drugs in the same class after the release of information by FDA on the dangers of that particular drug.

Disproportionality in signal strength was assessed using ROR and PRR 2 years before and after the alert. Few drugs showed increase in ROR and PRR after the alert, while it decreased for others. Statistical analysis showed no significant difference in the signal strength before and after the safety alert. This was contradictory to the study conducted by Pariente et al, which concludes that disproportionality in spontaneous reporting databases increases after safety alert.²¹ However, the above study analyzed the consequences of safety alert only on 4 drugs.

Notoriety bias/stimulated reporting of a drug is dependent on numerous factors such as specificity of the warning, seriousness of the alert, traditional media coverage, acceptance of warning/alert/label change, if the guidance increases the work load for the physicians and whether the alert comes with guidance for selection of safer alternatives.

The increasing use of social media sites to disseminate the emerging safety information of licensed medicinal products can have a significant impact on FDA warnings. A study conducted by Faasse et al reported that the traditional news coverage has the potential to considerably increase the overall AE reporting as it increase the anxiety in viewers.²² All the newspaper articles mentioned at least one benefit of the drug; however, only 32% mentioned one harmful effect. None listed harm without mentioning a benefit.²³ Public is much more in-tune with social media and direct-to-consumer advertising since 2009-2013 and extrapolating these results to present day is a question. Further analysis should be done using more recent data for better understanding on the impact of social media on safety alerts.

Another factor which can have a significant impact is the physician’s acceptance and willingness to follow the FDA guidance. Morrato et al compared the frequency of provider contact after FDA advisory of “close supervision” for risk of suicide and found that it was not increased after FDA advisory was issued. This might have occurred as FDA advisory was issued without lack of supporting evidence that close supervision may lead to improved outcomes and physicians believed that their current level of monitoring was sufficient and advisory was overlooked.²⁴ A study found that physicians are unlikely to abide to FDA guidance that requires increased patient contact.¹⁷

The effectiveness of risk communication is under question. A study conducted previously concluded that safety alert can be effective, but not always sufficiently.²⁵ Safety alert can result in both intended and unintended effects. There was decreased number of prescriptions for selective serotonin receptor inhibitors (SSRIs) after safety warning of increased risk of suicidality and suicide thoughts in children and adolescents.²⁶ The unintended effect was decrease in prescription of SSRIs in adults as well.²⁷ This requires adoption of clear and effective risk communication strategies.

In most of the cases, warnings are vague and difficult to interpret. FDA warnings/alerts/label changes are more successful if they are repeated, precise, brief, offer alternatives, communicated through media and newsletters, and come with decision support.²⁸

Limitations of this study were that the impact of confounding factors such as media coverage on FDA warning could not be assessed, drug selection was biased as we did not include all the alerts between 2009 to 2013, and generalizing this result to the entire database might not be appropriate. In the present day scenario, social media and advertising have more influence over public which could have shown an increase in the reporting ratio; however, due to the unavailability of “new safety information” our study includes only data from 2009 to 2013. The reasons for reporting and signal strength being unaffected by the FDA alert could not be evaluated.

Conclusion

Although few FDA safety alert/warnings had strong and immediate impact, many had no impact on reporting of AE and signal strength. This study found that overreporting due to notoriety bias does not exist in the FAERS database and the overall disproportionality in signal estimates is not altered by safety alert. Even though few drugs exhibited increase in reporting, it was not found to be significant. Future studies are required to assess the impact of confounding factors on notoriety bias.