Detecting Signals of Dietary Supplement Adverse Events from the CFSAN Adverse Event Reporting System (CAERS)

Abstract

Dietary supplement adverse events are potentially severe, yet knowledge regarding the safety of dietary supplements is limited. The CFSAN Adverse Event Reporting System (CAERS) contains records of adverse events attributed to supplements and is potentially useful for dietary supplement pharmacovigilance. This study investigates the feasibility of mining CAERS for dietary supplement adverse events as well as for monitoring the safety of dietary supplement products. Using three online resources, we mapped products in CAERS to their listed ingredients. We then ran four standard signal detection algorithms over the ingredient-adverse event and product-adverse event pairs extracted from CAERS and ranked the detected associations. Comparing 130 signals detected by all four algorithms with a dietary supplement resource, we found evidence for 73 (56%) associations. In addition, some detected product-adverse event signals were consistent with product safety information. We have made a database of the detected adverse events publicly available at https://github.com/zhang-informatics/DDSAE.

Introduction

Dietary supplements (DSs) are commonly consumed in the United States and across the globe despite limited knowledge about their safety and efficacy. According to data from the National Health and Nutrition Examination Survey, the age adjusted consumption of DSs has gradually increased, both in male (28% to 44%) and female (38% to 53%) populations predominantly among adults aged ≥60 years where 70% have reported using one or more DSs^1–3. Although DSs are regulated by the Dietary Supplement Health and Education Act of 1994, these regulations are less strict than those covering drugs and conventional foods⁴. Still, the adverse events (AEs) associated with the consumption of DSs could be severe and are implicated in approximately 23,000 emergency department visits and 2,000 hospitalizations in the United States annually, according to 10 years of data from 63 hospitals⁵. Yet the use of DSs is often self-initiated rather than based on clinician recommendations. Because DSs do not undergo the same clinical trials as drugs, post-market surveillance is important for the identification of AEs for various products.

In 2004, the US Food and Drug Administration’s (FDA) Center for Food Safety and Applied Nutrition (CFSAN) initiated the CFSAN Adverse Event Reporting System (CAERS) to monitor the safety of DSs, foods, cosmetics, and other products⁶. Like the Vaccine Adverse Event Reporting System (VAERS)⁷ and the FDA Adverse Event Reporting System (FAERS)⁸, which accept reports of AEs related to vaccines and drugs, respectively, CAERS is a main source for post-market safety surveillance for DS products. Reports can be submitted by manufacturers, health care professionals, and the public. The reported AEs encompass both major (e.g. death, hospitalization) and minor events (e.g. complaints about taste or color, defective packaging).

Previous studies have demonstrated the feasibility of mining adverse event reports to detect AEs associated with drugs and vaccines. Sakaeda et. al. detected multiple statin-associated AEs in FAERS, suggesting the need for further clinical studies⁹. Xu and Wang detected around 200 cardiovascular AEs in FAERS that were not present in FDA drug labels¹⁰. Tatonetti et. al. developed two databases of drug AEs and drug-drug interactions, respectively, mined from FAERS reports^11,12. For vaccines, studies by both Niu et. al. and Davis et. al. were able to successfully detect intussusception as an AE associated with rotavirus vaccines in VAERS, only a few months after cases were first reported^13,14. For DSs, Timbo et. al. did an initial analysis of CAERS, showing that the reporting of dietary supplement adverse events was positively impacted by the 2006 Dietary Supplements and Nonprescription Drug Consumer Protection Act¹⁵. Wallace et. al. reviewed the AEs associated with a few common botanical supplements, finding that CAERS may underrepresent AEs associated with DSs¹⁶. Sharma et. al. developed a method for finding DSs in FAERS reports, in which some over-the-counter drugs contain similar ingredients to DSs¹⁷. Haller et al. investigated dietary supplement adverse events by recording and reviewing calls to a poison control center, including laboratory analyses of implicated supplements and case reviews by an expert panel¹⁸. Also, Attipoe et al. developed a tool to allow consumers to estimate their risk of taking a given DS. Despite these studies, to our best knowledge, there is no prior study investigating the feasibility of using CAERS to detect dietary supplement adverse events.

This study investigates the potential for mining CAERS for dietary supplement adverse events (DSAEs hereafter) both for individual ingredients ingredients and multi-ingredient products. For single ingredients, we used existing DS resources to map products in CAERS to their listed ingredients. Also using these resources, we developed a thesaurus of DS ingredients allowing us to map different synonymous ingredient strings to a standard form. We then ran standard signal detection algorithms over the CAERS database at both the ingredient and product level to detect significant DSAEs. We have made a dataset of the detected ingredient-adverse event and product-adverse event associations available at https://github.com/zhang-informatics/DDSAE.

Methods

Figure 1 gives an overview of the signal detection system. We merged three existing DS resources (details below) to create a mapping from DS products to their listed ingredients as well as a thesaurus of normalized DS ingredient names. We then mapped the DS products reported in CAERS to their normalized ingredients and generated a set of ingredient-AE candidate pairs from CAERS. We also extracted the DS product-AE candidate pairs directly from CAERS. We input both these sets of candidate pairs into a signal detection system, which finds significant associations among the candidates. We then evaluated a subset of these detected signals by manually comparing them with an evidence-based online DS resource, the Natural Medicines Comprehensive Database (NMCD)¹⁹.

Figure 1:

Overview of the signal detection process using the CFSAN Adverse Event Reporting System (CAERS). NMCD: Natural Medicines Comprehensive Database. DSLD: Dietary Supplement Label Database. LNHPD Licensed Natural Health Products Database.

Preprocessing CAERS data: We used the publicly available CAERS dataset which contains reports from January 2004 to September 2017. Each report contains the date filed, the age and sex of the patient, the name of the suspected product, the type of the product, and a list of adverse events coded according to the Medical Data Dictionary for Regulatory Activities (MedDRA)²⁰. For this study we only used reports for DS products (industry code 54 in CAERS). While duplicate reports can cause spurious associations to be detected if not addressed, the FDA removes duplicates from CAERS prior to release and thus deduplication of reports was not necessary⁶. However, we did notice that in some reports the same AE was listed multiple times. We therefore preprocessed CAERS by removing these duplicated AEs.

Using online resources to generate the DS ingredient thesaurus: To get the mapping from DS products to ingredients and to create the thesaurus of ingredient names, we used three existing DS resources: The Natural Medicines Comprehensive Database (NMCD)¹⁹, the Dietary Supplement Label Database (DSLD)²¹, and the Licensed Natural Health Products Database (LNHPD)²². These databases contain product information and ingredient monographs for a variety of DSs. While DSLD and LNHPD release publicly available data files, NMCD is only accessible via a web browser. Therefore, we built a web crawler to scrape product and ingredient data from the NMCD website. We merged the ingredient names in each source by matching the ingredient name strings and any synonyms listed by the source monographs to obtain a thesaurus of DS ingredients which map synonymous ingredient names to a standard form.

Mapping DS products to ingredients: To get the mapping from DS products to ingredients, we merged the product information from the three aforementioned source databases by matching the product name strings. We used the existing links from the product information to ingredient monographs in each data source to create a mapping from DS products to their listed ingredients. We then normalized these ingredients using the ingredient thesaurus described above. We noticed that the source DS databases occasionally list the same ingredient more than once for a single product. We therefore removed any duplicated ingredients from each product listing. We then searched for the DS products in CAERS that were present in our products database and mapped them to their listed ingredients. Many products in CAERS are listed as single ingredients, e.g. “Vitamin D”. We mapped these to their normalized form using the ingredient thesaurus, increasing the number of products we were able to map.

Detecting DSAE Signals in CAERS: We employed four signal detection methods often used in pharmacovigilance: the Proportional Reporting Ratio (PRR)²³, the Reporting Odds Ratio (ROR)²⁴, the Gamma Poisson Shrinker (GPS)²⁵, and the Bayesian Confidence Propagation Neural Network (BCPNN)²⁶. All four methods compute the score of a given candidate pair from a 2x2 contingency table, which reports the counts of the cooccurrence of the given DS and the corresponding AE²⁷. PRR and ROR are fundamentally frequentist methods, while GPS and BCPNN are Bayesian. BCPNN is a method for estimating the information component (IC), which is the log ratio of the actual to expected count for a given candidate pair²⁸. We refer the reader to the literature for further details of each method^{23–26,28,29}. All the signal detection methods were implemented in R using the PhViD package³⁰.

For PRR and ROR, a signal was detected if the lower bound of the 95% confidence interval (CI) (hereafter referred to as PRR05 and ROR05, respectively) was greater than or equal to 1^31,32. For BCPNN, the information component (IC) was estimated using 50,000 Monte Carlo simulations and a signal was detected if the 2.5% quantile of the posterior distribution of the IC (IC025 hereafter) was greater than 0³³. For GPS, a signal was detected if the lower bound of the 95% CI was greater than or equal to 2 (EB05 hereafter)³⁴. For all methods, we required there to be at least 3 instances of a pair in the data for it to be considered³².

We obtained candidate pairs for both the products as listed in CAERS and their ingredients as obtained via the product to ingredient mapping. We ran the four signal detection methods over both sets of candidate pairs and thus obtained two sets of detected associations, which were evaluated separately.

Evaluation: A common issue with the evaluation of pharmacovigilance systems is the lack of a suitable gold-standard, making sensitivity and specificity impossible to compute³⁵. While other studies address this issue by using FDA warning labels and literature review, such labels are not present for DSs and the body of literature is much smaller^10,36. To evaluate the signal detection at the ingredient level, we obtained a sample of 130 ingredient-AE associations detected by all four methods and evaluated these against the AEs listed in NMCD ingredient monographs³⁵. NMCD was chosen for the evaluation because it provides manually curated, evidence-based descriptions of AEs for a large number of DSs. Two physicians and health informaticists (RR and TA) with knowledge of the DS domain performed the evaluation. One of four labels were assigned to each detected association according to whether the given AE was present, implicit, not present, or contradictory in the NMCD monograph for the given ingredient. The definitions of each label are given in Table 1. The evaluation of DS product AEs is even more difficult than for ingredients, as AEs are often the result of contaminants in manufacturing and occur within a small timeframe. We therefore performed a qualitative evaluation of the detected product DSAEs by comparing them with FDA recalls as well as by doing a web search for any complaints about the product in question. We then compared the nature of any discovered product recalls and complaints to the AEs detected.

Table 1:

Definitions of the annotations labels for the detected ingredient-adverse event (AE) associations.

Label	Present (P)	Implicit (I)	Not present (N)	Contradictory (C)
Description	The adverse event is explicitly mentioned in the NMCD monograph for the ingredient. This includes synonyms of the adverse event.	The adverse event is not explicitly mentioned in the NMCD monograph, but its existence can be inferred from the information provided.	The adverse event is not present either explicitly or implicitly in the NMCD monograph.	The adverse event is listed in the NMCD monograph as an indication of the ingredient, rather than an adverse event.

Results

After preprocessing CAERS by removing reports that do not contain a DS and removing duplicated AEs, the data contained 49,253 reports, 22,664 unique DS products, and 3,648 unique AEs. From this data we extracted 199,897 DS product-AE candidate pairs. Searching the products in CAERS for those in our DS products database, we found 1,688 unique products in 13,314 reports, which we mapped to 2,398 unique ingredients paired with 2,161 unique adverse events resulting in 389,909 ingredient-AE candidate pairs.

Table 2 lists the number of signals detected by each method for the ingredient-AE candidate pairs. PRR and ROR detected a similar number of signals on both the ingredient and product levels. BCPNN detected the most signals: about 180% more than the other methods at the ingredient level and 25% more at the product level.

Table 2:

The number of detected associations, unique ingredients, unique products, and unique adverse events for each of the signal detection methods.

Association Type		PRR	ROR	BCPNN	GPS
	Number of detected associations (%)	615 (0.16%)	615 (0.16%)	1,761 (0.45%)	149 (0.04%)
Ingredient - AE	Number of unique ingredients (%)	244 (10.18%)	238 (0.92%)	397 (16.56%)	48 (2.00%)
	Number of unique adverse events (%)	188 (8.70%)	191 (8.83%)	345 (15.96%)	51 (2.36%)
	Number of detected associations (%)	1,693 (0.85%)	1,670 (0.84%)	2,120 (1.06%)	732 (0.37%)
Product - AE	Number of unique products (%)	787 (3.47%)	753 (3.32%)	842 (3.72%)	372 (1.64%)
	Number of unique adverse events (%)	355 (9.73%)	358 (9.81%)	390 (10.70%)	243 (6.66%)

The Venn diagram in Figure 2 shows the overlap in detected ingredient associations between the four methods. All the associations detected by PRR, ROR, and GPS were also detected by BCPNN. Furthermore, all but 1 association detected by GPS were also detected by the other three methods. GPS did not detect 452 associations that were detected by the other three methods. 148 associations were detected by all four methods.

Figure 2:

Overlap of the ingredient-adverse event signals detected by each method.

Of the 148 associations detected by all four methods, we found that 10 were actually product-AE associations due to a product-to-ingredient mapping error caused by DSLD. The ingredients for a further 8 associations were not present in NMCD. We removed these associations and thus used a total of 130 detected ingredient-AE associations for the evaluation. Table 3 gives the number of detected associations for each label after evaluation. More than half (56%) of the ingredient-AE associations were evaluated as either present or implicit in the monograph.

Table 3:

Labels of detected ingredient-adverse event associations.

Label	N (%)
Present	17 (13%)
Implicit	56 (43%)
Not Present	53 (41%)
Contradictory	4 (3%)

Table 4 lists example ingredient-AE associations detected by the four methods with their scores. All four metrics generally compute similar scores for each ingredient-AE pair. Note that the scores for Pygeum – Dysuria are high, ranging from 54.041 to 61.087, yet this pair actually represents an indication.

Table 4:

Selected examples of detected ingredient-AE signals, their labels, scores, and the corresponding adverse events in the NMCD ingredient monograph.

Detected Ingredient - Adverse Event	Label	PRR (PRR05)	ROR (ROR05)	BCPNN (IC025)	GPS (EB05)	Corresponding adverse event term in NMCD monograph
Tomato - Abdominal pain	P	3.891 (0.865)	4.042 (0.877)	3.882 (0.708)	3.882 (2.049)	Abdominal pain
Caffeine - Heart rate abnormal	P	8.256 (1.406)	8.287 (1.407)	7.977 (0.931)	7.977 (2.687)	Tachycardia
Free plant sterols - Prostate cancer	N	7.714 (1.341)	7.749 (1.342)	7.517 (0.890)	7.517 (2.365)	AE not in monograph
Hesperidin - Alopecia	N	3.809 (0.935)	4.044 (0.962)	3.801 (0.828)	3.801 (2.473)	AE not in monograph
Boron - Hemorrhage	I	7.427 (1.768)	7.628 (1.787)	7.159 (1.686)	7.159 (5.615)	Hematemesis
Cocoa - Urticaria	I	5.226 (1.100)	5.420 (1.110)	5.211 (0.861)	5.211 (2.596)	Allergic skin reactions and rash
Green tea- dehydration	I	4.753 (1.043)	4.894 (1.053)	4.735 (0.855)	4.735 (2.588)	Diuretics
Zinc - Blood urine present	I	4.091 (1.262)	4.159 (1.276)	3.818 (1.193)	3.818 (3.359)	Acute renal tubular necrosis and interstitial nephritis
Pygeum - Dysuria	C	54.591 (3.342)	61.087 (3.375)	54.041 (1.775)	54.041 (7.782)	People use this for dysuria
Selenium - Urinary retention	C	4.837 (1.231)	4.855 (1.233)	4.516 (1.117)	4.516 (3.230)	People us this for Benign Prostatic Hyperplasia (BPH)

We provide detected AEs for two DS products of interest in Table 5. The first product, Hydroxycut, was recalled in 2009 along with 13 other products under the same brand name after reports of liver injuries, jaundice, and death³⁷. AEs related to the liver made up the majority of AEs detected for Hydroxycut products. The second product, All Day Energy Greens, was found to be contaminated with bacteria that caused gastrointestinal injury, resulting in a class-action lawsuit being filed against the product’s manufacturer in 2016 ³⁸. Again, related AEs made up the majority of AEs detected for this product.

Table 5:

The adverse events detected for two products.

Product	Detected AE	PRR (PRR05)	ROR (ROR05)	BCPNN (IC025)	GPS (EB05)	Note
Hydroxycut	Hepatic Function Abnormal	15.845 (2.127)	16.026 (2.131)	14.923 (1.491)	14.923 (7.561)	In 2009, 14 Hydroxycut products were recalled by the manufacturer after reports of liver injuries, with one case resulting in death37.
	Hepatocellular Injury	14.550 (2.008)	14.698 (2.011)	13.773 (1.362)	13.773 (6.540)
	Jaundice	5.119 (1.226)	5.236 (1.237)	5.033 (1.092)	5.033 (2.703)
All Day Energy Greens	Gastric Disorder	23.987 (2.724)	24.923 (2.744)	22.718 (2.200)	22.718 (14.134)	In 2016, a class action lawsuit was filed against the manufacturer of All Day Energy Greens, after it was found that the drink contained active bacteria that can cause gastrointestinal injury38.
	Diverticulitis	14.095 (1.987)	14.342 (1.993)	13.660 (1.351)	13.660 (6.477)
	Abdominal Pain Upper	6.757 (1.611)	7.273 (1.658)	6.664 (1.514)	6.664 (4.936)

Discussion

We found that 13% of DSAEs evaluated were explicitly mentioned in the corresponding NMCD ingredient monograph and 43% were implicitly mentioned. Combined, more than half (56%) of the DSAEs evaluated were supported by evidence in the monograph, demonstrating the potential for mining DSAEs from CAERS using standard signal detection methods. However, we noticed that a substantial number of detected DSAEs were not simply synonyms of AEs reported in NM monographs: many AEs in CAERS were parent or child concept of AEs in NMCD. For example, abnormal heart rhythm was detected as an AE of Boron, which was matched to tachycardias and atrial fibrillation in the NMCD monograph. Also, the detected AE was often a cause or a manifestation of the AE given in NMCD. For example, convulsions (detected for taurine) can be matched to its potential cause or effect, encephalopathy. Urolithiasis (detected for silicon) could be mapped to its manifestations, dysuria or hematuria in NMCD.

We found that there was a great deal of overlap between the associations detected by each method, despite varying numbers of detections. Similar results were reported in a study mining FAERS³². This suggests that the usefulness of each method depends on the context in which it is used: GPS, having the fewest number of detections, is the most cautious and could be used when specificity is most important; BCPNN, with the greatest number, is the most lenient and could be used when sensitivity is most important. However, this study only evaluated pairs detected by all four methods, so it remains future work to evaluate if the signals detected by BCPNN are often false positives or if GPS misses true signals that are found by the other methods.

We also found that it is possible to detect DSAEs at the product level. While a large number of DSAEs were detected by each method, we noticed that many related AEs for a given product or brand indicates a potential issue. As mentioned above, many of the AEs detected with Hydroxycut products were liver related and many of those detected with All Day Green Energy were gastrointestinal. Additionally, we found that DS products in tablet form are often associated with non-medicinal AEs such as choking, throat irritation, and difficulty swallowing. Leveraging this finding, future work could use MedDRA to group related adverse events together and weight signals according to the number of related adverse events for the same product in order to better rank the detected associations.

A major limitation of this study was the lack of a proper gold standard against which to evaluate the detected associations. This is a limitation of pharmacovigilance studies in general. Knowledge of AEs, whether for drugs or DSs, can only be obtained by extensive clinical research. Nevertheless, the body of knowledge surrounding DSAEs is much less than that for drugs. While nearly half of the detected DSAEs were not found in our evaluation resource, this does not mean that they are not true, but further research is required to determine their status.

Some of the DSAEs detected by our system are questionable. We found 4 AEs that were given under “people use this for” or “effectiveness” sections NMCD ingredient monograph rather reported as an AE. For example, prostatomegaly is actually a use for selenium rather than an AE according to its monograph. Using stricter signal detection methods such as GPS could help alleviate this issue. Also, methods for filtering detected signals have also been shown to improve the precision of the signal detection algorithms. These could be employed in future work to remove these types of DSAEs¹⁰.

Another limitation is that we did not employ any confounder control methods in our analysis. To account for potentially confounding variables, future work could employ the stratification of reports on age and sex or more advanced confounder control techniques as discussed in Low et al.³⁹ However, the public release of the CAERS database provides limited information. Patient demographics are limited to age and sex and there is no information regarding concurrent use of other medications or DS nor regarding the presence of any comorbidities. In addition, the estimated reporting rate to CAERS is 2% for dietary supplement related adverse events¹⁵. This means that the reports are not necessarily representative of the population, which could also impact the reliability of the detected signals.

Conclusion

This study investigated the feasibility of detecting dietary supplement adverse events (DSAEs) and of monitoring the safety of DS products in CAERS reports. We evaluated 130 DS ingredient-AE associations detected by four signal detection methods against ingredient monographs in an online DS resource. 56% of these associations were mentioned in the monographs either explicitly or implicitly. We also found some detected product-AEs signals to be consistent with product safety information. These results show that it is possible to detect DSAEs in CAERS using standard methods. More work is required to better understand which signal detection method is most suitable for this data, both for mining ingredient AEs as well as monitoring the safety of DS products.