Have Current Systems of Pharmacovigilance Had Their Day?

The safety of a newly approved medication is based primarily on the results of preapproval clinical trials.^1,2 This can be problematic because it means that many medications are approved according to the results of 1 or 2 clinical trials. These trials typically enrol fewer than 1000 participants, who are often healthier than patients in routine clinical care.³ In addition, although preapproval trials may accurately estimate the rate of common adverse events, rare and serious adverse events may go undetected.^4–6 Therefore, postmarketing safety studies are needed to identify potential rare adverse events associated with newly approved medications. This practice is referred to as pharmacovigilance, which the World Health Organization has defined as “the science and activities relating to the detection, assessment, understanding and prevention of adverse effects or any other possible drug-related problem.”⁷

In Canada, pharmacovigilance occurs predominantly through spontaneous reporting to Health Canada.⁸ Adverse reactions can be reported—by patients, health care professionals, or drug manufacturers— to the Canada Vigilance Adverse Reaction Online Database, which is similar to the Adverse Event Reporting System of the US Food and Drug Administration (FDA).⁹ Many adverse drug reactions have been detected through this mechanism; however, there are limitations to this approach.¹⁰ First, reporting is voluntary and thus prone to selection bias. Second, the quality and completeness of reports are highly variable.¹¹ Third, because the total number of patients who received the drug (i.e., the denominator) is not reported, the relative and absolute risks cannot be accurately quantified. For example, in 2012, after dabigatran was approved, spontaneous reports of bleeding associated with this drug greatly outnumbered reports of bleeding associated with warfarin use.¹¹ This might suggest a higher rate of bleeding with dabigatran than with warfarin. However, a cohort study of more than 140 000 patients conducted in response to these reports showed that the rate of bleeding was about 2-fold higher with warfarin than with dabigatran.¹¹ Thus, the higher number of spontaneous reports associated with dabigatran resulted from reporting bias, likely because dabigatran was a new medication and warfarin was not.

An alternative to reliance on spontaneous reporting for pharmacovigilance is the use of data mining. This method uses advanced statistical methods to identify patterns and associations in large data sets. In contrast to a traditional research study, in which the researcher starts with a hypothesis and designs a study to test it, data mining involves a data-driven process in which the researcher “lets the data speak for themselves”. The data sources may include health care databases (e.g., those held by ICES), prospective registries, or electronic health records.¹⁰ Several data-mining approaches exist, including tree-based statistical scanning (described in more detail below), Gamma Poisson Shrinker, and text mining through natural language processing.^12,13

The tree-based scan statistic has been applied in North America. TreeScan, one of the more common data-mining software programs, was developed specifically for pharmacovigilance and was first introduced in 2013.¹² The term “tree” refers to the hierarchical grouping of related diagnostic codes used with this approach.¹² For example, within the International Statistical Classification of Diseases and Related Health Problems (ICD) coding system, the code I21 (acute myocardial infarction) is considered to be a branch. The codes I21.0 (ST-elevation myocardial infarction of the anterior wall) and I21.4 (non-ST elevation myocardial infarction) are considered to be sub-branches. These codes can be further specified to the second decimal digit (e.g., I21.01 for ST-elevation myocardial infarction of the left-anterior descending artery); this terminal level is referred to as a leaf. One strength of TreeScan is that the investigator does not have to specify a priori the outcome of interest or the level of detail of the outcome. Instead, TreeScan evaluates data across all possible branches, sub-branches, and leaves to identify potential adverse events.¹²

Recently, TreeScan was used to identify potential adverse events associated with the quadrivalent human papillomavirus (HPV) and live attenuated herpes zoster vaccines.^14,15 Among 1.9 million people who received the HPV vaccine, and across 6551 potential ICD codes, TreeScan identified only 2 potential signals: cellulitis and complications of the injection.¹⁴ Similarly, in a study of more than 1.2 million herpes zoster vaccinations, TreeScan found that local skin reactions and skin infections were the only statistically significant adverse events.¹⁵

Data mining has several advantages over spontaneous reporting systems. First, it leverages large sample sizes, which allows for the detection of rare adverse events.⁵ Second, it does not require a priori (hypothesis-free) knowledge of a potential association between a medication and an adverse event.¹² This advantage is particularly important given that knowledge of potential adverse events is often limited when a drug first enters the market, and it therefore allows for comprehensive evaluation of all possible adverse events. Third, in the case of TreeScan, all results are adjusted for multiple-hypothesis testing, to limit the number of potential false signals.¹²

Data mining also has important limitations. First, it often uses ICD codes; therefore, associations can be measured only for diagnoses with a relevant ICD code. Second, the validity of ICD codes is variable depending on the diagnosis. Third, data mining produces statistical association signals that may not represent true adverse events (e.g., because of confounding).¹² Therefore, signals detected from data mining should be formally evaluated with directed pharmacoepidemiologic studies (e.g., new-user active-comparator cohort study).⁵

In 2017, the FDA released the Sentinel Initiative: Final Assessment Report, which outlined how the agency planned to modernize the process of postmarketing drug safety surveillance, including through implementation of TreeScan and other data-mining tools.¹⁶ In Canada, the Drug Safety and Effectiveness Network (established by the Canadian Institutes of Health Research) created CNODES, the Canadian Network for Observational Drug Effect Studies, in 2011, which is able to access data for millions of patients across the country. CNODES now plays an essential role by conducting pharmacoepidemiologic studies in response to requests from Health Canada. A natural extension of this work would be the incorporation of TreeScan or another data-mining technique to advance the current process of pharmacovigilance in Canada with the ultimate goal of preventing adverse events.