Revisiting the reported signal of acute pancreatitis with rasburicase: an object lesson in pharmacovigilance

Abstract

Introduction:

There is an interest in methodologies to expeditiously detect credible signals of drug-induced pancreatitis. An example is the reported signal of pancreatitis with rasburicase emerging from a study [the ‘index publication’ (IP)] combining quantitative signal detection findings from a spontaneous reporting system (SRS) and electronic health records (EHRs). The signal was reportedly supported by a clinical review with a case series manuscript in progress. The reported signal is noteworthy, being initially classified as a false-positive finding for the chosen reference standard, but reclassified as a ‘clinically supported’ signal.

Objective:

This paper has dual objectives: to revisit the signal of rasburicase and acute pancreatitis and extend the original analysis via reexamination of its findings, in light of more contemporary data; and to motivate discussions on key issues in signal detection and evaluation, including recent findings from a major international pharmacovigilance research initiative.

Methodology:

We used the same methodology as the IP, including the same disproportionality analysis software/dataset for calculating observed to expected reporting frequencies (O/Es), Medical Dictionary for Regulatory Activities Preferred Term, and O/E metric/threshold combination defining a signal of disproportionate reporting. Baseline analysis results prompted supplementary analyses using alternative analytical choices. We performed a comprehensive literature search to identify additional published case reports of rasburicase and pancreatitis.

Results:

We could not replicate positive findings (e.g. a signal or statistic of disproportionate reporting) from the SRS data using the same algorithm, software, dataset and vendor specified in the IP. The reporting association was statistically highlighted in default and supplemental analysis when more sensitive forms of disproportionality analysis were used. Two of three reports in the FAERS database were assessed as likely duplicate reports. We did not identify any additional reports in the FAERS corresponding to the three cases identified in the IP using EHRs. We did not identify additional published reports of pancreatitis associated with rasburicase.

Discussion:

Our exercise stimulated interesting discussions of key points in signal detection and evaluation, including causality assessment, signal detection algorithm performance, pharmacovigilance terminology, duplicate reporting, mechanisms for communicating signals, the structure of the FAERs database, and recent results from a major international pharmacovigilance research initiative.

Introduction

There is an understandable interest in methodologies that expeditiously detect credible signals of drug-induced pancreatitis [Hauben and Reich, 2004]. A case in point is the reported signal of acute pancreatitis with rasburicase that emerged from a study by Harpaz and colleagues of combining quantitative signal detection findings from spontaneous reporting system (SRS) data with that of electronic health records (EHRs) [Harpaz et al. 2013], herein referred to as the ‘index publication’ (IP).

The authors of the IP attempted to boost signal detection (SD) performance by overlapping positive data-mining findings from SRS and EHRs [Harpaz et al. 2013]. In a manner of speaking, this represents performing the well established SD and initial signal evaluation (SE) steps concurrently.

The study relied on disproportionality analysis (DA) [Hauben and Bate, 2009; Hauben and Noren, 2010] applied to both SRS data and EHRs. DA calculates observed to expected reporting (O/E) frequencies, proportions, or odds, for drug–event combinations (DECs). An O/E exceeding a predefined threshold was interpreted in the IP as a ‘signal’. DA of SRS data is an accepted option in the pharmacovigilance (PhV) toolkit but has not lived up to its initial hype [Hauben and Noren, 2010]. The events studied in the IP were the Medical Dictionary for Regulatory Activities (MedDRA) Preferred Term (PT) Pancreatitis acute (in SRS data) and elevated pancreatic enzyme related codes (in EHRs).

The IP has a focused discussion of one signal emerging from the analysis involving rasburicase, a recombinant form of uric acid metabolizing enzyme uric acid oxidase (indicated for tumor lysis syndrome in patients treated with cytotoxic medicines for hematological malignancies) [Cammalleri and Malaguarnera, 2007], and the event acute pancreatitis (also described in the IP as a signal involving the event ‘elevated pancreatic enzymes’). The reported signal is noteworthy, being initially classified in the IP as a false-positive finding but then reclassified as a ‘clinically supported’ signal, possibly due to a serendipitous discovery of a published case report [Bauters et al. 2011]. The IP reports a paper in progress with detailed clinical analysis of the EHR-based case series confirming this reclassification. It seems to be a crown jewel of PhV: a prospective validated, real-world signal.

To increase our working knowledge of quantitative PhV we often perform DA exercises based on published literature, including attempts to replicate findings [Hauben and Hung, 2015]. Doing this for the IP, stimulated discussions between the authors on multiple topics in SD and SE. Because these discussions seemed novel, interesting and/or worthy of reinforcement, the current paper has dual objectives: to revisit the signal and extend the analysis with alternative analytical choices and more contemporary data; and to provide an object lesson in PhV SD and SE, including recent findings from a multinational research initiative. The focus is on the SRS component of the IP’s analysis since we could not access the EHRs.

Methodology

The IP defined a signal from the SRS data as a suspect drug-MedDRA PT pancreatitis acute having an EB₀₅ of at least 2 in a deduplicated extract of the FAERS database containing reports through the third quarter of 2010 (3Q2010). The EB₀₅ is the lower fifth percentile of a range of O/Es calculated by one variation of DA known as the multi-item gamma Poisson shrinker (MGPS) that very roughly attempts to reduce noise by shrinking O/Es towards a global mean O/E [Hauben and Bate, 2009; Hauben and Noren, 2010]. A DEC exceeding the latter threshold in SRS data combined with an elevated odds ratio (OR) for that drug and elevated pancreatic enzymes in the EHRs formed another set of signals. The IP briefly refers to additional thresholds tested (EB₀₅ > 1.5, 2.5) but these were introduced in the results rather than the methodology section, so they may define post hoc analysis thresholds.

We adhered to the above methodology for the SRS data using the same dataset, software, vendor, PT, and metric /threshold combination (EB₀₅ ⩾ 2) for baseline analysis. Our baseline results, reported herein, prompted supplementary analysis using alternative analytical choices, including examining a more comprehensive set of individual pancreatitis-related PTs; using additional forms of DA, including the proportional reporting ratio (PRR) [Evans et al. 2001], and the reporting OR (ROR), the disproportionality metric applied in the IP to EHR data; investigating the subsequent evolution of the reporting of the DEC and associated DA using more contemporary FAERS data (through the first quarter of 2015); using unstratified data; including suspect plus concomitant drugs; and retaining clinical trial reports.

We performed a comprehensive literature search to identify additional published case reports, including the EHR-based case series reportedly in progress, given the clinical support from the experts’ case reviews as per the IP. We searched the following databases: OVID MEDLINE® (1946 to present), OVID MEDLINE® (In-Process), BIOSIS Previews (1969 to week 5 2016), Derwent Drug File (1964 to week 52 2015), Embase Daily Alerts (26 October 2015 to 31 December 2015), and Embase (1974 to 31 December 2015). We used a combination of keyword/string (‘pancreat’ and ‘rasburicase’) and Medical Subject Heading (MeSH) searching combined with Boolean intersection search operator without truncation. Abstracts of all articles retrieved by the aforementioned literature search were reviewed for clinical relevance.

Results

We were unable to replicate positive findings (i.e. three reports with EB₀₅ ⩾ 2) from the SRS data using the same algorithm, software, dataset and vendor specified in the IP. In fact, using deduplicated data, as per the IP, there are only two, not three reports coded as ‘pancreatitis acute’ in the specified dataset. While there was a third report of ‘pancreatitis acute’ in the fourth quarter of 2010 (beyond the time frame specified in the IP) the EB₀₅ for the DEC did not exceed and never subsequently exceeded the threshold of EB₀₅ equal to 2 or higher. The EB₀₅ never exceeded 2 in any supplementary analysis. This reflects a shrinkage adjustment by MGPS evidenced by an unadjusted O/E of 17.10 and a corresponding shrinkage adjusted O/E of 2.56 with corresponding EB₀₅ of 0.75 (in one analysis as an example). The DEC was persistently highlighted in default and supplementary analysis using the PRR-based [Evans et al. 2001] (as soon as the third report was entered in the fourth quarter of 2010) and OR-based protocols.

The two cases in the FAERS dataset analyzed in the IP seemed likely duplicates since both involved men, originated from France, with acute myeloid leukemia indication, event dates 2 days apart, 17 shared recorded suspect and concomitant medications (not all standard chemotherapy protocol drug combinations) with identical reported administration dates, although one of the reports coded additional events related to cholestasis (a potential etiology of pancreatitis). If so, it escaped the vendor’s deduplication algorithm.

Our literature search returned 16 references. A review of the corresponding abstracts did not identify additional relevant case reports.

We did not identify additional reports in FAERS corresponding to the reportedly persuasive cases identified in the EHRs. Therefore, as of our cutoff date, they were not submitted as MedWatch reports, or were submitted but not yet entered into FAERS.

Discussion

The results section in the IP states ‘Based on a set of configuration parameters, the combined system highlighted the association between rasburicase and elevated pancreatic enzymes’. We are not sure what ‘Based on a set of configuration parameters …’ means. In fact, it was the vagueness of this statement that caught our attention in the first place and prompted our investigation. Using the threshold criteria prespecified in the IP for highlighting a DEC in FAERS, we could not reproduce an EB₀₅ of at least 2 for the DEC of interest in default or supplementary analysis. It has been documented that analytical choices, and even choice of software vendor, can affect results of DA [Hauben et al. 2007]. The possibility that analytical choices related to the selection or implementation of a DA algorithm explains the discrepancy is minimized given that we used the same algorithm, dataset and software vendor, and tried multiple analytical choices. Undisclosed post hoc additions or modifications to the IP’s declared methodology could explain the discrepancy but we discourage [Hauben and Reich, 2006] such practices due to lack of transparency and the temptation to retrofit results to pre-existing expectations.

We now introduce a first terminological note. The IP continues the long-term practice of using the term ‘signal’ without further specification. With increasing methods and sources of signal-related information, the need for more precision in the use of the term signal has been noted [Hauben and Reich, 2005; Hauben and Aronson, 2009]. Therefore two fundamental types of signals in PhV are now recognized: a signal of disproportionate reporting (SDR) which does not automatically require formal SE, and a signal of suspected causality (SSC) [Hauben and Aronson, 2009], which by definition entails formal SE.

Now consider the following definition of signal from the Merriam–Webster Dictionary: ‘An event or act which shows that something exists or that gives information about something’. Given this dictionary definition of ‘signal’, an SDR indicates that disproportionate reporting (nothing more) of a DEC exists, for example, a positive result from DA, which may or may not be related to causality. A SSC indicates a suspicion (i.e. a thought that something is possible) of causality exists. Use of the prepositional phrase ‘signal of …’ rather than just ‘signal’, resolves semantic ambiguity and aligns with a common sense definition of signal. An SDR may be elevated to a SSC depending on the clinical context [Hauben et al. 2009].

A basic question leads naturally to object lessons in PhV SD and SE: is the relationship between rasburicase and pancreatitis causal? An answer is elusive. The case series manuscript in preparation in 2013 as per the IP, reportedly providing persuasive evidence supporting the signal in the form of three cases reviewed by pediatric oncology experts to ‘rule out alternative explanations and confounders’, could be enlightening, but we were unable to identify it in the published literature.

Assessing case reports of drug-associated pancreatitis requires careful thought. Even drug–pancreatitis associations cited as being causal may still be controversial [Steinberg and Lewis, 1981; Hung and Abreu Lanfranco, 2014; Tenner, 2014]. Numerous elements must be considered, especially in patients with hematological malignancies including coadministered oncology medications associated with pancreatitis and frequently used to treat hematological malignancies such as L-asparaginase and corticosteroids [Steinberg and Lewis, 1981; Raja et al. 2012; Hung and Abreu Lanfranco, 2014; Tenner, 2014], metabolic derangements afflicting these patients (e.g. hypercalcemia) [Inukai et al. 2007] and even pancreatic leukemic infiltration (though often asymptomatic) [Collado et al. 2011; Skeith et al. 2013]. The published case report [Bauters et al. 2011] cited in the IP has several interesting features, such as the apparently rapid normalization of serum lipase. Although not incompatible with the lower end of the range of this enzyme’s half life, it typically remains elevated longer, raising a question of pre-existing pancreatic inflammation [Matull et al. 2006].

The IP assumes positive dechallenge or rechallenge is probative (‘conclusive evidence’ of ‘drugs confirmed to be causally related’), for establishing reference standards and assessing causality in EHRs. For retrospective SD exercises it is an appropriate criterion to locate DECs of legitimate interest in PhV [Hauben and Reich, 2004, 2006]; however, it is generally not conclusive proof [Grendell, 2011; Tenner, 2014]. It requires judicious interpretation in the context of SE since its persuasiveness is not all that it is cracked up to be and highly situation dependent [Girard, 1987]. Caution is especially pertinent here since rasburicase is used to treat or prevent a complication of cytotoxic, and often potentially pancreatotoxic, chemotherapy, often in poly-drug regimens or repeated administration cycles. However, the ability of rasburicase to induce hemolysis in susceptible individuals (e.g. G6PD deficiency) provides potential mechanistic plausibility because hemolysis itself may induce pancreatitis [Saruc et al. 2003; Cook and Bruns, 2012]. So, rasburicase may not be intrinsically pancreatotoxic, but may induce pancreatitis as a complication or epiphenomenon of hemolysis.

If the signal is persuasive, as the IP suggests, the PRR- and ROR-based DA protocols highlighted a credible signal that was missed by the MGPS protocol, the latter described in the IP as an ‘undisputed methodology’ (ambiguous terminology that we have never previously observed applied to DA). What does this say about comparative DMA performance? This question is still valid, though much more contentious in earlier days when issues related to transparency and commercial and intellectual conflicts of interest emerged [Almenoff et al. 2004; Erratum, 2007] alongside aggressive promotion of proprietary software. The latter were of sufficient import to merit an admonition from a major international SD expert working group to consider commercial and intellectual conflicts of interest when interpreting the PhV data-mining literature [CIOMS, 2010].

Though consistent with observations of other investigators that the empirical Bayesian shrinkage employed by MGPS ‘might obfuscate a real signal by reducing it to a non-conspicuous level’ [Suling and Pigeot, 2012], it does not say much about comparative algorithm performance. Regardless of whether the MGPS protocol missed, and the PRR- and ROR-based protocols detected a credible signal (or the opposite), a case study is not adequate to infer absolute or relative DMA performance. This requires systematic testing using multiple metric/threshold combinations in different datasets with a comprehensive reference set of DECs reflecting the quantitative and phenotypic diversity of real-world PhV. A study from the Innovative Medicines Initiative Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium (IMI-PROTECT) may be the latest word on this question for real-world PhV of SRS datasets [Candore et al. 2015]. IMI-PROTECT is a major multinational research consortium composed of experts from academia, industry and health authorities coordinated by the European Medicines Agency [IMI-PROTECT, 2016]. The goal of IMI-PROTECT is development of innovative tools and methods to enhance early detection and assessment of adverse drug reactions. They studied the performance of 15 different algorithms representing specific implementations of five variations of DA (PRR, ROR, Urn, BCPNN and MGPS) [Candore et al. 2015].Applying the latter methods to a reference set of adverse events with 220 drugs in four pharmaceutical companies, one national and two international safety databases, they concluded: ‘The choice of disproportionality statistic does not appreciably affect the achievable range of SD performance and so this can be primarily based on ease of implementation, interpretation and minimization of computing resources’. Even with new varieties of DA, such as p-plots, Ƴ-multinomial modeling and likelihood-based methods [Johnson et al. 2012; Huang et al. 2013; Hauben et al. 2015], the latest data do not support a religious devotion to a single method.

On a related terminological note, description of the relative reporting ratio (RR or RRR) calculated by MGPS as a ‘corrected’ or ‘improved’ ‘estimate’ [Nagel et al. 2016] is problematic on two counts. Individual shrinkage-adjusted DA metrics are not necessarily corrected or improved, in the sense of being more true; nor are they appropriately regarded as estimates.

While the elegance and advantages of Bayesian methods are clear, interpreting individual DA calculations as ‘corrected’ or ‘improved’ is misleading, obfuscates their limitations and belies a misunderstanding of Bayesian theory. Famed statistician Bradley Efron makes clear the fallacy of interpreting any individual EB statistic as ‘corrected’: ‘if you use an empirical Bayesian estimate everything gets pulled towards the central bulge. You have to grit your teeth and believe the fact that event though any one estimate may be off overall you are getting a lot improvement …’ [Holmes et al. 2003]. In other words, shrinkage-based approaches may provide more accurate binary classifications over the long run than frequentist methods but can miss signals, either absolutely or relatively in terms of timing. How good or bad that it is on balance depends on costs and utilities associated with correct versus incorrect signal classifications. We suggest using ‘adjusted’ rather than ‘corrected’ when describing these calculations.

Disproportionality calculations performed on SRS data are not ‘estimates’. DA tabulates ratios of spontaneous report counts. If they are estimates, what are they estimates of? A true underlying ratio of spontaneous report counts? Where does this underlying population of spontaneous reports reside? The SRS database is not a sample of spontaneous reports from an underlying true population of spontaneous reports. It is more properly considered a collection submitted to census. Therefore, the traditional meaning of ‘estimate’, and associated p values and confidence intervals, are dubious, though the latter calculations provide useful and expedient heuristics. In other domains investigators have used the terms ‘pseudo-estimates’ and ‘pseudo-confidence intervals’ to avoid overinterpreting models and outputs [Friedman et al. 2006] and this terminology may be apt for statistical analysis of SRS data. The only possible rationalizations for “estimate” would be related to, first, the fact that regulatory agencies have different rules by which submitted reports are or are not entered into their SRS databases. Thus, the RR could be theoretically considered a relative recording ratio that is an estimate of the relative reporting ratio. Second, it represents an estimate of the ratios of report counts that would be obtained if everyone that experienced an adverse event with any or every drug taken submitted a MedWatch report which were all entered into the database. Both seem a speculative stretch.

More concerning is interpreting RRs as estimated relative risks (which shares ‘RR’ as an acronym). While another recent study from IMI-PROTECT [Macia-Martinez et al. 2016] (with substantial limitations) [Evans, 2016] reported a positive association between PRR values calculated using SRS data and subsequently estimated relative risk in confirmed safety issues, correlation is not synonymous with agreement, and DA metrics should not be misinterpreted or misrepresented as estimated relative risks, which are duly estimated from more robust datasets.

Reclassification in the IP of rasburicase–acute pancreatitis from a false positive to a credible signal underscores the importance of accurate reference standard classifications. It is challenging, though not impossible [Aronson and Hauben, 2006; Hauben et al. 2007], to assemble a 24-karat gold standard in PhV and tarnished gold standards, silver standards and bronze standards must often be substituted. However, the potential impact of imperfect standards on both absolute and relative DMA performance remains underappreciated and underemphasized [Hauben et al. 2016].

The likely duplicate report escaping vendor deduplication has different implications for SD versus initial SE. It is not an isolated phenomenon and underscores that one or more current vendor approaches to deduplication are not foolproof [Hauben et al. 2007]. The impact of duplicate reporting on DA is incompletely defined. ‘Extreme’ duplication [Hauben et al. 2007] of reports of DECs could have substantial impacts via both false-positive (more so) and false-negative (via masking) [Maignen et al. 2014a, 2014b) findings. Reanalysis after selective deduplication of the drug or DEC of interest may suggest potential impacts on DA, but may be misleading [Hauben et al. 2007]. DA is a comparison of a DEC with the database background. Both are vulnerable to duplicate reporting. So, if duplicate reports are more or less randomly distributed, their effect on the corresponding ratio may be minimal. It does potentially impact an early step in SE-clinical review of a relevant case series. The impact here will depend on ease of recognition. Another work stream of the IMI-PROTECT reported that probabilistic record matching, currently implemented as vigiMatch by the World Health Organization’s Uppsala Monitoring Center may do better than conventional rule-based methods [Tregunno et al. 2014].

The IP asserts that this DEC would likely be overlooked due to the low number of reports, supporting the need to augment FAERS with EHRs. This is a dubious assertion. Most organizations [including the Food and Drug Administration (FDA) Office of Safety Evaluation, which the authors include among organizations prone to overlook three reports of pancreatitis) maintain a list of designated medical events (DMEs) that trigger a closer look with as few as one to three cases of a novel event [Hauben and Reich, 2004; Brajovic, 2010]. While organizations are free to establish their own inclusion criteria for such lists and their implementation, DMEs are typically rare, serious and have high drug attributable risk. Pancreatitis and its variants are often included on DME lists, including those used by the FDA. It is unlikely that three reports of pancreatitis would have been overlooked or dismissed out of hand by the FDA or other PhV organizations

The IP further claims examining all signals is not typical practice due to resource constraints. We are unaware of any organization that ignores all but the top k-ranked SDRs. It is standard practice in real-world PhV to examine all SDRs by a first-pass triage procedure to mitigate false-positive findings, including assessment of whether the DEC is literally or conceptually labeled, likely confounding by indication, previously assessed, and its seriousness or public health impacts. The resource intensity of this SDR triage is maximum, with the first DA performed on mature drugs with a large volume of unanalyzed data.

The reported signal highlights available mechanisms for suspected adverse drug reaction reporting. Reports can be submitted to regulatory authorities or first appear in the published literature. The two are not mutually exclusive. If the three reportedly persuasive reports from the EHRs were not submitted as MedWatch forms by the IP’s authors, or the three pediatric oncology experts that scrutinized the EHR cases, it demonstrates the potential delay from reporting via the published literature [Klein and Bourdette, 2013].

Finally, not excluding clinical trial cases as one of our alternative analytical choices highlights a possibly under-recognized feature of FAERS. Default analysis in real-world PhV and the published literature excludes clinical trial cases from DA analysis. However, the default analysis of this particular vendor includes clinical trial cases; that is, the user has to ‘opt out’ of including the clinical trial cases. We have also discovered clinical trial cases recorded as spontaneous reports in FAERS (unpublished data). We do not know the prevalence or distribution of these reports, but it should be kept in mind as an additional potential limitation of FAERS.

In summary, we could not replicate positive findings reported by the IP involving rasburicase and acute pancreatitis in FAERS. We were unable to identify additional case reports of the association in the published FAERS or literature. Going by the brief description of these findings in the IP, the aforementioned manuscript in progress in 2013 as per the IP, could enlighten if it exists or was missed by our search. This paper updates a previously reported finding and presents an interesting object lesson in PhV pertinent to SD and SE, including DA performance, duplicate reporting, signal triage, causality assessment, PhV terminology, adverse event reference standards, safety database structure, and recent findings from a major multinational PhV research initiative. These points are novel, interesting and worthy of reinforcement. We hope our analysis and/or the ensuing discussions contribute to improved design, execution, reporting and interpretation of PhV research.