A study of agreement between the Naranjo algorithm and WHO-UMC criteria for causality assessment of adverse drug reactions

Abstract

Objectives:

Reliability and usefulness of various adverse drug reaction (ADR) causality assessment scales have not been fully explored. There is no universally accepted method for causality grading of ADRs. In the present study we assessed agreement between the two widely used causality assessment scales, that is, the World Health Organization-Uppsala Monitoring Center (WHO-UMC) criteria and the Naranjo algorithm.

Materials and Methods:

The same observer assessed all ADRs (n = 913) collected between January 2010 and December 2012 using the WHO-UMC criteria and Naranjo algorithm at a tertiary care hospital in India. We found that the most frequently assigned causality category was “possible” with both the scales.

Results:

A disagreement in the causality assessment was found in 45 (4.9%) cases reflecting “poor” agreement between the two scales (Kappa statistic with 95% confidence interval = 0.143 [0.018, 0.268]). The mean time taken to assess causality of the ADR using the WHO-UMC criteria was shorter than that by the Naranjo algorithm.

Conclusion:

This study showed that there is a poor agreement between the WHO-UMC criteria and Naranjo algorithm with the former being less time-consuming.

Introduction

An adverse drug reaction (ADR) is any noxious and unintended response to a drug that occurs at doses normally used in man for prophylaxis, diagnosis, therapy, or modification of a physiological function.[1] ADRs are considered as one among the leading causes of morbidity and mortality.[2]

Causality assessment is the evaluation of the likelihood that a particular treatment is the cause of an observed adverse event.[3] It assesses the relationship between a drug treatment and the occurrence of an adverse event. It is an important component of pharmacovigilance, contributing to better evaluation of the risk-benefit profiles of medicines and is an essential part of evaluating ADR reports in early warning systems and for regulatory purposes.[4,5] Causality assessment of ADRs may be undertaken by clinicians, academics, pharmaceutical industry, and regulators and in different settings, including clinical trials.[6]

At an individual level, health care providers assess causality informally when dealing with ADRs in patients to make decisions regarding therapy. Regulatory authorities assess spontaneous ADR reports, where causality assessment can help in signal detection and aid in risk-benefit decisions regarding medicines.[4] Algorithms, being structured systems specifically designed for the identification of an ADR, should theoretically make a more objective decision on causality. The objective causal assessments are based on four basic principles-temporal eligibility, dechallenge and outcome, rechallenge and outcome, and confounding factors.[7]

A number of algorithms or decision aids have been published including the Jones’ algorithm,[8] the Naranjo algorithm,[9] the Yale algorithm,[8] the Karch algorithm,[8] the Begaud algorithm,[8] the Australian ADR advisory committee[8] and the World Health Organization-Uppsala Monitoring Center (WHO-UMC) criteria.[3] Each of these algorithms has similarities and differences. WHO-UMC system has been developed in consultation with the National Centers participating in the Program for International Drug Monitoring and is meant as a practical tool for the assessment of case reports. It is basically a combined assessment taking into account the clinical-pharmacological aspects of the case history and the quality of the documentation of the observation.[3] The Naranjo criteria[9] classify the probability that an adverse event is related to drug therapy based on a list of weighted questions, which examine factors such as the temporal association of drug administration and event occurrence, alternative causes for the event, drug levels, and previous patient experience with the medication. None of the causality assessment tools have been universally accepted as the gold standard.[8] Hence, the present study was conducted to assess the agreement between the WHO-UMC criterion and Naranjo algorithm-the two widely accepted tools in pharmacovigilance.

Materials and Methods

Ethics

The Institutional Ethics Committee for Academic Research Projects approved the study and also granted waiver of informed consent as the data assessed was anonymized.

Study Procedure

The Central Drug Standard Control Organization ADR reporting forms were used for the collection of ADRs. An intensive ADR reporting was done from January 2010 to December 2012 from the departments of medicine, pediatrics, intensive care units, therapeutic drug monitoring, dermatology, psychiatry, radiotherapy, antiretroviral therapy center and hematology.

The diagnosis of ADRs was primarily based on detailed histories and the correlation between drug intake and the onset of the ADR. One of the authors determined causality assessment using WHO causality assessment scale for each ADR report form. Causality assessment of ADRs obtained with WHO-UMC criteria were categorized into certain, probable, possible, unlikely, unclassified, and unclassifiable. The same author also used the Naranjo algorithm to categorize ADRs into definite, probable, possible, and doubtful.

ADRs were rated by the clinical pharmacologist, initially by the WHO-UMC criteria scale to assess the ADR causality and later by the Naranjo algorithm. The rating of the probability of an ADR depends on: the characteristics of the ADR; the characteristics of the rater (some raters are more reliable than others); the quality of the information (in some ADRs the information is incomplete or lacking, also it varies over time); and finally, it will also depend on the scale used to assess the ADR. Therefore, to make a proper comparison of the two scales, we maintained the three variables constant: the same raters assessed the same reactions and had identical information available.

Statistics

To test the null hypothesis of H₀ : k₀ = 0.1 against the alternative hypothesis H_A : k > 0.1 with significance level (α) of 0.05 and power (1-β) of 0.90 for k₁ = 0.21 (from previous studies), using the formula given by Cantor,[10] that is, the minimum required sample size was 830 ADR cases.

Data are expressed as proportions or percentages of total observations. The agreement between two ADR causality scales was assessed using weighted kappa (κ) test. The κ value ranges from −1 (perfect disagreement) to +1 (perfect agreement). Statistical analysis was performed using GraphPadQuickCalcs software available online at http://graphpad.com/quickcalcs/

Results

We found that out of 913 ADRs, 61.1% were seen in male patients and 38.9% were in female patients. The proportion of patients aged < 18, 18-65 and > 65 years were 15.5%, 80.1%, and 4.4%, respectively. As intensive ADR reporting was done by the medical doctors of our department, all collected ADR reports were complete. The ADRs were evaluated retrospectively for causality assessment using the Naranjo algorithm and WHO-UMC criteria. The most frequently assigned causality category with Naranjo algorithm and WHO-UMC criteria was “possible” (99.2% and 93.9%, respectively), followed by “doubtful” (Naranjo, 0.6%) or “unlikely” (WHO-UMC, 3.8%) [Table 1].

Table 1.

Causality category-wise distribution of adverse drug reactions (ADRs) reported at a tertiary care hospital in India

The agreement between the two scales was the highest for “possible” category (94.5%) and no agreement at all for “certain” category. Overall disagreement in causality assessment was found in 45 (4.9%) cases. However, there was “poor” agreement between Naranjo and WHO-UMC (Kappa statistic with 95% confidence interval = 0.143 [0.018, 0.268]) [Table 2].

Table 2.

The percentage agreement of causality assessment between Naranjo algorithm and World Health Organization-Uppsala Monitoring Center causality criteria

The mean time taken to assess causality of an ADR using the WHO-UMC criteria and Naranjo probability scale was 4.1 ± 0.27 and 10.32 ± 1.05 min, respectively.

Discussion

In the present study, there was “poor” agreement between Naranjo algorithm and WHO-UMC criteria.

The causality assessment has pivotal role in clinical practice as well as in drug development. Three approaches are mainly used to assess the causal relationship between drug treatment and the occurrence of adverse events: expert judgment (global introspection), probabilistic approaches, and algorithms.[11] Studies have shown that there is a lot of variation in between rater and within rater decisions on causality of ADRs; this applies both to pharmacologists and physicians.[12,13]

The percentage disagreement (discordance) in causality assessment between the Naranjo algorithm and WHO-UMC criteria was lower in the present study (4.9%; κ = 0.145) compared with that by Rehan et al.,[14] (31%; κ = 0.214), Son et al.,[15] (45%), Macedo et al.,[4] (51%; κ = 0.23) and Lei et al.,[16] (84.9%). However, the observed differences between the present study and earlier studies could be because of subjective assessment inherent in many methods of ADR assessment. The Council for International Organizations of Medical Sciences (CIOMS) ⁄ Roussel Uclaf Causality Assessment Method (RUCAM) scale, used by many expert hepatologists, researchers, and regulatory authorities, is based on the international drug-induced liver injury consensus criteria. The causality score ranges from −8 to 14 and categorized as highly probable (>8), probable (6-8), possible (3-5), unlikely (1-2), and excluded (<0).[17] Garcia-Cortes et al., have shown that in cases of hepatotoxicity, the concordance between the Naranjo scale and the CIOMS-RUCAM scale was 24% (kw = 0.15), and that the former scale lacked validity and reproducibility in the attribution of causality.[17]

Pere et al.,[18] also pointed out disagreements between the different algorithms in assessing causality for the same ADR reports. Disagreements was considerable for three major criteria: timing of event, dechallenge, and alternative etiologic candidates.[18] In the present study, few ADR cases were evaluated using Naranjo algorithm for parameters like objective evidence of ADR, ADR to previous exposures, rechallenge with suspected drug, responses to placebo, and the dose adjustment of drugs. Son et al.,[15] and Arimone et al.,[11] demonstrated similar findings.

The results of our study showed that the most common causality category using the WHO-UMC criteria as well as the Naranjo algorithm was “possible” which substantiates the findings by Macedo et al.,[4] and Lei et al.[16] The present study corroborates the findings by Rehan et al.,[14] that the time taken to assess causality using WHO-UMC criteria was shorter than that with the Naranjo algorithm.

The assumption that a drug is or was the cause of an illness may have far-reaching consequences for the current or future treatment of the patient. The decision to stop an effective drug, or the conclusion that a drug or group of drugs is contraindicated for future use, may disadvantage and even endanger patients. In individual patients, the outcome of an assessment algorithm may strengthen a conclusion but should not replace the clinical diagnosis.[19] Since several studies[20,21] accept the categorization of possible or greater as an ADR, then perhaps clinicians simply need to understand how the scales perform at assessing ADRs using this widely agreed upon threshold of an ADR.[22]

However, our study is limited by the fact that only two ADR causality assessment scales were used for assessment of agreement.

Conclusions

Disagreement exists among the WHO-UMC criteria and the Naranjo probability scale, but the former method is simple and less time-consuming.

Future Implications

None of the available ADR causality assessment methods have been validated to give reproducible results and shown agreement with other methods. Though use of more than one algorithm is advisable, this may not be feasible in clinical practice. However, in clinical research use of a specific ADR causality assessment method may impact the safety profile of an interventional drug.

Recommendation

Due to simplicity and ease of use, the WHO-UMC criteria may be preferred over Naranjo algorithm especially by clinicians in day-to-day practice.