Comparison of agreement and rational uses of the WHO and Naranjo adverse event causality assessment tools

Sir,

Causality assessment of adverse events (AEs) is the standardized and detailed assessment of individual case safety reports for the likelihood of involvement of the suspected drug/s in causing the particular AE. The basic knowledge of causality assessment is indispensable for healthcare professionals as uncertainty of the potential causal relationship between drug and AE remains one of the major reasons of under-reporting in pharmacovigilance.[1] Methods for causality assessment of AEs can be broadly categorized as expert judgment/global introspection (GI), Bayesian methods and algorithms.[2] Despite the availability of a number of methods, no true gold standard exists. The World Health Organization (WHO) GI method[3] and the Naranjo adverse drug reaction (ADR) Probability Scale,[4] although the two most widely used and accepted causality assessment methods in both clinical and experimental settings, have not been validated so far. Hence, it becomes very important to explore the extent to which various methods agree with each other. The comparison of agreement between various methods of causality assessment has been reported by few researchers from the west.[5,6] In a previous study, the agreement between various algorithms and the WHO GI method was reported as 21–56%.[7] However, due to the fact that extent of agreement between two methods may vary in different settings owing to the understanding, judgment and interpretation by experts and personnel assessing the causality, the present study aimed to compare the agreement between the WHO and Naranjo methods in an Indian setting, with a focus to identify the reasons for their mutual disagreement and address peculiar issues related to their practical applicability. To the best of our knowledge, no such comparison has been performed previously in India.

The study was conducted at the Pandit Bhagwat Dayal Sharma Postgraduate Institute of Medical Sciences (PGIMS), Rohtak, Haryana, India, which is a regional pharmacovigilance center under the Pharmacovigilance Programme of India (PvPI). We randomly selected 200 forms from all the ADR proformas collected within the period June 2012–June 2013. Causality assessment was performed by two well-trained independent clinical pharmacologists by applying the two methods – WHO and Naranjo – on each ADR proforma, after which they discussed the causality with each other and discrepancies, if any, were solved. Agreement between the two algorithms was compared using the Cohen's weighted kappa statistic.

For the present study, 200 ADR forms were included. The cases represented a wide spectrum of manifestations, the most common being cutaneous (28%) and gastrointestinal (22%) [Figure 1]. The mean age of the studied population was 35 ± 16 years, with more than 65% males (male/female: 133/67). The number of different branded/generic drugs suspected for causing ADRs was 173. A total of 34 (17%) ADRs were labeled as serious according to the WHO criteria. The use of concomitant medications was present in 108 (61%) cases. All the 200 AEs were probably or possibly caused by the suspected drugs. Causality was probable in 134 cases and possible in 42 cases with both methods. On the other hand, 24 cases were labeled as possible according to the WHO and probable according to the Naranjo algorithm. None of the cases was labeled as certainly/definitely or unlikely to be caused by the suspected drug/s. Full agreement between the two methods was seen in 88% (134 probable + 42 possible) cases while 12% (24) cases had partial agreement [Table 1]. Kappa analysis demonstrated a moderate to good agreement between the two scales (value of kappa coefficient = 0.701).

Figure 1.

Distribution of various adverse events. Miscellaneous included psychosis, renal and metabolic effects

Table 1.

Comparison of agreement between the WHO and Naranjo methods for causality assessment of adverse events

In the 24 discordant cases, positive dechallenge criteria was not met for the following reasons: Dechallenge not performed or information regarding dechallenge lacking or a negative dechallenge, i.e. AE not subsided on drug withdrawal. Hence, according to the WHO, causality was labeled as possible in these cases. But, due to fulfillment of few other criteria, e.g., previous conclusive reports on the reaction, change in reaction severity with dose alteration, presence of an objective evidence and toxic drug concentrations in the blood (e.g., antiepileptics), these were able to attain a score compatible with probable association by the Naranjo algorithm.

Hence, more number of cases were labeled as probable by the Naranjo algorithm compared with the WHO criteria (158 vs 134); a similar observation was previously reported (391 by Naranjo vs. 196 by WHO).[8] Few reasons for disagreement between the two scales as observed can be mentioned. As per the WHO, ADR is labeled as probably caused by the suspected drug if all these criteria are met, i.e., reasonable time sequence to administration of the drug, unlikely to be attributed to concurrent disease or other drugs or chemicals and presence of a clinically reasonable response on withdrawal, i.e., a positive dechallenge. Therefore, in the absence of any criteria, a lower level of causality (possible/unlikely etc.) is labeled. On the other hand, the Naranjo algorithm assigns causality on the basis of total score obtained after evaluating the 10 predefined questions, e.g. if the total score calculated is 5–8, then ADR is probably due to suspected drug. Hence, even if all the criteria as per the WHO are not fulfilled, the causality can be labeled as probable if there are favorable responses to other criteria. A similar scenario can be expected where an AE scores 1–4 on the Naranjo scale and, hence, is labeled as possibly caused by the particular drug. But, due to the absence of mandatory criteria for possible as per the WHO, i.e. reasonable time relationship, causality with this method becomes conditional/unlikely due to drug.

The WHO and Naranjo causality assessment methods are generic in nature and their applicability in different situations has been time tested. There are certain issues in which each method has its own distinct identity. The WHO method takes into account the clinical–pharmacological aspects of the case history and the quality of the documentation of the observation, with a less prominent role of previous knowledge and statistical chance. The Naranjo scale is particularly popular among clinicians because of its simplicity.

Both the WHO and the Naranjo methods are semi-quantitative as they do not exactly measure the likelihood of a relationship. An explicit outcome (unrelated or certain) is given in a minority of cases. Another major issue is a high threshold in assigning a “definite/certain” ADR, which is of paramount importance from a public safety and regulatory point of view. Besides, none of the methods addresses the issue of lack of efficacy/unexpected therapeutic failure, which is also categorized as an ADR (type F).[9] The knowledge of dechallenge and rechallenge holds a high importance when assigning causality by both methods. However, few peculiar areas pertaining to these concepts exist. For example, the concept of dechallenge may not be applicable where the drug is a one-dose treatment (e.g., vaccine), reaction resulting in death and reaction occurring after discontinuation of drug. Also, in certain cases, the evaluation may be tedious, like irreversible or long-lasting reaction, e.g., congenital anomaly, hepatotoxicity, bone marrow suppression with cytotoxic drugs. Moreover, dechallenge cannot be addressed in cases of adverse reactions showing spontaneous recovery despite continuation of therapy. A rechallenge may range from a similar episode in the past to a true planned prospective reexposure. Because of ethical and medical concerns, a true rechallenge is a rarity, particularly in serious reactions. Also, in a subjective reaction, more than one rechallenge may be required to achieve a convincing result. Moreover, the situation during rechallenge (dose or duration of treatment) may not be exactly identical to the original episode.

The Naranjo scale carries a low rate of inter-observer agreement; full agreement between two assessors was observed in only 35% of 106 case reports.[10] One reason for this disagreement might be subjective assessment in a few areas. For example, question no. 1 says: Are there previous conclusive reports on this reaction? This usually means “previous bibliographic description” and is based on judgment and needs clarification if there are few published case reports providing inconclusive evidence. Question no. 7 asks: Was the drug detected in the blood (or other fluids) in a concentration known to be toxic? Although this point is expected to give reproducible results with dose-related (type A) ADRs, the same does not stand true for type B reactions, where, in fact, it is inapplicable. The scale assesses the likelihood of ADR due to a single drug. In cases of multiple drug exposure, it is recommended to apply the scale to each of the possible causes, the most likely being the drug with highest score. However, the scale cannot be applied in AEs arising due to potential drug interaction/s between two or more drugs.

Keeping in mind these practical issues, some areas need to be revised in the causality assessment methods. Because of the practical nonfeasibilty and ethical concerns, should rechallenge should be considered a major criterion for labeling the causality as certain/definite. Certain areas should be addressed, like type B (nondose related) ADRs, lack of efficacy/therapeutic failure as an AE and AEs arising as a result of drug interactions. Few suggestions on rational usage and interpretation of the WHO and Naranjo methods can be made. The presence of a convincing evidence in the report, even in the lack of information on dechallenge or rechallenge with satisfactory outcome, suffices to decide the causal association, e.g., avascular necrosis with corticosteroids, grey baby syndrome with chloramphenicol, etc. Establishing the causal role of a drug and ruling out alternate causes can be strengthened by appropriate laboratory findings and clinical features. Besides, drug interaction-induced AEs demand knowledge of either drugs involved and the basic mechanisms of interactions. Nevertheless, the whole process of assessment and interpretation of causality is quite tedious in the presence of complex real clinical situations.