Evaluation of inter-rater agreement between three causality assessment methods used in pharmacovigilance

Sir,

Causality assessment is the evaluation of a possibility that a particular treatment is the cause of an adverse event.[1] It plays a key role in pharmacovigilance both toward signal generation and risk benefit evaluation. There are several methods and algorithms for causality assessment, but none is considered the gold standard and use varies from country to country. This is due to inter-individual variation in using the methods as also varying sensitivities and specificities of each method.[2,3] Bayesian methods of causality have greater credence but have lower acceptance as they are rigorous to perform and involve use of complex calculations.[2,4] In the absence of a universally accepted method the present study was carried out to assess the intra- and inter-rater variations among three widely used causality methods—the WHO-UMC method (recommended by the Pharmacovigilance Program of India (PvPI)), Naranjo's algorithm (a widely quoted method in case reports in literature) and the European ABO method (used in European Union to harmonize decision making). To the best of our knowledge, a study comparing these three methods has never been carried out.

The study protocol was deemed exempt from review by the Institutional Ethics Committee. A total of 401 adverse events collected as part of the surveillance of the PvPI between January and April 2013 were assessed independently by three different raters who had at least 3 years of training in the discipline. The WHO- UMC method classifies causality as certain, probable, possible, unlikely, unclassified and unclassifiable, that is, six categories. Naranjo's algorithm categorizes causality into four categories—definite, probable, possible and doubtful. The European ABO system also categorizes causality into three categories—A, B and O.[1,2,3] Taking into consideration these differences, agreement was assessed by collapsing the categories and making them uniform. While comparing the agreement between European ABO method with the other two methods, all scales were classified into three ranks as follows: (a) certain/probable/A (rank 1); (b) possible/possible/B (rank 2) and (c) unlikely/doubtful/unclassified/unclassifiable/O (rank 3). However, an agreement between WHO-UMC scale and Naranjo's algorithm was made using four ranks—(a) certain/definite (rank 1); (b) probable (rank 2); (c) possible (rank 3) and (d) Unlikely/unclassified/unclassifiable/doubtful (rank 4). Ranks were compared with each other and agreement was calculated using Fleiss kappa index.[5,6]

A majority of the ADRs found by WHO-UMC method were possible (39.06%), followed by probable (34.33%) and certain (26.18%). Naranjo's algorithm found 52.54% of the ADRs as probable followed by possible (47.30%). Causality assessment by European ABO method assessed 56.87% of the ADRs as A followed by B (42.86%). When the three methods were compared, the highest inter-rater agreement was obtained with the Naranjo's algorithm (314/401 = 78%). The overall inter-rater agreement in all three methods was high (kappa statistic = 0.69) [Table 1]. Overall, intra-rater agreement was substantial in two raters (kappa = 0.69 and 0.74) and moderate (kappa = 0.66) in the third rater. The highest agreement was observed between Naranjo's algorithm and European ABO method (kappa = 0.74) followed by WHO-UMC scale and Naranjo's algorithm (kappa = 0.69) on pair-wise comparisons.

Table 1.

Intra-rater and inter-rater agreement in three different causality assessment methods

The present study found good inter- and intra-rater concordance between the three commonly used causality methods—the WHO causality scale, the Naranjo's algorithm and the European ABO system. Davies et al.[7] observed a fair agreement (kappa = 0.31) between six assessors using the WHO-UMC method, Naranjo's algorithm and Venulet algorithm which was lower than that observed in the present study. A higher agreement between the WHO-UMC method and Naranjo's algorithm was reported by them which has been corroborated by our observation as well (kappa = 0.61).[7] Arimone et al.[8] found poor agreement between five raters in 31 adverse drug reaction pairs using seven criteria for causality assessment and global causality derived from a questionnaire.[8] Most common causality rating in our study with the WHO-UMC method was “possible” which corroborates with Macedo et al.,[9] while Naranjo's algorithm assessed most common causality rating as “probable” which is in consonance with Sriram et al.[10]

None of the causality assessment methods is either complete or reproducible. Very few methods take into consideration prior probability of the occurrence of an event with actual causal association. Also, categorization of causality into “possibly related” or “probably related” does not practically offer any additional advantage and leads to poor inter-rater agreement. The causality methods used in the present study have their own merits and demerits [Table 2]. We did find a good agreement between the three raters which could be the result of an actual causal association between the drug and the adverse event or due to individual expertise or commission of same error by all the raters simultaneously. However, the presence of confounding variables like underlying disease, concomitant medications, and absence of de-challenge/re-challenge information could have prevented the achievement of complete agreement.

Table 2.

Strengths and limitations of the three causality assessment methods

Although the WHO-UMC scale is widely accepted and has been recommended by the National Pharmacovigilance Programme of India, we obtained the highest inter-rater agreement with Naranjo's algorithm. Additionally, it had a very good agreement with the other two methods as well. Hence, Naranjo's algorithm can be considered as a preferred method in individual case causality ascertainment because it is more objective and less dependent on personal expertise.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.