Serious Adverse Events: A Replicability and Validation Study of Naranjo Causality Assessment Tool in a Canadian Clinical Setting

ABSTRACT

Rationale

Patient safety has become a major concern in healthcare today as 5%–10% of patients experience serious adverse events (SAE) during their hospital stay. The causal assessment of SAE is the responsibility of healthcare professionals (HCP), who use their judgment or a standardize tool. Whether those two methods are replicable to provide similar results remains unclear.

Objective

Our aim was to evaluate if causality assessment performed by HCP is replicable when systematically assessed with the Naranjo tool and to validate its performance in Canadian clinical context.

Methods

We performed pilot retrospective cohort study which included patients with SAE admitted to a Quebec hospital in 2021. Twelve SAE were randomly selected, and two reviewers independently assessed their causality using Naranjo tool. Inter‐rater reliability among two reviewers and between HCP was evaluated. Along with criterion validity, sensitivity and specificity were calculated for validation study.

Results

Weighted kappa was 0.92 (good inter‐rater reliability) where kappa was 0.84 (good agreement between reviewers). No causality assessment by HCP was documented leading to impossibility in computing replicability. The Naranjo tool showed positive monotonic correlation with expert opinion resulting in r _s = 0.208 (p < 0.001). Classification of Naranjo scores to binary variables resulted in sensitivity of 1.00 and specificity of 0.31.

Conclusion

Our study suggested that Naranjo tool is reliable and valid to be used in a clinical setting and was able to classify all drug products involved in the occurrence of SAE. Larger scale studies need to be conducted in real‐time clinical settings to investigate its performance and utility.

1. Introduction

As the pharmaceutical landscape continues to expand with a myriad of new drugs entering the market, their consumption by patients is on the rise [1]. Amidst the quest for heightened efficacy, often‐overlooked aspect is the safety profile of these drugs [2]. Neglecting the safety dimension becomes particularly precarious, given the increasing likelihood of serious adverse events (SAE). Prioritizing patient safety over mere effectiveness underscores the imperative to pinpoint and scrutinize drugs that may pose risks [3]. A tragic chapter unfolded in the annals of the Canadian healthcare system in 2000 when Vanessa Young, a 15‐year‐old girl, succumbed to a SAE triggered by a prescribed drug. This incident prompted the enactment of Vanessa's Law in December 2019, which mandates the reporting of SAE by the hospital when a SAE is observed by healthcare professionals (HCP) [4]. According to Health Canada, if the adverse event is serious enough to result in inpatient hospitalization or prolongation of hospitalization, leads to persistent or significant disability, causes a birth defect, is life‐threatening, or results in death, it is considered as a SAE [5]. This law led to a minimal impact as the major challenge for HCP is to investigate if the exact prescribed drug is indeed the culprit behind the occurrence of a SAE [3]. The crucial step here is to suspect a SAE (causal link) and then to “prove or disprove” it [6]. Causality assessment plays a vital role in this investigation, as it helps HCP to detect a causal link between a SAE and the culprit drug [7]. Different causality assessment approaches have been adapted to bridge this gap [8]. However, there is no causality assessment tool universally accepted as the gold standard [8]. This absence might confer a benefit of doubt to the suspected drug, potentially resulting in the oversight of SAE by HCP, ultimately contributing to an underestimation of reports submitted to Health Canada. The Naranjo causality assessment tool was recognized to be the most exhaustive and easy‐to‐use in different clinical contexts [8] for the adult population and is widely used by different pharmaceutical industries, pharmacovigilance centers, researchers as well as clinicians [7, 8, 9, 10]. Nonetheless, to our knowledge the performance of the Naranjo's tool has never been validated in a clinical setting. The objectives of the study were: (1) to evaluate if the causality assessment performed by HCP is replicable when systematically assessed with a standardized tool, and (2) to validate the Naranjo's tool performance in a real‐time clinical setting.

2. Methodology

2.1. Study Design, Period, and Setting

This was a retrospective cohort pilot study including data set from 2021/01/01 to 2021/12/31. This study was the sub‐study of a larger study conducted at the Institut universitaire de cardiologie et pneumologie de Québec – Université Laval (IUCPQ‐ULaval), including data set from 2018 to 2021 [11]. The IUCPQ‐ULaval was a 338‐bed capacity tertiary care hospital specialized in cardiovascular, pulmonary, obesity, and metabolic diseases [12].

2.2. Study Population With Inclusion and Exclusion Criteria

Adult patients hospitalized at IUCPQ‐ULaval with at least one SAE in 2021 [11]. Patients neither exposed to drug products nor developed any SAE or found to be participating in clinical trials which would linger knowledge of drugs involved were excluded.

2.3. Study Sample Size

As this was a pilot study, simple random selection of 12 SAE from 77 SAE was done [11]. Sample size was calculated using 95% confidence interval (95% CI) at 5% margin of error [13]. The calculation of the sample size was based on an estimation of the statistical precision necessary to meet the main objective of this study [14].

2.4. Data Source and Data Collection

The source of data was the electronic medical records (EMR) collected via Cristal‐Net^TM Platform (https://www.dcicristalnet.com/). Data related to patients' history, comorbidities, hospitalization dates, and SAE were obtained from Research Electronic Data Capture (REDCap^TM) database, which was created for a larger cohort [11]. The pertinent data was analyzed by two reviewers (PP and SC), which included the medication, medical notes, and clinical parameters. Of the two reviewers, one had expertise in working in the pharmacovigilance industry (PP), while the other had expertise in working in a Canadian clinical setting as an HCP (SC).

2.5. Instructions for Causality Assessment Tool for Clinical Settings

The Naranjo tool was utilized to score the suspected drug [8]. In a clinical setting, the patients are mostly administered multiple drugs at the same time interval. Hence, we have accounted few features while performing causality assessment.

• a.When > 1 drug product was suspected, the tool was applied separately to each of the suspected drug product, and the drug with the highest score was considered the causative agent.
• b.Regarding “previous reports”, the drug monograph was scrutinized along with Canada Vigilance adverse reaction online database [15], FDA Adverse Event Reporting System (FAERS) [16], and WHO VigiAccess [17].
• c.Half‐life was considered to be playing a crucial role; 94%–97% of the drug is considered to be eliminated from the body after 4–5 half‐lives [18]. Hence, five half‐lives were considered while assessing the dechallenge and rechallenge.

2.6. Causality Assessment

Twelve SAE files were provided to two reviewers (PP and SC). Each suspected drug was assessed separately by each individual (PP and SC) for their causality. This double assessment was performed to attenuate a potential information bias [19] during the assessment by a single professional.

2.7. Replicability Study

Once 12 SAE files were assessed by both reviewers, we examined the causality assessment performed by HCPs in the EMRs. Subsequently, the causality assessment performed by both reviewers using the Naranjo tool was compared with the causality assessment performed by HCP.

2.8. Validation Study

We analyzed the criterion validity [20]. Since there is no gold standard for evaluating SAE, the expert opinion was considered as “the gold standard”. Regarding expert opinion, we considered an HCP (IC) to provide the final verdict. IC is a pharmacist with several years of experiences in a Canadian clinical setting. We chose an expert to be a pharmacist, as research reports that pharmacist's knowledge and attitude towards SAE reporting are considered as an influential factor [21, 22, 23, 24]. To assess causality, the expert performed assessment of 12 SAE files and each suspected drug involved and the verdict was provided in the form of “Yes”, “Maybe” or “No”. Different values (+1, 0, −1) were assigned as per the verdict. The expert was blinded to the previous assessments performed by the research team. Thereafter, the Naranjo scoring evaluations performed were compared with an expert opinion.

2.9. Statistical Analysis

All analyses were performed with IBM SPSS Statistics for Windows, Version 29.0. Armonk, NY: IBM Corp, released in 2021.

2.9.1. Reliability and Replicability Analysis

Cohen's weighed kappa was calculated to assess the degree of inter‐rater agreement between the two reviewers (PP and SC). The agreement between both reviewers for each score was calculated. Kappa coefficient was used to calculate the reliability of the Naranjo tool and the degree of SAE causality assessment agreement between the research team and the HCP. A value ≥ 0.80 was considered as an adequate match [25]. The results were interpreted according to the statistical significance threshold of p < 0.05% and 95% CI.

2.9.2. Validation Study Analysis

The correlation between the Naranjo scoring of two reviewers (PP and SC) from the research team and an expert opinion (IC) was compared by Spearman's rank correlation coefficient. To do so, the total score of the Naranjo tool was categorized in 3 forms; (1) values of definite and probable category were scored +1, (2) value in possible category was scored 0 and, (3) value in doubtful category was scored −1. Likewise, expert opinion was scored in three categories (yes as +1, maybe as 0 and no as −1). Sensitivity and specificity were carried out to evaluate if the Naranjo tool was able to identify the true drugs involved in causation of SAE.

2.10. Ethical Data Protection and Licensing

We obtained authorization from the Director of Professional Services of the IUCPQ‐ULaval to assess all included patient EMR and received ethical approval from the Ethics Committee of the IUCPQ‐ULaval's Research Center (2021‐3556, 22014).

License (5286751106024) to utilize the Naranjo tool was obtained from the publishers John Wiley & Sons Inc.

3. Results

The 95% CI calculated for sample size ranged from 78% to 89%. A total of 256 drug products were found to be associated with 12 SAEs. The 12 SAEs mainly fell under three of the five criteria named by Health Canada: (1) prolongation of hospitalization (n = 1), (2) life‐threatening (n = 12), and (3) death (n = 1). The sequential diagram for the replicability study is detailed in Figure 1.

Figure 1.

Sequential representation of the replicability study.

3.1. Reliability Analysis

The inter‐rater reliability between PP and SC was found to be significant with weighted kappa 0.926 (p < 0.001). The interclass correlation coefficient was 0.98 (p < 0.001) with an inter‐item correlation of 0.970.

The agreement between PP and SC for each score was calculated and detailed in Table 1. Kappa values depicts agreement between PP and SC. The average kappa statistic shows a high level of agreement of 0.84 (p < 0.001). Scores assigned to drug by PP and SC for each question as well as calculated percentages is showed in Table 2. Several questions (n = 6) were found to be analyzed in a clinical setting whereas some questions (n = 4) remained impractical unrealistic to be analyzed at a clinical setting. These domains are: (1) reappear on placebo, (2) toxicity, (3) susceptibility, and (4) objective evidence.

Table 1.

Agreement between PP and SC.

Naranjo category	Probability	Kappa	95% CI
−1	0.813	0.800	(0.678–0.922)
0	0.935	0.907	(0.785–1.030)
1	0.914	0.908	(0.785–1.030)
2	0.820	0.795	(0.673–0.918)
3	0.896	0.828	(0.706–0.950)
4	0.667	0.648	(0.526–0.771)
5	1.000	1.000	(0.878–1.122)

Table 2.

Distribution of Naranjo score for each drug by PP and SC.

Domains	PP				SC
	+2	+1	0	−1	+2	+1	0	−1
1. Are there previous conclusive reports on this reaction?		217 (85)	39 (15)			205 (80)	51 (20)
2. Did the adverse event appear after the suspected drug was administered?	146 (57)		22 (9)	88 (34)	145 (57)		23 (9)	88 (34)
3. Did the adverse reaction improve when the drug was discontinued, or a specific antagonist was administered?		21 (8)	235 (92)			14 (6)	242 (95)
4. Did the adverse reaction reappear when the drug was readministered?			245 (96)	11 (4)			245 (96)	11 (4)
5. Are there alternative causes (other than the drug) that could on their own have caused the reaction?				256 (100)			3 (1)	253 (99)
6. Did the reaction reappear when a placebo was given?			256(100)				256 (100)
7. Was the drug detected in the blood (or other fluids) in concentrations known to be toxic?			256 (100)				256 (100)
8. Was the reaction more severe when the dose was increased, or less severe when the dose was decreased?		1 (0)	255 (100)			1 (0)	255 (100)
9. Did the patient have similar reaction to the same or similar drugs in any previous exposure?			256 (100)				256 (100)
10. Was the adverse event confirmed by any objective evidence?		256 (100)				256 (100)

Note: Data expressed as n (%).

3.2. Replicability Analysis

We did not find any causality assessment tool used or causality assessment documented by HCP for all 12 SAEs in EMR. As the HCP analysis was considered to be 0, we were unable to calculate Kappa or perform replicability analysis.

3.3. Validation Study

As Naranjo score of PP and SC were found similar (Figure 2), we considered (PP) Naranjo score for this analysis and calculated the correlation between Naranjo score and expert opinion. The correlation between the expert opinion and the Naranjo tool was r _s = 0.208 (p < 0.001) which indicated that it has a positive monotonic correlation with a relatively weak strength.

Figure 2.

Distribution of score by PP, SC and the expert (IC). *N = 256 drug product, Yes = If the drug is involved in causing SAE, Maybe = If the drug might be involved in causing SAE, No = If the drug is not involved in causing SAE.

3.4. Sensitivity and Specificity

We calculated the sensitivity and specificity and categorizing all the score of Naranjo score (PP) and the expert in two forms “Positive” and “Negative”. “Yes” and “Maybe” scores were considered “1”. “No” was categorized as “0” (Table 3).

Table 3.

Sensitivity and Specificity.

		Expert
		Yes	No	Total
Naranjo score	Yes (n)	15	148	163
	No (n)	0	93	93
	Total (n)	15	241	256

Note: *Sensitivity = 15/15 = 1.00 and Specificity = 93/241 = 0.38; **Naranjo Score interpretation (values of Definite and Probable = Yes, Values of possible = Maybe, Values of doubtful‐NO) And Expert Score interpretation (Yes, Maybe, No); ***Sensitivity and Specificity scoring for Naranjo and Expert (values of Yes and Maybe = Yes, values of No = No).

4. Discussion

4.1. Major Findings

In this pilot study conducted in a Canadian clinical setting, we found that the Naranjo causality assessment tool is both reliable and valid. When compared with an expert opinion, the Naranjo's tool was able to identify drugs that caused a SAE with a high probability. Our study suggested highly significant inter‐rater reliability between the two reviewers and a positive monotonic correlation between the Naranjo tool and the expert opinion. Additionally, we observed that a few questions integrated in the Naranjo's tool cannot realistically be answered in a clinical setting leading to low scoring, which may interfere with the interpretation of the causality assessment of a SAE. Moreover, our investigation revealed that the absence of causality assessment in clinical settings granted a benefit of a doubt to the suspected SAEs, potentially leading to failure in reporting these SAEs to Health Canada.

4.2. Prevalence With Other Studies

In 1981 when the Naranjo tool was published (31), it was tested for reliability and validity in 63 cases in clinical trials among six raters. The Naranjo tool showed relatively high inter‐rater reliability (ranging from 0.84 to 0.94) and consensual, content, and concurrent validity [26]. However, when assessed for its reliability in Intensive care unit (ICU) by 4 raters, the Naranjo tool showed poor inter‐rater reliability (ranging from 0.14 to 0.33) [27]. Murali et al. demonstrated that the use of the Naranjo tool in a child‐care unit is limited as most of the questions of the Naranjo tool was scored “unknown” by the raters [28]. Studies also documents that the Naranjo tool lacks validity while assessing causality in hepatotoxicity patients [29, 30]. Nevertheless, Naranjo is widely used tool by different HCP [8, 31] in different clinics and pharmaceutical industries [8, 10] because it is easy‐to‐use and less time consuming when compared to other causality assessment tools [32]. Study shows that the Naranjo tool has better reliability and is valid while processing case files [33]. Murayama et al. (2017) performed their study in Japan daily practice using the Naranjo tool and showed that it was able to categories true adverse drug reaction (ADR) from suspected ADRs with high specificity (0.95) and moderate sensitivity (0.59) [34]. However, to our knowledge, our study is the first‐ever pharmacovigilance study attempting to replicate the causality assessment performed by HCP and to validate the feasibility of the Naranjo tool in a Canadian real‐time clinical setting.

4.3. Is the 1981 Naranjo Tool Feasible to be Used in a Clinical Setting?

While our pilot study emphasizes the replicability and validity of the Naranjo tool for clinical applications, it is imperative to acknowledge certain limitations. In clinical settings, the occurrence of a SAE often coincides with the administration of multiple drug products to patients and may engender suspicion regarding all drugs involved. Additionally, one need to recognize the potential impact of drug‐drug interactions [35], as well as the role of pharmacokinetics and pharmacodynamics while assessing the causality [36]. The Naranjo tool currently lacks coverage in these domains and integrating them could potentially increase the time required for assessment, considering the perpetual busyness of HCP in clinical settings. When assessed in Canadian, Brazilian, Indian and Hungarian clinical settings, Pradhan et al. (2023) documented that among all tools, the Naranjo tool is most exhaustive and easy‐to‐use [8]. While this approach proves practical for evaluating the causality of a single or dual drug association with SAE, it becomes more time‐consuming and cumbersome for HCP in clinical settings when multiple suspected drugs may be involved. Other factors, which need to be considered involve some domain like: (1) reappear on placebo, (2) toxicity, (3) susceptibility, and (4) objective evidence do not play a major role in a clinical setting. The inclusion of the “reappear in placebo” domain in the Naranjo tool stemmed from its development within the context of clinical trials. It is important to note that in clinical settings, placebos are administered most exclusively to patients who consent to participate in research studies, in contrast to the entire patient population. The assessment of “drug toxicity in body fluids” is typically not conducted for all prescribed medications in clinical settings and are restricted to specific drugs with a physician's prescription. The “susceptibility” domain holds significant importance and can substantially contribute to causality assessment. However, our pilot study focused solely on the year 2021, not allowing us to gather information about the patient's past experiences with the drug and/or similar drugs. We anticipate that this domain would prove important in assessments conducted by HCP in clinical settings. Hence, most of the time, those three domains could not be assessed by the reviewers and the score remained the same for all suspected drugs (Table 3). In the “objectivity” domain, the scores remained constant, as the SAE data extraction relied on objective evidence. The signal detection and subsequent causality assessment inherently imply the presence of concrete evidence substantiating the occurrence of a SAE. Consequently, this domain might not play a major in clinical setting.

4.4. Canadian Law on Causality Assessment Versus Role of HCP

In 2019, when the Vanessa law was introduced, it stated that if the HCP observes or suspects a SAE, it is the responsibility of the hospital to report it to Health Canada [4]. The HCP do not need to perform any causality assessment or investigate the causal relationship, although it is encouraged that this information be provided, if available [37]. However, this government law contradicts studies which states that the detection, prevention, reporting, and causal assessment is the responsibility of HCPs [31, 38]. Of importance, the involvement of all HCPs who provide patient care in the SAE assessment process is one key to good post‐marketing safety surveillance [21]. It was documented that, when compared with other groups, HCPs assess the causality better and report a higher proportion of SAE [39]. This is because HCPs play a more active role in patient care within a clinical setting, particularly when confronted with SAE, as they possess a thorough understanding of the patient's susceptibility, comorbidities, and the drugs implicated in a given clinical manifestation. Additional study indicates that a predominant obstacle to SAE reporting across various professions stems from the ambiguity surrounding the causal link between drugs and associated reactions [40, 41]. This trend aligns with findings from prior investigations within the field of pharmacy [42, 43, 44, 45] and nursing [46]. This situation persists despite regulatory authorities merely necessitating a suspicion of a drug's association with an SAE. There is a pressing need to cultivate awareness of the actual causal relationships among HCPs. With the changing era and the growth of artificial intelligence in today's world, an essential approach to enhancing the feasibility of using tools in clinical settings is the integration of machine learning technologies. By assessing causality through machine learning models with pertinent and targeted questions, HCPs can achieve higher specificity and improved reliability, ultimately making the decision‐making process easier for HCP and simultaneously leading to an increase in SAE reports [47, 48].

4.5. Strengths and Limitations

This study is not without limitations. In the field of pharmacovigilance, the term “suspicion” plays a pivotal role, spanning from initial detection to subsequent assessment (36). However, while the computation of sensitivity and specificity, we found it necessary to group the responses of “yes” and “maybe” together, as “maybe” carries a significant weight. Yet, this amalgamation poses a dilemma, as “maybe” cannot be unequivocally aligned with either “yes” or “no.” This nuanced categorization inevitably introduces potential interference and information bias. Another influential factor was the number of suspected drugs involved. While assessing causality, we had to consider all the drugs consumed by patients throughout hospital stay, necessitating a considerable investment of time and effort. The pilot nature could be another limitation of this study, which allowed us to assess only 12 SAEs and hence limiting the involvement to more reviewers. However, despite its pilot nature, this study describes an important milestone by evaluating the replicability and validity of the Naranjo tool, for the first time in the Canadian clinical setting. This study is innovative and stands as the inaugural Canadian clinical setting study that documents how causality assessment can effectively identify drugs involved in causing SAE, concurrently leading to an increased rate of reporting to Health Canada. This investigation conclusively demonstrates that the absence of a mandate for causality assessment in clinical settings results in the oversight of SAE occurrences.

5. Conclusion

With the increase of drug production in pharmaceutical industries, there is an urge to increase the reporting of potential SAEs. This can be possible if the suspected drugs may be assessed for their causality by the HCP in the clinical setting. The knowledge and awareness of causality assessment among HCP must be enhanced, who could become better at recognizing and reporting SAE to Health Canada. The Naranjo tool is found to be reliable and valid in our study. However, large‐scale causality assessment study must be carried out in a real‐time clinical setting to check its performance.

Conflicts of Interest

The authors declare no conflicts of interest.

Supporting information

Annex.

Data Availability Statement

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.