Determinants of signal selection in a spontaneous reporting system for adverse drug reactions

Abstract

Aims

Detection of new adverse drug reactions (ADR) after marketing is often based on a manual review of reports sent to a Spontaneous Reporting System (SRS). Among the many potential signals that are identified, only a limited number are important enough to require further attention. The goal of this study is to gain insight into factors contributing to the selection and dissemination of possible signals originating from the SRS maintained by the Netherlands Pharmacovigilance Foundation.

Methods

In a case control design, all signals (n = 42) disseminated to the Medicines Evaluation Board from the second quarter of 1997 until the third quarter of 2000, which could be expressed as a combination of a single ATC code and a single WHO preferred term, were included. For each case, four controls were matched in time. Logistic regression analysis was used to investigate the influence of various factors, such as the fact whether the ADR or drug is new, the strength of the association, the seriousness of the reaction and the documentation of the reports.

Results

Multivariate analysis showed that the presence of a ‘serious report’ (Odds Ratio 3.8, 95% CI 1.3, 11.0), a WHO ‘critical term’ (OR 4.7, 95% CI 1.8, 13), the ADR being unlabelled (OR 6.1, 95% CI 2.3, 16) and the presence of a disproportionate association (OR 3.5, 95% CI 1.4, 8) were all independently associated with signal selection. The number of reports and the time after marketing of the drug had no influence.

Conclusions

This study showed that selection of signals is based on both qualitative and quantitative aspects. Knowledge of these factors may improve the efficiency of the underlying signal selection process.

Introduction

After marketing of a drug, close monitoring for unexpected adverse drug reactions (ADRs) remains necessary due to the limited size of premarketing trials, the selection of the patients involved and the limited duration of the trials [1]. For this reason, post-marketing surveillance aims at a timely detection of either new ADRs or an increase of the frequency of ADRs which are already known to be associated with the drugs involved. Spontaneous reporting systems (SRSs) still play an important role in providing early signals concerning suspected ADRs. A signal can be defined as reported information on a possible causal relation between an adverse event and a drug, the relation being previously unknown or incompletely documented [2]. The detection of signals from SRSs generally results from a systematic manual review of every incoming report sent to pharmacovigilance centres. In the present way of signal detection many potential signals are identified. However, relatively few of them are important enough to require further attention. For this reason, criteria need to be formulated to help to determine if there is a need for further attention concerning a particular signal. Several factors may help to determine whether or not a particular signal is worth further investigation. In general, four categories may be identified, which are for instance represented by the acronym ‘SNIP’, used by the Medicines Control Agency. These categories refer to the Strength of a signal, whether or not the issue is New, the clinical Importance as judged by the seriousness of the reaction and severity of the cases and finally the potential for Preventive measures by the regulatory authorities [3]. Most pharmacovigilance centres, however, do not systematically apply such criteria.

At the Netherlands Pharmacovigilance Foundation Lareb, maintaining the SRS in the Netherlands, signal selection is carried out by a review of every incoming report in a weekly meeting between trained assessors. Working on behalf of the Dutch Medicines Evaluation Board (MEB), this agency is informed periodically about generated signals. On a 3 monthly basis the most relevant signals are published in a so-called quarterly report to the MEB. Since the goal of this report is to give an early warning to the authorities, the signals described have a premature character. Criteria for selecting interesting signals about which the MEB should be informed, however, are not predefined and depend on the knowledge and experience of the assessors involved. Although in a later stage the signals described may be superseded, these quarterly reports give a unique chance for studying the early stages of the process of signal selection and dissemination of information from SRSs. The goal of this study is to gain insight in the various factors that play a role in the subjective process of signal selection. Knowledge of these factors may improve the efficiency of the underlying signal selection process.

Methods

Setting

The Netherlands Pharmacovigilance Foundation Lareb collects and analyses reports of suspected ADRs reported by Dutch physicians and pharmacists. A report may concern one or more suspected ADRs and one or more suspected drugs. The reports are evaluated, coded and filed in a database. To the reporting health professional a letter of confirmation is sent together with information regarding the reported association. All ADRs are coded according to the WHO adverse drug reaction terminology [4]. In this respect, possible ADRs are assigned to a so called ‘preferred term', which gives a detailed description of the clinical event. Preferred terms are linked to ‘high-level terms’, which provide a code for qualitatively similar conditions. As an example, the preferred terms ‘anxiety’ and ‘nervousness’ share the same high-level term ‘anxiety’. In this way clustering of ADRs for analysing purposes is possible. The suspected drugs as well as concomitantly used drugs are coded according to the ATC classification.

Design

Associations described in the quarterly reports to the MEB may represent possible new signals of possible ADRs, drug–drug interactions but also reviews of drugs that were marketed recently in the Netherlands. In a case control design, factors associated with the selection and dissemination of signals to be reported to the MEB were evaluated. All signals described in the quarterly reports from the first publication in the second quarter of 1997 until the third quarter of 2000 were reviewed. Cases were defined as associations that could be expressed as a combination of a single ATC code and a single WHO preferred term. The following exclusion criteria were applied [1]: associations concerning a group of related substances or related ADRs, like ‘neuropsychiatric ADRs’ or ‘allergic reactions’ [2], ‘review’ articles [3] drug–drug interactions, and [4] associations concerning a single ATC code and single preferred term but associated with an additional factor, for example gender or a special circumstance. An example of the latter one is for instance a possible withdrawal syndrome during the use of labetolol in a newborn infant.

The specific report that triggered the particular association to be selected as a signal was called the ‘index report’. For each case, four controls were selected. Controls consisted of those associations, not being published in the quarterly reports, which were described in reports received by Lareb, 10 and 20 reports previously and 10 and 20 reports later than the index report. In this way cases and controls were matched in calendar time. In case a physician or a pharmacist described two or more associations on the reporting form, the first association that was mentioned was chosen. Similar to the ‘index report’ the reports that were used to select the control associations are called ‘control reports’.

Factors

Factors possibly related to the selection of signals were distinguished in four different types, namely the fact whether the association or drug is new, factors related to the strength of the association, factors related to the seriousness of the reaction involved and factors related to the documentation of the reports to the SRS. The following factors were studied:

1. The fact whether the association or drug is new

1. In the event the suspected ADR is mentioned in the Dutch text books ‘Farmacotherapeutisch Kompas’ or the ‘Informatorium Medicamentorum’, being annually updated standard works often consulted by Dutch physicians or pharmacists in the Netherlands, the ADR was considered labelled, otherwise unlabelled [5, 6].

2. Period of marketing of the drug involved. A distinction was made between the drugs that have been marketed for less than 5 years on the Dutch market.

1. Factors related to strength of the association

1. The total number of reports received on this particular association at the time the index or control report was received. Three categories were defined: 1–2 reports, 3–4 reports and more than 4 reports.

2. The presence of a disproportionate association. For every association, disproportionality was calculated [Van Puijenbroek et al., submitted]. This was done by calculating a 2 × 2 contingency table for every association involved (Figure 1). For calculating the number of reports concerning the suspected drug, the full ATC code was used, for calculating the number of ADRs, the preferred term was used. As a measure of disproportionality, the corresponding reporting odds ratio (ROR) was calculated [7–9]. The ROR can be calculated as a*d/b*c (see Figure 1) and expressed as a point estimate with the corresponding 95% confidence interval. An association was considered to be disproportionate in the event that the lower limit of the 95% confidence interval was greater than 1. In the event the denominator of the fraction is zero, the ROR cannot be calculated. This may occur in the event of a rare ADR. In this situation, the association involved was considered to be reported more frequently than its background frequency and having the same impact on the assessors as a statistically significant ROR.

Figure 1.

2 × 2 contingency table used for the calculation of ADR reporting odds ratios.

1. Factors related to the seriousness of the reaction involved.

1. The index or control was considered serious in the event of death, a life threatening situation (prolonged) hospital admission, disability, or congenital malformations.

2. The presence of a so-called ‘critical term’. Critical terms are a subset of the WHO preferred terms, being indicative of serious disease states, which can be regarded as important to follow up [10]. For this reason, critical terms may be of particular interest for signal generation.

1. Factors related to documentation of the reports

1. The dechallenge of the index or control report is positive, i.e. the ADR disappeared after cessation of the drug.

2. The rechallenge of the index or control report is positive, i.e. the ADR reappeared after the drug was used again.

3. The fact that the reports was sent by the physician (in attendance) of the patient.

Analysis

Logistic regression analysis was used to analyse the influence of the various factors. In the first part of the study a univariate analysis was carried out. Odds ratios were expressed as point estimates with corresponding 95% confidence intervals. Factors that were statistically significantly associated with the selection of signals were analysed in the second part of the study in a multivariate analysis.

Finally, the independent statistically significant factors resulting from the multivariate analysis were subsequently analysed in more detail. In respect to the selection of signals, the performance expressed as sensitivity, specificity, positive and negative predictive value of both the separate factors, as well as the combination of these factors, was calculated in respect to the manual selection of the associations in the signal selection process as the gold standard. For logistic regression analysis the statistical package SPSS 10.0 was used, for calculating the parameters of performance Microsoft excel 97 was used.

Results

Between the second quarter of 1997 and the third quarter of 2000 a total number of 76 associations were published in the quarterly reports to the MEB. Sixteen associations were excluded, because they concerned a group of related substances or ADRs, 7 associations were excluded because they were published as a ‘review’ article, another 7 associations were excluded because they concerned possible drug–drug interactions, and finally 4 associations were excluded because an additional factor was involved. A total number of 42 signals were included in the analysis and were matched with 168 controls. An overview of the selected signals is shown in Table 1.

Table 1.

Selected signals from the quarterly reports to the MEB from the 2nd quarter of 1997 till the 3rd quarter of 2000.

Association	Date of publication in quarterly report to the MEB
Norfloxacin – fixed drug eruption	2nd quarter 1997
Oxybutinin – hallucination	2nd quarter 1997
Losartan – taste disorder	2nd quarter 1997
Mefloquine – convulsions	3rd quarter 1997
Paroxetine – restless legs syndrome	3rd quarter 1997
Losartan – angiooedema	4th quarter 1997
Cisapride – QT prolongation	4th quarter 1997
Lamotrigine – death	4th quarter 1997
Terbinafine – arthralgia	1st quarter 1998
Lithium – decrease in libido	1st quarter 1998
Miconazole – influence on prothrombin time	1st quarter 1998
Tramadol – micturition disorder	2nd quarter 1998
Irberstartan – angiooedema	2nd quarter 1998
Miconazole oral gel – chocking	2nd quarter 1998
Rulizole – thrombopenia	2nd quarter 1998
Vigabatrin – visual field defect	3rd quarter 1998
Tolcapone – leucopenia	3rd quarter 1998
Nefazodone – priapism	4th quarter 1998
Olanzapine – death	4th quarter 1998
Fexofenadine – QT prolongation	4th quarter 1998
Sildenafil – death	4th quarter 1998
Itraconazole – dyspnoea	1st quarter 1999
Diclofenac – anaphylactic reaction	1st quarter 1999
Quetiapine – leucopenia	1st quarter 1999
Diclofenac – haemolytic anaemia	2nd quarter 1999
Oral budesonide – anaphylactic reaction	2nd quarter 1999
Atorvastatin – rhabdomyolysis	2nd quarter 1999
Interferon alfa 2B – Raynaud's syndrome	2nd quarter 1999
Alendronate – alopecia	3rd quarter 1999
Lamotrigine – sialoadenitis	3rd quarter 1999
Valproic acid – parkinsonism	3rd quarter 1999
Metronidazole – hepatitis	4th quarter 1999
Valproic acid – polycystic ovary syndrome	4th quarter 1999
Acitretin – taste loss	4th quarter 1999
Simvastatin – eczema	1st quarter 2000
Loperamide – urinary retention	1st quarter 2000
Cotrimoxazole – tremor	2nd quarter 2000
Lamotrigine – Stevens Johson syndrome	2nd quarter 2000
Minocycline – interstitial pneumonia	2nd quarter 2000
Clopidrogel – thrombotic thrombocytopenic purpura	3rd quarter 2000
Rofecoxib – death	3rd quarter 2000
Pergolide – pulmonary fibrosis	3rd quarter 2000

Univariate analysis

Table 2 shows the differences between cases and controls and the results of the univariate analysis. The absolute number of reports concerning the association does not contribute in the selection of signals. However, the proportion of associations with a statistically significant ROR, is higher among the cases (Odds Ratio 3.2 (95% CI 1.6, 6.4)). Also the proportion of unlabelled ADRs is statistically significantly higher among the cases (OR 2.7 (95% CI 1.3, 5.5)), as well as the proportion of drugs that are shortly marketed than five years (odds ratio 2.2 (95% CI 1.1, 4.6)).

Table 2.

Differences between cases and controls and the results of the univariate analysis.

	Cases n (%) n = 42	Controls n (%) n = 168	Univariate analysis odds ratio (95% CI)
How new is the association or the drug?
ADR unlabelled	27 (64)	67 (40)	2.7 (1.3, 5.5)
Suspected drug shorter than 5 years marketed	17 (41)	39 (23)	2.2 (1.1, 4.6)
Factors related to strength of the association
Absolute number of reports
1 or 2 reports (reference category)	22 (52)	69 (41)	1 (ref)
3 or 4 reports	8 (19)	28 (17)	0.9 (0.4, 2.3)
More than 4 reports	12 (29)	71 (42)	0.5 (0.2, 1.2)
ROR full ATC code/preferred term statistically significant (lower limit 95% CI > 1)	23 (55)	46 (27)	3.2 (1.6, 6.4)
Factors related to the seriousness of the reaction involved
Index or control report concerns a ‘serious’ ADR	21 (50)	20 (12)	7.4 (3.5, 16)
WHO critical term present	26 (62)	30 (18)	7.5 (3.6, 16)
Factors related to documentation of the reports to the SRS
Index or control report mentions a positive dechallenge	17 (41)	60 (36)	1.2 (0.6, 2.5)
Index or control report mentions a positive rechallenge	4 (10)	16 (10)	1.0 (0.3, 3.2)
Source of reports: number of reports by physicians	31 (73.8)	107 (64)	1.6 (0.8, 3.4)

The impact of a ‘critical’ term (OR 7.5 (95% CI 3.6, 16)) present among the cases is comparable with the presence of a ‘serious’ (OR 7.4 (95% CI 3.5, 16) report. Information regarding the quality of the documentation of the reports like the presence of a positive de- or rechallenge, or the source of the reports does not make a statistically significant contribution to the selection of possible signals.

Multivariate analysis

In the second part of the study, the influence of factors that were positively associated with signal selection in the first analysis was analysed in a multivariate logistic regression analysis. The results are shown in Table 3. The presence of a ‘serious report’ (OR 3.8, 95% CI 1.3, 11), a WHO ‘critical term’ (OR 4.7, 95% CI 1.8, 13), the ADR being unlabelled (OR 6.1, 95% CI 2.3, 16) and the presence of a disproportionate association (OR 3.5, 95% CI 1.4, 8.4) were all independently associated with signal selection. The time since marketing of the suspected drug apparently did not make an additional contribution to the selection of signals.

Table 3.

Results of the multivariate logistic regression analysis.

	Odds ratio (95% confidence interval)
ROR full ATC code/preferred term statistically significant	3.5 (1.4, 8.4)
Index or control report concerns a ‘serious’ ADR	3.8 (1.3, 11)
Critical term present	4.7 (1.8, 13)
ADR unlabelled	6.1 (2.3, 16)
Suspected drug shorter than 5 years marketed	1.4 (0.5, 3.5)

Performance of factors

Table 4 shows sensitivity, specificity, positive and negative predictive values for the factors contributing significantly to the selection of cases. Sensitivity is highest for the ADR being unlabelled [0.64], but this factor also has the lowest specificity [0.60] and lowest positive predictive value [0.28]. In respect of the negative predictive value, all factors are comparable.

Table 4.

Sensitivity, specificity, positive predictive value and negative predictive value of independent individual factors contributing to the signal selection process, as well as the combination of these factors.

	Sensitivity	Specificity	Positive predictive value	Negative predictive value
Individual factors
ROR full ATC code/preferred term statistically significant	0.55	0.73	0.33	0.87
Index or control report concerns a ‘serious’ ADR	0.50	0.88	0.51	0.88
Critical term present	0.62	0.82	0.46	0.90
ADR unlabelled	0.64	0.60	0.28	0.87
Combined factors
One or more factors present	1	0.29	0.26	1
Two or more factors present	0.72	0.80	0.48	0.92
Three or more factors present	0.52	0.95	0.53	0.89
All four factors present	0.07	0.99	0.6	0.81

Concerning the combination of the aforementioned factors, combining for instance two or more factors both yields a high sensitivity [0.52], specificity [0.95] and positive [0.48] and negative predictive value [0.92].

Discussion

Our study showed that the seriousness of the reaction, the presence of a critical term, a disproportionate number of associations in the database and the fact that the suspected ADR is unlabelled, all play a role in the signal selection process. Although slight differences exist, all four independent factors resulting from this analysis are more or less comparable in respect to the performance.

Factors responsible for the selection of signals were divided in four categories, being the fact if the association or the drug involved was new, factors related to the strength of the association, the seriousness of the association and factors related to the documentation of the reports. Except for the latter one, all types seem to have influence on the selection of signals for dissemination to the MEB. Except for the factors related to the documentation, the aforementioned categories are also part of the ‘SNIP’ criteria [3]. A retrospective analysis of pharmacovigilance topics published between 1973 and 1990 in the Netherlands, revealed that 46% of the topics were new and 57% referred to established products [11]. Although the definition of the various factors slightly differs, the results of our study are in accordance with these findings. In our study, 64% of the ADRs were unlabeled and 60% of the reports referred to established products.

The quality of the reports is essential for an optimal assessment and for maintaining a high quality of the database. Factors indicating the presence or absence of information about a possible dechallenge or rechallenge in the index or control report and the source of the reports, however, were not statistically significant. The way the reports are presented seems not to be decisive in the selection of signals. Nevertheless, the level of documentation of a report also involves other aspects, like the completeness of the medication history or the absence or presence of detailed clinical information. Therefore, further research will be necessary to gain insight into the contribution of the level of documentation of the reports in the selection of possible signals.

Not only the information present in the index report may have contributed to the selection of an association, but also other reports that were received previously may have contributed to the selection. For this reason, we analysed the contribution of information concerning the fact whether a report was ‘serious’, the presence of a critical term, and the presence of a positive dechallenge or rechallenge in any other report than the index or control report. None of these factors, however, had an additional contribution to the selection of signals compared with the information present in the index of control report.

Spontaneous Reporting Systems for ADRs have been used over the past 40 years. Since the amount of data is increasing there is a growing need for additional quantitative signal detection techniques. Napke used a so-called pigeonhole system in which every incoming report was colour-coded [12]. The optical impact focused the attention to specific associations. In the mid 1970s the first ideas for computer programmes were developed [13, 14]. The development and implementation of these techniques, which are mainly based on searching for a disproportionate number of associations, is still increasing [9, 15]. Recently new approaches have been developed, like the Bayesian Confidence Propagation Neural Network [16, 17]. Also techniques for the identification of possible drug–drug interactions and syndromes are being developed [18]. Quantitative techniques, however, cannot replace the traditional case by case approach, but serve as an additional tool in signal analysis. Although information technology may be helpful in identifying possible signals, possibilities for including clinical information in this decision making process are not yet available. The results of our study showed that, in contrast to the absolute number of reports sent to the SRS, disproportionality is a predictive factor for selecting possible signals. However, in the event the numerator is zero, a ROR cannot be calculated. This is a drawback for the use of this measure, but since the assessors are familiar with the interpretation of the ROR, it was also used in our analysis. The ROR could not be calculated for two index cases and one control case. In all three cases the poison probability was calculated, for which P < 0.001. For this reason, in the event a ROR could not be calculated, the association was considered to be disproportionate present in the database.

In the present study, the ROR was calculated in respect to the full ATC code and the preferred term, but the ROR can be calculated in various other ways. By choosing another level of aggregation in respect to the suspected drugs or ADRs, different information can be obtained. For instance, in the event the ROR is calculated based on the first five positions of the ATC code (ATC5) and the preferred term, information is provided concerning the occurrence of the suspected ADR in chemically related substances in the database. Similarly, the reporting odds ratio can be calculated concerning the full ATC code (ATC7) and the high level term, which provides information concerning the occurrence of the related ADRs associated with the suspected drug in the database. Finally, the reporting odds ratio concerning the first five positions of the ATC code (ATC5) and the high level term can be calculated, which provides information concerning the occurrence of the related ADRs in chemically related substances. All these various approaches, however, yielded similar results as the calculation of the ROR based on the full ATC code and the preferred term, indicating that these other approaches did not provide any additional information on top of the normal approach.

In preparation for the weekly assessment meeting where the selection of possible signals take place, concise information concerning the reports is presented to the assessors on an overview form. This concerns information about the gender and age of the patient, the source of the reports, a description of the event, coding of the suspected drug and suspected ADR, time of onset of the ADR and the fact whether the suspected ADR is labelled or unlabelled. Furthermore, quantitative information is provided like the number of associations reported, the number of reports on the ATC code, the number of reports on the high-level term and the standard residual value as a measure of disproportionality. The assessors may refer to the database for additional information like a more extensive description of the clinical event and the concomitant medication the patient used. Since 1999 also information concerning the extent of disproportionality in the Lareb database is presented as a ROR with corresponding 95% confidence interval. Data concerning the presence of the association in the WHO database are available but are not presented in advance. Although for instance information about the extent of disproportionality, the fact if the ADR is labelled and the seriousness of the report is available, selection bias may be present, but this reflects the present procedure of selection and dissemination of signals.

To make a distinction between ‘unlabelled’ and ‘labelled’ associations, two standard works that are frequently used in daily practice by physicians and pharmacist were used. Historically, coding of the reports at Lareb always required the use of these two standard works and not the official SPC text. Although slight differences with the official SPC occasionally may exist, this coding was also used in this study.

Signals published in the quarterly reports have a preliminary character. After all, the goal is to inform the MEB about possible new signals in a rather early stage. This implies that the causality of the signals mentioned in Table 1 does not necessarily have to be proven. Selected cases serve as a point of attention for further research or attention for other cases that might be reported. Media attention of previous publication of a case report might raise the number of reports on an association, which may cause reporting bias. Although this cannot completely be ruled out in individual reports, this does not necessarily have to be unfavourable, since this may lead to an early warning of the association under concern.

The results of this study may be used to improve the signal selection process, by making a pre-selection of associations based on the presence of one of these factors. However, the performance of these factors is based on calculations on the dataset of the Netherlands Pharmacovigilance Foundation. On other datasets differences of the performance may exist.

Conclusion

This study showed that selection of signals generated by the subjective review of data sent to a spontaneous reporting system for adverse drug reactions is based on both qualitative and quantitative aspects. Both the extent of disproportionality as well as the seriousness of the reports, the presence of a ‘critical term’ and the fact whether or not the association under concern is labelled, seem to have a comparable impact in the process of the selection signals to be disseminated. The results of this study revealed which factors are primarily responsible for signal selection and dissemination. A better understanding of these factors may eventually improve the efficiency of the signal selection process.