The Reporting of a Disproportionality Analysis for Drug Safety Signal Detection Using Individual Case Safety Reports in PharmacoVigilance (READUS-PV): Development and Statement

Abstract

Background and aim

Disproportionality analyses using reports of suspected adverse drug reactions are the most commonly used quantitative methods for detecting safety signals in pharmacovigilance. However, their methods and results are generally poorly reported in published articles and existing guidelines do not capture the specific features of disproportionality analyses. We here describe the development of a guideline (REporting of A Disproportionality analysis for drUg Safety signal detection using individual case safety reports in PharmacoVigilance [READUS-PV]) for reporting the results of disproportionality analyses in articles and abstracts.

Methods

We established a group of 34 international experts from universities, the pharmaceutical industry, and regulatory agencies, with expertise in pharmacovigilance, disproportionality analyses, and assessment of safety signals. We followed a three-step process to develop the checklist: (1) an open-text survey to generate a first list of items; (2) an online Delphi method to select and rephrase the most important items; (3) a final online consensus meeting.

Results

Among the panel members, 33 experts responded to round 1 and 30 to round 2 of the Delphi and 25 participated to the consensus meeting. Overall, 60 recommendations for the main body of the manuscript and 13 recommendations for the abstracts were retained by participants after the Delphi method. After merging of some items together and the online consensus meeting, the READUS-PV guidelines comprise a checklist of 32 recommendations, in 14 items, for the reporting of disproportionality analyses in the main body text and four items, comprising 12 recommendations, for abstracts.

Conclusions

The READUS-PV guidelines will support authors, editors, peer-reviewers, and users of disproportionality analyses using individual case safety report databases. Adopting these guidelines will lead to more transparent, comprehensive, and accurate reporting and interpretation of disproportionality analyses, facilitating the integration with other sources of evidence.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40264-024-01421-9.

Key Points

Misreported and misinterpreted disproportionality analyses contribute to poor research quality.

The REporting of A Disproportionality analysis for drUg Safety signal detection using individual case safety reports in PharmacoVigilance (READUS-PV) guidelines will help researchers to adequately report their study and interpret the results.

Introduction

Pharmacovigilance is the science and activities relating to the detection, assessment, understanding, and prevention of adverse drug reactions (ADRs) or any other possible drug-related problems [1]. The main data source for detecting new ADRs after drug marketing approval consists of databases of individual case safety reports (ICSRs) recording suspected ADRs—whether spontaneous or generated within active surveillance activities [2–5]. A range of methods have been developed for mining these databases [6]. Disproportionality analyses are statistical methods that aim at quantifying the association between drug(s) and adverse event(s) comparing the number of observed ICSRs recording both the drug(s) and the event(s) with the number of ICSRs that would be expected in the absence of any association between the drug and the event, based on the underlying drug and event rates within the same ICSR database [7]. These analyses generate a signal of disproportionate reporting (SDR) when the statistic significantly exceeds a predefined threshold [1, 8, 9]. Because of the lack of exposure data and the unquantified under- and selective reporting, SDRs cannot be interpreted in themselves as conclusive scientific evidence of a causal relationship between a drug and an adverse event [1, 10]. Consistently, a signal of suspected causality [11, 12] should only be presented after the SDR has undergone an initial triage including, whenever possible, a case-by-case assessment of the ICSRs and the contextualization within knowledge already accrued from other sources of evidence (e.g., clinical trials, observational studies, case reports/series, literature, and animal experiments) [3, 4, 13, 14].

Nowadays, disproportionality analyses are increasingly used by multiple stakeholders, including pharmaceutical companies, regulatory agencies, and researchers. Their findings are frequently published in both specialized pharmacology journals and general medical journals [15–18]. However, they are often poorly reported and interpreted [17, 19]. Indeed, over 75% of published disproportionality studies failed to report essential elements needed to understand and reproduce the analyses and results, and more than two thirds of authors over- or misinterpreted their findings, notably in the abstract [17, 19]. Poor reporting and interpretation hamper the use and ability to assess research findings and contribute to research waste (i.e., avoidable inappropriate conduct and dissemination of research) [20–22]. Reporting guidelines, representing a minimum standard of items that should be reported in published articles, have therefore been developed for a range of study designs [23–26]. However, the design of disproportionality analyses has specific features that are not covered by existing reporting guidelines for observational and pharmacoepidemiological studies such as Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) and REporting of studies Conducted using Observational Routinely collected health Data statement for PharmacoEpidemiology (RECORD-PE) [25, 26]. We have therefore developed a guideline specifically tailored to the reporting and interpretation of disproportionality analyses: the REporting of A Disproportionality analysis for drUg Safety signal detection using individual case safety reports in PharmacoVigilance (READUS-PV).

The READUS-PV guidelines are intended as a guidance to support the reporting (and the publication) of disproportionality analyses to ensure that readers can easily determine what was planned (i.e., the research question), done (i.e., the methodology used), and found (i.e., the results and drawn conclusions). They should therefore not be considered as a tool to explicitly assess the quality of a published manuscript or the validity of an SDR as a true safety signal, which also requires considering additional methodological aspects. Nonetheless, these guidelines can indirectly improve the quality of research by pointing to items that should be already addressed during study design and reducing the risk of misinterpretation of results. In this regard, these guidelines should be viewed in conjunction with other recommendations on signal detection practices, such as those promoted by pharmacovigilance experts, the Innovative Medicine Initiative Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium (IMI PROTECT) for methodological considerations on disproportionality analyses, and by regulatory agencies for assessment of safety signals [12, 27–29]. The READUS-PV statement comprises a checklist of 14 items recommended for reporting disproportionality analyses, and four additional items for the abstract. Box 1 includes a glossary of terms used throughout the READUS-PV statement. This article is simultaneously published with an “explanation and elaboration” article providing additional reporting guidance for each item, along with examples of good reporting [30].

Box 1.

Glossary of terms

Term	Definition
Adverse event*	Any untoward (i.e., noxious and unintended) medical occurrence that develops in an individual exposed to a medicinal product. Possible conditions of exposure include appropriate medical use, medication errors, off-label use, overdose, misuse, abuse, and occupational exposure. The medical occurrence does not necessarily have a causal relationship with the exposure.
Adverse drug reaction (ADR)*	Any adverse event characterized by an at least reasonable possibility that the medicinal product has caused the event.
Causality assessment*	The process of evaluating and assigning a causal judgment to an observed association between a medicinal product and an adverse event, at the level of either individual ICSRs or case series. Causality assessment can rely on expert judgment/global introspection, structured guidelines and algorithms, or probabilistic approaches [31].
Drug	A drug is usually defined as any chemical substance that causes a change in an organism's physiology or psychology when consumed. To be consistent with pharmacovigilance terminology (e.g., drug-related problem, adverse drug reaction, drug–event combination) we adopted the use of the term drug, but these guidelines are valid for disproportionality analyses on any medicinal product used in the prevention, diagnosis, or cure of diseases (e.g., vaccine, medical device, gene therapy, cell therapy, supplements).
Drug–event combination	The specific combination of medicinal product(s) and event(s) of interest.
Individual case safety reports (ICSRs)**	Format and content for the reporting of one or several adverse events occurred in a single individual at a specific point of time. It accommodates clinical phenotypes involving multiple events that may manifest sequentially over time.
Pharmacovigilance*	The science and activities relating to the detection, assessment, understanding, and prevention of ADRs or any other drug-related problem.
Case-by-case analysis	Analysis of each ICSR recording the drug–event combination to collect further information useful for the causality assessment.
Safety signal*	Information that arises from one or multiple sources (including observations and experiments), which suggests a new potentially causal association or a new aspect of a known association between medicinal product(s) and adverse event(s). The information is judged to be sufficient to justify verificatory actions.
ICSR database	A surveillance database that relies on ICSRs submitted by multiple stakeholders (healthcare providers, consumers, and pharmaceutical companies) because of spontaneous initiative or mandatory reasons).
Signal of disproportionate reporting (SDR)*	A statistical association between medicinal product(s) and event(s) identified by any disproportionality analysis within an ICSR database.

* Definitions adapted from the Council for International Organizations of Medical Sciences (CIOMS) cumulative glossary with a focus on pharmacovigilance (version 2.1)

**Definition adapted from the European Medicines Agency (EMA) definition of ICSR

The READUS-PV Checklist

Conception and Development of the Checklist

We followed the recommendations by the Enhancing the QUality And Transparency Of health Research (EQUATOR) Network for the development of reporting guidelines [32]. The protocol was registered with the EQUATOR Network website on March 7, 2022, and published on a dedicated website (https://readus-statement.org/).

Based on a bibliometric analysis and taking into account expertise and geographical provenance, we established a diverse group of international individuals from universities, the pharmaceutical industry, and regulatory agencies, with expertise in pharmacovigilance, disproportionality analyses, and assessment of safety signals. Of the 70 invitations sent out, 34 experts agreed to take part in the project. We followed a three-step process to develop the checklist: (1) an open-text survey to generate a first list of items to be potentially included in the checklist, (2) a modified Delphi method to select and rephrase items important for the reporting, and (3) a final online consensus meeting.

Open-Text Survey

We first used an open-text survey in which the participants were asked to suggest items that they thought should be included in articles reporting disproportionality analysis results. To guide participants in their suggestions, we used the same subsection of the STROBE and RECORD-PE checklists (e.g., background/objective/study design/data source). Among the panel members, 32 participants compiled the open-text survey. The steering committee (MF, FS, ER, CK) then summarized the comments retaining all unique ideas (regardless of frequency) and proposed an initial list of 97 items.

The Delphi Method

Panel members were asked to decide whether they believed the proposed items should be included in the READUS-PV reporting guideline, using an online modified Delphi method through Sphinx Online^© software version 4.30. We used a logical algorithm to select items to be included in the following steps (Supplementary Figure 1, see the electronic supplementary material). Briefly, items were included in the checklist if more than 80% of participants agreed, with no major comments on wording or concept. In the case of major comments, independently of the agreement rate, items were modified and proposed to the second round, in which the agreement on both the phrasing and the inclusion of the item in the checklist were assessed. Among the panel members, 33 experts responded to round 1 and 30 to round 2 of the Delphi. Overall, 60 recommendations for the main body of the manuscript and 13 recommendations for the abstracts were retained by participants after the Delphi method.

Online Consensus Meeting

Before the meeting, the steering committee merged related items, harmonized terminologies, and shared with all participants the draft version of the checklist. All participants were also asked to review the final list of items and share comments to be discussed in the final meeting through a shared online document. There were 25 attendees from 14 countries participating in the online consensus meeting (https://readus-statement.org/). Each item selected in previous rounds of the Delphi method was discussed, and participants were invited to share their views on the proposal and vote for final decisions about wording of items. In this step, minor comments collected in previous rounds were considered. Decisions were adopted by vote, and items were rediscussed and rephrased, if necessary, until at least 80% of participants agreed. The panel agreed on 14 items for manuscript body, and four items for the abstract.

Post-Meeting Activities

Following the online consensus meeting, the steering committee of READUS-PV prepared the first draft of the two manuscripts: the statement and the explanation and elaboration. The manuscripts were shared with the entire panel for two rounds of revisions.

Using the READUS-PV Checklist

The READUS-PV checklist templates for the manuscript body (Table 1) and the abstract (Table 2) can be downloaded from either the electronic supplementary material (Supplementary Tables 1 and 2) or the READUS-PV statement website (http://www.readus-statement.org/). We recommend authors refer to READUS-PV already when designing the study to achieve a more complete and transparent documentation. Journals and publishers may impose restriction on word count, manuscript structure, and number of tables and figures allowed. The recommendation for placing information in specific manuscript sections should not be viewed as mandatory, but relevant information should still be presented, if not in the manuscript body, at least in properly referenced supplementary materials, protocols, and/or data repositories (e.g., Open Science Framework, Dryad, Figshare).

Table 1.

The READUS-PV checklist

Section and topic	Item #	Checklist item
Title
	1a	If disproportionality analyses are a prominent component of the published study, the study should be identified as a “disproportionality analysis.” The type of data and name of the database(s) should be specified.
	1b	Report the name of adverse event(s) and/or drug(s) under study, when applicable.
Introduction
Background	2a	Describe the drug(s) and its utilization, the nature of the adverse event(s) under study and its frequency, and the existing knowledge on the drug–event combination.
	2b	Specify the rationale for performing the analysis, e.g., as part of routine pharmacovigilance, to investigate an overall safety profile, or to assess a pre-specified hypothesis.
	2c	Explain why individual case safety report databases and disproportionality analysis are suitable to fill the knowledge gap.
Objectives	3	State specific objectives, identifying the adverse event(s), the drug(s), and the reference group, including any pre-specified hypothesis, if applicable.
Methods
Study design	4a	Identify the study (i.e., “disproportionality analysis”) and the type of data used (e.g., “individual case safety reports”).
	4b	Provide an outline of the entire study design, including primary and sensitivity analyses performed, and other designs such as case-by-case analysis or literature review.
Data description, access, and pre-processing	5a	Specify the name of the database(s), the database(s) custodian, and the coverage. Specify the type/number of drugs included within the database and the thesaurus, taxonomies, or ontologies used for coding drugs and events.
	5b	Specify the extraction dates and describe and justify all choices used for data pre-processing, including any data transformation or exclusion, if appropriate.
Variables definition	6a	Describe the study population, including any restriction.
	6b	Describe the nature and the meaning of key variables assessed in the work.
	6c	Specify and justify any grouping of drugs or events. For drugs, specify and justify whether active ingredients/trade names/salts were considered and/or the selected role.
	6d	Describe any additional data source used, the type of data, and how they interact with individual case safety reports.
Statistical methods	7a	Present any descriptive analysis performed, specifying variables investigated, statistical tests, and significance thresholds.
	7b	Describe the measure(s) selected for the disproportionality analysis including any threshold used to identify signals of disproportionate reporting. Explain the reason for this choice if applicable.
	7c	Clearly describe any sensitivity analysis and any tool to control confounding, including any restriction, subgroup, stratification, adjustment, or interaction.
	7d	Specify the variables and methods used for the case-by-case analysis, including any algorithm or criteria used to assess causality, if performed.
	7e	Specify any statistical methods used for other data sources.
Results
Participants	8a	Specify the number of individual case safety reports included at each stage, including reasons for exclusion.
	8b	Provide key demographic and clinical characteristics of cases, if possible comparing cases with any appropriate reference group.
Disproportionality analysis	9	Present all results including confidence intervals. Present also results of sensitivity analyses, if performed.
Case-by-case analysis	10	Present the case-by-case analysis of key variables. Present the causality assessment, if applicable.
Discussion
Key results	11	Discuss key results with reference to study objectives and contextualize them within the current literature and other consulted sources. Clearly discriminate between expected reactions and emerging safety signals.
External validity	12a	Discuss the external validity of the results to the general population.
	12b	Discuss the potential relevance of results in clinical practice
	12c	Propose further study designs if applicable
Limitations	13	Present general limitations, making clear that disproportionality analysis alone cannot prove causation or measure incidence, and specific limitations, including confounding and reporting bias and efforts to mitigate them.
Declarations
	14a	Provide the source of funding/sponsorship and the role of the funders/sponsors for the present study and for any original study on which the present article is based.
	14b	Clearly identify potential commercial and intellectual conflicts of interest (e.g., link to any drug/event investigated, whether financial, legal action, or software used).
	14c	Declare any institutional approval needed or granted in the investigation.
	14d	Include a statement on data availability, code availability (including the version of the statistical software used), and protocol registration.

READUS-PV REporting of A Disproportionality analysis for drUg Safety signal detection using individual case safety reports in PharmacoVigilance

Table 2.

The READUS-PV checklist for abstracts for standalone studies

Section and topic	Item #	Checklist item
Background	1a	State the aim/rationale for performing the study.
	1b	Specify the adverse event(s) and/or the drug(s) under study, when applicable.
	1c	Specify the specific population or setting, when applicable.
Methods	2a	Identify the study as a “disproportionality analysis” and specify the type of data used.
	2b	Specify the name of the database(s) used and the type of access.
	2c	Specify the timeframe and geographical region, when applicable.
	2d	Specify the disproportionality measure(s) used and their statistical significance threshold(s).
	2e	Specify if a case-by-case analysis is performed.
Results	3	Report main findings including their precision (e.g., 95% confidence intervals), together with a short summary of the case-by-case analysis.
Conclusion	4a	Clearly report key conclusions.
	4b	Acknowledge that the disproportionality analysis is a hypothesis generating or refinement approach.
	4c	State the implications and clinical relevance of the findings.

READUS-PV REporting of A Disproportionality analysis for drUg Safety signal detection using individual case safety reports in PharmacoVigilance

Discussion

The READUS-PV guidelines were designed to improve the reporting of disproportionality analyses using ICSR databases. The completeness and accuracy of research reporting are ethical requirements endorsed by leading international statements and recommendations [33, 34]. Indeed, complete reporting enables readers to assess the appropriateness of methods, and thus the validity of research results. Moreover, improving the transparency of the reporting, these guidelines bear the potential to enhance the reproducibility and replicability of disproportionality analysis.

Reporting guidelines have been shown to improve the completeness and quality of reporting (cf. Consolidated Standards of Reporting Trials [CONSORT] for the reporting of randomized controlled trials) [35, 36]. However, compliance with these guidelines is suboptimal even in journals that actively endorse them [37]. Various strategies to increase the use of reporting guidelines have therefore been proposed, e.g., training, encouraging and checking adherence, providing feedback, or involving experts [38]. Moreover, endorsement by scientific societies and professional networks and inclusion in pharmacovigilance curricula should be envisioned [39]. Most of these strategies have not been evaluated by robust designs, while convincing evidence suggests a pivotal role for the involvement of journal editors [32, 38, 40]. We therefore encourage journal editors and publishers to raise awareness of READUS-PV, endorse its use (for example, by referring to it in the “Instructions for authors” section of the journal), and advise editors and peer reviewers to consider the READUS-PV checklists. Guidance for journal editors for endorsing READUS-PV guidelines is provided on the website. Moreover, we recommend authors submit the reporting template (downloadable from https://readus-statement.org/) along with their submission, indicating where information for each item is reported.

Although these guidelines primarily focus on disproportionality analyses, SDRs alone cannot constitute safety signals and should instead be assessed through a case-by-case analysis and contextualized with evidence from other data sources and methods [13]. The checklist items are partly applicable to other signal detection studies on ICSR databases, such as other observed-to-expected analyses (i.e., where the expected number of ICSRs is defined using epidemiological data), mixed approaches with meta-analyses, or pharmacoepidemiologic studies, but should be used in concomitance with other reporting guidelines when appropriate.

Limitations

In developing the READUS-PV guidelines, we have set up a large panel of international experts in pharmacovigilance, disproportionality analysis, and assessment of safety signals, including researchers from universities, the pharmaceutical industry, and regulatory agencies. Although we have tried to widely spread our invitations to participate, we may have missed out on people with expertise in the field, particularly in non-European countries, but we hope that more experts will be involved in revising or extending the guidelines. We acknowledge that periodic updates and refinements will be necessary to ensure the READUS-PV guidelines encompass the latest advancements in the field, and item inclusion may be rediscussed (see Supplementary Table 3 in the electronic supplementary material for excluded items). We encourage all users of this checklist to share comments and suggestions for improvement through the dedicated platform in the READUS-PV website. Finally, the actual impact of these guidelines deserves specific studies and will be assessed after relevant implementation and dissemination within the scientific community. Given translation of reporting guideline checklists to scoring systems has been shown to be heterogenous among meta-research studies and poorly reported, the development of a uniform adherence-scoring tool to evaluate adherence to READUS-PV guidelines will be developed [41, 42].

Conclusion

The READUS-PV statement is provided to benefit authors, editors, peer-reviewers, and users of disproportionality analyses on ICSR databases. Ultimately, we hope that the uptake of these guidelines will lead to more transparent, complete, and accurate reporting and interpretation of disproportionality analyses, thus facilitating integration of evidence from disproportionality analyses with evidence from other data sources and actual clinical transferability for evidence-based decision-making.

Supplementary Information

Below is the link to the electronic supplementary material.

Declarations

Funding

Open access funding provided by Alma Mater Studiorum - Università di Bologna within the CRUI-CARE Agreement.

Conflict of Interest

Gianmario Candore and Katrin Manlik are full-time employees at Bayer AG. Olivia Mahaux and Andrew Bate are full-time employees at GSK and own GSK restricted shares. Manfred Hauben was a full-time employee at Pfizer when the Delphi was conducted and owns stock/stock options in pharmaceutical companies that may manufacture/market drugs mentioned in this paper. The remaining authors declare no conflict of interest specific for this research.

Ethics Approval

The Research Ethics Committee of the University Hospital of Bordeaux have certified that the study does not need to be submitted to a Research Ethics Committee according to French regulations.

Consent to Participate

Not applicable.

Consent for Publication

Not applicable.

Availability of Data and Material

Additional data have been made available in the electronic supplementary material of the present paper and its companion article [30].

Code Availability

Not applicable.

Protocol Registration

The study was preregistered on the EQUATOR registry, and the protocol was made available at https://readus-statement.org/.

Author Contributions

MF, CK, ER, and FS conceptualized and designed the study and developed the methodology. MF synthetized all answers at each step. CK wrote the original draft. All the authors participated in the Delphi and contributed to the interpretation of results and to the review and editing of the draft. All the authors read and approved the final version.

Disclaimer

Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of their organizations.