Uncovering Pregnancy Exposures in Pharmacovigilance Case Report Databases: A Comprehensive Evaluation of the VigiBase Pregnancy Algorithm

Abstract

Background

Information on the safety of medicine use during pregnancy is limited at the time of marketing, making post-marketing surveillance essential. However, the lack of a specific indicator for pregnancy-related case reports within the international standard for transmission of individual case safety reports complicates the retrieval of such reports in pharmacovigilance databases. To address this, an algorithm to identify reports of exposures during pregnancy was developed in VigiBase, the World Health Organization global database of adverse event reports.

Objective

We aimed to evaluate and characterise the VigiBase pregnancy algorithm.

Methods

The rule-based algorithm uses multiple structured data elements in the International Council of Harmonisation (ICH) E2B transmission format that could potentially hold pregnancy-related information, to determine if a case report qualifies as a pregnancy case. Free text information is not considered. Three datasets were used for the evaluation. The “Full dataset” comprised deduplicated VigiBase data up to January 2023. The “Downsampled dataset” was a subsample of the Full dataset, adjusted to increase the prevalence of pregnancy reports by excluding individuals aged 45 years or older and male individuals aged 18 years or older, used to evaluate recall (i.e. sensitivity). The “Random dataset” was a straight random sample of the Full dataset, used to evaluate precision (i.e. positive predictive value). As a baseline for comparison, the Standardised Medical Dictionary for Regulatory Activities (MedDRA^®) Query (SMQ) “Pregnancy and neonatal topics (narrow)” was used. To provide a gold standard for the evaluation, case reports were manually annotated as either “pregnancy case” or “non-pregnancy case”, for all reports in the Downsampled dataset, and for the reports flagged as pregnancy cases by the algorithm or the SMQ baseline in the Random dataset.

Results

In the Downsampled dataset with 7874 annotated reports, 253 reports were annotated as pregnancy cases. Of those, the algorithm recalled 75% (95% confidence interval [CI] 69–80), increasing to 91% (95% CI 86–95) when restricting the analysis to reports adhering to the ICH E2B format. Preprocessing obstacles of incomplete mapping of specific pregnancy terms to MedDRA^® led to most false negatives followed by pregnancy information confined to free text information. The SMQ baseline had a lower recall of 62% (95% CI 56–68). In the Random dataset with 30,000 reports, the algorithm flagged 344 reports, among which 316 were annotated as pregnancy cases, leading to a precision of 92% (95% CI 88–95). The main reasons for false positives were postpartum indications, non-pregnancy-specific events or information miscoded as pregnancy related. The SMQ baseline had a lower precision of 74% (95% CI 69–78).

Conclusions

The VigiBase pregnancy algorithm demonstrates robust performance, highlighting its potential to facilitate pharmacovigilance related to pregnancy. Our evaluation establishes a valuable benchmark for future research and emphasises the need for global harmonisation of standards for reporting pregnancy exposures.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40264-025-01559-0.

Key Points

The VigiBase pregnancy algorithm was developed to address the challenge of identifying pregnancy-related case reports in pharmacovigilance databases.

A thorough evaluation of the algorithm showed reliable performance and provided insights into strengths and areas for improvements, underscoring its capability to advance pregnancy-related pharmacovigilance.

The study highlights the importance of harmonised global standards for reporting pregnancy exposures to improve data consistency, completeness and retrievability.

Introduction

Pregnant individuals may experience a variety of health conditions and symptoms that necessitate medical treatment during their pregnancy. In a multinational study, over 80% of pregnant individuals used at least one medication during pregnancy [1]. Other studies have shown that 50% of pregnant individuals use at least one prescription medication during the first trimester [2, 3], and that polypharmacy during pregnancy is increasing [4]. The decision to administer medication during this critical period requires careful consideration of the risks associated with untreated illness and the potential harms of treatment to the pregnant person and the developing foetus. This underscores the importance of comprehensive medication safety information in guiding therapeutic decisions during pregnancy.

Despite this, there is a wide information gap related to harmful effects from the use of medicinal products during pregnancy [5, 6]. Citing ethical considerations, pregnant persons are generally excluded from pre-marketing clinical trials, except when the product is developed specifically for use in pregnancy, resulting in sparse safety information at the time of marketing. The monitoring of medicine safety during pregnancy in the post-marketing phase is thus a crucial aspect of pharmacovigilance. It was this concern that led to the concept and establishment of national spontaneous reporting systems for suspected adverse drug reactions in the 1960s [7]. This initiative was a response to the thalidomide tragedy, where its use as an anti-emetic during pregnancy led to severe congenital malformations in newborns.

The national systems aimed to facilitate earlier recognition of adverse reactions post-marketing, through collaboration between national regulatory authorities, healthcare professionals, and later with consumers and pharmaceutical companies. Recognising the need for international cooperation, the World Health Organisation (WHO) Programme for International Drug Monitoring was founded in 1968 [7–9]. Currently, national pharmacovigilance centres in more than 150 countries share, as members of the WHO Programme for International Drug Monitoring, their nationally collected case reports with VigiBase, the WHO global database of adverse event reports for medicines and vaccines.

Regrettably, one of the significant challenges in pharmacovigilance is the difficulty in retrieving case reports involving pregnancy exposures within adverse event reporting systems. While some databases have a local indicator for pregnancy-related cases, this information is not systematically preserved when the data are converted to the International Conference on Harmonisation (ICH)1 E2B format, the current standard for electronic transmission of individual case safety reports [10–12]. The E2B format includes many different data elements with the potential to hold pregnancy information; however, there is no specific field dedicated to indicating that a case is pregnancy related. Therefore, studies that investigate pregnancy-related issues in pharmacovigilance databases use different and, often, several strategies combined to retrieve relevant reports. The most common approach is to include cases with at least one reported event from a set of selected Medical Dictionary for Regulatory Activities (MedDRA^®)2 terms [13–27] or from the “Pregnancy and Neonatal topics” Standardised MedDRA^® Query (SMQ) [13–15, 21, 25–30]. Less common approaches use a transplacental or intraamniotic route of administration [15, 19, 28, 29, 31], a positive pregnancy test [30], or matching search queries in free text fields [18, 24]. Some studies also rule out reports with ineligible patient age and/or sex [13, 15, 16, 19–21, 24, 25, 27–29, 31], paternal exposure [15, 21, 27, 31, 32], exposure during breast feeding [16, 27] or a negative pregnancy test [20, 24]. As a last example, the reported indication for use of the medicinal product has been utilised both for retrieval of pregnancy-related reports [15, 24, 28, 29] and for ruling out reports where the indication is a neonatal disorder or a congenital event [15, 21, 28, 29].

The chosen approach may depend on the feasibility of data retrieval and its intended use, for example, whether it is more important to capture all potential pregnancy cases or to be precise. Despite the frequent use of various search strategies, their performances have rarely been evaluated or acknowledged to be a potential source of bias in the selection of cases.

Recognising the importance of facilitating pharmacovigilance studies on medicine safety during pregnancy and the need for robust, evaluated data retrieval methods, an algorithm to identify pregnancy cases was recently developed and implemented in VigiBase. Acknowledging the same need, a similar initiative has been undertaken in EudraVigilance, the European database for suspected adverse reactions to medicines [33].

The VigiBase pregnancy algorithm was developed with a global perspective, intending to account for the variability and heterogeneity of data from diverse reporting practices worldwide. Understanding the properties of this algorithm is crucial for its potential application. Therefore, the aim of the present study was to evaluate and characterise the VigiBase pregnancy algorithm.

Methods

VigiBase

The current study utilised VigiBase as the source of data. VigiBase is maintained by Uppsala Monitoring Centre and to date contains more than 40 million individual case safety reports submitted since 1968. These reports arise from a variety of sources including healthcare professionals, pharmaceutical companies and consumers, and are submitted to the relevant national pharmacovigilance centres who then share them with VigiBase.

Preprocessing of Reports in VigiBase

All reports submitted to VigiBase are processed to conform to a common data model, which is based on the ICH E2B(R2) format for electronic transmission of individual case safety reports. Regardless of the report exchange format, for example E2B(R3), the International Drug Information System - INTDIS, or custom formats, all reports are first converted to this common data model. Conversion from E2B(R3) to (R2) is based on the official scripts [34], for other formats custom conversions are used.

The preprocessing of reports involves mapping reported reaction/event and indication terms to MedDRA^® (version 25.1 in this study). For terms coded in other standard terminologies, such as the International Classification of Diseases (ICD), bridges to MedDRA^® are used; for example, the official ICD Tenth Revision (ICD-10) to MedDRA^® mapping [35]. Terms reported as free text are automatically mapped to a standard terminology, with manual mapping if automatic mapping fails. Some terms remain unmapped because of limited coding resources and unclear information.

The VigiBase Pregnancy Algorithm

The VigiBase pregnancy algorithm is designed to flag pregnancy cases where there has been a maternal exposure to a medicinal product, shortly before or during pregnancy or labour, with or without an adverse event occurring in the pregnant person or the foetus/prenatally exposed child. Paternal and lactation exposures are outside the scope. The algorithm is completely rule based and utilises multiple structured data elements in the E2B format [10] that could potentially hold pregnancy-related information. It does not account for any free text information.

Algorithm Rules

The algorithm consists of several rule-out and rule-in steps to determine if a case report qualifies as a pregnancy case (Fig. 1). First, reports are ruled out if the patient age is above 50 years (Fig. 1o1), or if the age group is “Elderly” (Fig. 1o2), with the aim of avoiding the capture of reports that are unlikely to be pregnancy related. As paternal exposures are not in the scope, reports are ruled out if the parent sex is "Male" in a parent-child/foetus report (Fig. 1o3), or if a paternal exposure term, without any co-reported maternal exposure term, is reported as a reaction/event (Fig. 1o4). If a report meets one of these rule-out steps, it will not be flagged as a pregnancy case (i.e., non-flagged case).

Fig. 1.

Flowchart describing the logical steps in the VigiBase pregnancy algorithm

Second, if no rule-out step is met reports are ruled in, i.e. flagged as pregnancy cases, if one of the following is true: a data element regarding gestation period is populated (Fig. 1i1), a seriousness criterion is given as “Congenital anomaly/birth defect” (Fig. 1i2), the route of administration is “Transplacental” (Fig. 1i3) or pregnancy is given as a concurrent condition in the medical history section (Fig. 1i4). Furthermore, reports are ruled in if the indication or reaction/event fields include a pregnancy or a foetal term (Fig. 1i5, i6), if a neonatal event is reported in a newborn up to the age of 7 days (Fig. 1i7), or if a congenital event is reported in a child under the age of 2 years (Fig. 1i8). As the neonatal and congenital events are not specific to pregnancy exposures, the age conditions aim to avoid capturing reports with pre-existing congenital events and events related to neonatal exposures. Last, reports are ruled in if the parent sex is "Female" in a parent-child/foetus report, with no reported evidence of lactation (Fig. 1i9). Reports that do not meet any of the rule-in steps (Fig. 1i1–i9) are not flagged as pregnancy cases (i.e. non-flagged cases). The details of all steps and corresponding E2B data elements are described in the Table S1 of the Electronic Supplementary Material (ESM).

Evaluation of the Algorithm

Recall and Precision Analysis

The performance of the algorithm was evaluated using recall (i.e. sensitivity) and precision (i.e. positive predictive value). For calculations of these measures, true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN) were defined according to the confusion matrix in Table 1.

Table 1.

Definition of true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN) for the algorithm

			According to algorithm
			Flagged as pregnancy case	Non-flagged case
According to annotations	Pregnancy case		True Positive (TP)	False Negative (FN)	Recall = TP/(TP+FN)
	Non-pregnancy case		False Positive (FP)	True Negative (TN)
			Precision = TP/(TP+FP)

Recall, measuring the proportion of annotated pregnancy cases that were flagged, was calculated as $\text{TP}/(\text{TP}+\text{FN})$. Precision, measuring the proportion of annotated pregnancy cases among all flagged cases, was calculated as $\text{TP}/(\text{TP}+\text{FP})$. Confidence intervals (CIs) were calculated using the binomial distribution.

Annotations

To serve as a gold standard for identification of pregnancy cases in the recall and precision analysis, manual annotations were performed on datasets of VigiBase reports (see Sect. 2.3.4.2. and 2.3.4.3.). Four experienced pharmacovigilance assessors (two medical doctors BR, RS; two pharmacists LS, AR) performed the annotations according to a pre-defined guideline, yielding the labels “pregnancy case” or “non-pregnancy case” for each annotated case (see S2 of the ESM). All pregnancy cases were also classified according to whether they concerned the pregnant person and/or the foetus/prenatally exposed child. Any uncertainties in labelling were resolved through consensus discussion between the annotators.

Inter-annotator agreement was calculated using Fleiss’ kappa score [36] on 297 randomly selected reports that were independently annotated by all four assessors (based on the initial labels before any consensus discussion). Fleiss’ kappa score is a statistical measure that is used for assessing the reliability of agreement between a fixed number of annotators when assigning categorical classifications. It accounts for class imbalance by incorporating the expected agreement in the formula.

Baseline Comparator

To contextualise the performance of the algorithm, it was compared to a baseline defined by the SMQ “Pregnancy and neonatal topics (narrow)”, subsequently referred to as the “pregnancy SMQ”. Reports that included a reaction/event term from the pregnancy SMQ were considered flagged by this baseline.

Datasets

The study used three datasets extracted from VigiBase for the evaluation: the “Full dataset”, the “Downsampled dataset” and the “Random dataset” (Fig. 2).

Fig. 2.

Sampling strategy for the three datasets used in the study. For the manual annotations in the Random dataset, note that some reports are flagged by both the pregnancy algorithm and the pregnancy Standardised Medical Dictionary for Regulatory Activities Query (SMQ) [Pregnancy and neonatal topics (narrow) SMQ] hence, the figures from the respective branches are not summable

Full Dataset

The Full dataset contained all reports in VigiBase up until the data lock point, 1 January, 2023. Excluding suspected duplicates, as identified by vigiMatch [37], this dataset comprised 33,251,759 reports and was used to analyse the algorithm’s overall output with regard to the rule-in and rule-out steps.

Downsampled Dataset

The Downsampled dataset was derived from the Full dataset and aimed to estimate the algorithm’s recall. Given the imbalanced nature of the Full dataset, with pregnancy cases estimated to be around 1% based on pre-study results, we downsampled reports less likely to involve pregnancy to increase the prevalence of pregnancy-related cases during annotation. By excluding all patients aged 45 years or older and male patients 18 years or older from this dataset, we aimed to increase the prevalence of pregnancy-related cases in the sample for annotation while minimising the risk of omitting true pregnancy cases (referring to pregnant persons aged 45 years and above or prenatally exposed adult male patients) from this relatively small sample. To maintain representativeness, the sample was stratified to match the proportions of reports with missing age and/or sex information in the Full dataset. The stratified sample was randomly ordered and served as the source dataset for the annotators.

While the primary aim of the study was not to directly compare the algorithm with the pregnancy SMQ, we ensured that it was adequately powered to detect a statistical difference. To this end we used estimates from a pre-study to inform the decision on sample size. Simulations targeting a power of 80% under a significance level of 5% revealed that approximately 250 reports labelled as pregnancy cases would be required, assuming a true relative difference of at least 10%. Case-by-case annotation produced a final Downsampled dataset of 7874 reports, including 253 pregnancy cases. See Fig. 2 for an overview of the sampling strategy.

Recall was calculated for the pregnancy algorithm and the pregnancy SMQ, performance difference assessed with McNemar’s test. The reasons for non-flagged cases were analysed together with a characterisation of all FN reports.

Random Dataset

To study precision, a separate dataset was created by drawing a random sample of 30,000 reports from the Full dataset. This sample size would ensure enough annotated pregnancy cases to compare precision of the algorithm and the pregnancy SMQ (according to simulations targeting power of 80% under a significance level of 5% provided a true relative difference of at least 10%, using estimates from a pre-study).

As precision is calculated on the positive outputs, only reports flagged by either the algorithm or the pregnancy SMQ were annotated in this dataset. This resulted in 448 annotated reports, of which 73 were flagged only by the algorithm, 104 only by the pregnancy SMQ and 271 by both approaches. Annotators received these cases for annotation blinded to whether they were flagged by the algorithm or the pregnancy SMQ. See Fig. 2 for an overview of the sampling strategy.

Precision was calculated for the algorithm and the pregnancy SMQ on the annotated part of the Random dataset, performance difference tested with the Kosinski weighted generalized score. The reasons for flagged pregnancy cases (TP, FP) were analysed, together with a characterisation of all FP reports. For transparency, precision was also calculated on the Downsampled dataset but, to avoid any biased estimations due to the enriched nature of this dataset, the main precision analysis was performed on the Random dataset.

Results

Dataset Characteristics

In the Full dataset, 80% of the reports were submitted in the E2B format (see Table 2, part i). This proportion remained consistent in both the Downsampled dataset (79%) and in the annotated part of the Random dataset (81%). The proportion of E2B reports was lower among the annotated pregnancy cases than among the non-pregnancy cases in both datasets.

Table 2.

Characteristics of reports in the Downsampled dataset and the annotated part of the Random dataset, with regard to report transmission format (i), reporting decade (ii) and whether the report concerns the pregnant person or the foetus/prenatally exposed child (iii). Figures might not equal 100% because of rounding

	Full dataset	Downsampled dataset			Annotated part of Random dataset
		Total	Annotated as non-pregnancy case	Annotated as pregnancy case	Total	Annotated as non-pregnancy case	Annotated as pregnancy case
(i) Report transmission format
E2B	26,710,932 (80%)	6251 (79%)	6076 (80%)	175 (69%)	363 (81%)	105 (88%)	258 (78%)
Other	6,537,431 (20%)	1623 (21%)	1545 (20%)	78 (31%)	85 (19%)	14 (12%)	71 (22%)
(ii) Reporting decade
2020–23	15,215,206 (46%)	3583 (46%)	3491 (46%)	92 (36%)	198 (44%)	62 (52%)	136 (41%)
2010–19	13,810,593 (42%)	3245 (41%)	3111 (41%)	134 (53%)	197 (44%)	54 (45%)	143 (43%)
2000–9	2,435,589 (7%)	585 (7%)	569 (7%)	16 (6%)	31 (7%)	2 (2%)	29 (9%)
1990–9	1,175,787 (4%)	299 (4%)	292 (4%)	7 (3%)	11 (2%)	0 (0%)	11 (3%)
1980–9	457,669 (1%)	120 (2%)	117 (2%)	3 (1%)	6 (1%)	1 (1%)	5 (2%)
–1979	156,915 (< 1%)	42 (1%)	41 (1%)	1 (< 1%)	5 (1%)	0 (0%)	5 (2%)
(iii) Patient type
Pregnant person	NA	216 (3%)	NA	216 (85%)	250 (56%)	NA	250 (76%)
Foetus/child	NA	34 (< 1%)	NA	34 (13%)	72 (16%)	NA	72 (22%)
Both	NA	3 (< 1%)	NA	3 (1%)	7 (2%)	NA	7 (2%)

NA Not applicable, either due to not annotated (Full dataset) or because annotated as non-pregnancy case (Downsampled and Random dataset)

Most reports in all datasets were submitted to VigiBase in 2010 or later. For a breakdown of reports by the decade they were submitted, see Table 2, part ii.

Among the annotated pregnancy cases in the Downsampled dataset, 85% related only to the pregnant person, 13% related only to the foetus/prenatally exposed child and 1% to both (see Table 2, part iii). In the annotated part of the Random dataset, the proportions were 76%, 22% and 2%, respectively. This is largely in accordance with the principles outlined in the ICH E2B(R3) implementation guide, as well as the European guideline on good pharmacovigilance practices (GVP), advising that a pregnancy case either relates to the pregnant person or the foetus/child and not to both in the same report [10, 38]. The proportion of reports including a pregnancy exposure term as a coded adverse event, as suggested in the ICH-endorsed guide for MedDRA^® users, “MedDRA^® Term selection: Points to consider” [39], was 35% and 47% among the annotated pregnancy reports in the Downsampled and Random datasets, respectively.

Output Analysis in the Full Dataset

In the Full dataset, the algorithm flagged 372,660 reports (1.1%) as pregnancy cases. The most prominent step for ruling in reports was “Pregnancy/foetal event” (i6), where 287,895 (77%) of pregnancy flagged reports met this rule and for 193,473 (52%) this was the sole rule met (see Fig. 3). This was followed by “Pregnancy/foetal indication” (i5), where 91,797 (25%) of flagged reports met this rule and for 51,314 (14%) this was the sole rule met. The seven rule-in steps with smaller contributions captured 25,469 (7%) of the flagged reports missed by the two most effective rule-in steps.

Fig. 3.

Distribution of rule-in steps met when the algorithm flagged reports as pregnancy cases in the Full dataset. To the left, the horizontal bars show the numbers of reports meeting the rule-in step (regardless of other rules met). The vertical bars show the numbers of reports meeting the combination of rule-in steps marked by the nodes. The tail is cropped at combinations representing less than 1% of the reports. An additional 225 combinations are not shown

Of the non-flagged cases, 32,870,896 (99.97%) reports did not meet any rule-in step and 12,813,743 (39%) reports met at least one rule-out step (see Fig. S3-1 of the ESM). Of these, 8203 reports (0.025%) were excluded solely because of meeting a rule-out step. These also met at least one rule-in step and, if not for the rule-out steps, they would have been flagged as pregnancy cases. The most prominent rule out steps met for these reports were “Patient age above 50y” (o1), met by 4962 (60%) reports, followed by “Paternal exposure event” (o4), met by 3012 (37%) reports (see Fig. S3-2 of the ESM).

Recall and Precision Analysis

Recall Analysis in the Downsampled Dataset

The Downsampled dataset consisted of 7874 annotated reports, of which 253 (3.2%) were annotated as pregnancy cases. The algorithm flagged 190 of these, resulting in a recall of 75% (95% CI 69–80) (see Table 3). The algorithm failed to flag 63 reports, of these, 48 were submitted in other transmission formats than E2B. Restricting the analysis to E2B reports increased recall to 91% (95% CI 86–95).

Table 3.

Confusion matrix for the algorithm in the downsampled dataset, for all reports (total) and E2B reports only (E2B). (TP: true positives; FP: false positives; FN: false negatives; TN: true negatives). Note that the precision estimate is only given for transparency and should be interpreted with caution because of the enriched nature of this dataset

		According to algorithm
		Flagged as pregnancy case	Non-flagged case
According to annotations	Pregnancy case	TP Total: 190 E2B: 160	FN Total: 63 E2B: 15	Recall Total: 75% E2B: 91%
	Non-pregnancy case	FP Total: 20 E2B: 16	TN Total: 7 601 E2B: 6 060
		Precision Total: 90% E2B: 91%

All but one of the 7601 true-negative cases did not meet any rule-in step. Of these, 31 reports also met a rule-out step; however the effects of the rule-out steps cannot be fully captured because of the down sampling strategy excluding certain patients from the dataset.

False-Negative Analysis in the Downsampled Dataset

The reasons for the 63 false-negative reports not being flagged as pregnancy cases by the algorithm are presented in Table 4. For 46 of those reports, a pregnancy-related ICD term, reported as an indication or a concurrent condition, failed to be mapped to a MedDRA^® term in the database preprocessing. All but two of these reports were submitted in another transmission format than E2B and the majority reported the term in non-English text.

Table 4.

Reasons for annotated pregnancy cases not being flagged as pregnancy cases by the algorithm in the Downsampled dataset

Reason for annotation as pregnancy case	Reason not flagged as pregnancy case by the algorithm		Total number of reports	Number of E2B reports
Pregnancy-related indication	Indications reported in non-English ICD text, not mapped in the database preprocessing		25	0	Preprocessing related
	Indications reported in ICD-8, or ICD-10 codes not included in the ICD-10—MedDRA® official mappinga, not mapped in the database preprocessing		5	1
Pregnancy-related concurrent condition	Pregnancy-related term reported in non-English ICD text, not mapped in the database preprocessing and unclear if ongoing pregnancy
	Pregnancy-related term among rule-in terms		8	0
	Pregnancy-related term not among rule-in terms		8	1	Preprocessing/Algorithm related
Pregnancy described in free text in medical history or narrative	Algorithm does not capture free text information		8	8	Algorithm related
Indirect information suggesting a pregnancy case	Lack of specific pregnancy information but pieces of indirect information		4	3
	Lack of specific pregnancy information other than the reported medicinal product being indicated for use only during pregnancy		3	2
Congenital event	Congenital anomaly in pregnant person report, not captured because of age conditional rule		1	0
Neonatal event	Neonatal event in pregnant person report, not captured because of age conditional rule		1	0

ICD International Classification of Diseases, MedDRA^® Medical Dictionary for Regulatory Activities

^aICD terms in chapter XXI: “Factors influencing health status and contact with health services”

For eight reports, pregnancy information was described only in free text, either in the medical history or in the case narrative, which the algorithm does not account for. For seven reports, lack of information restricted the capability of the algorithm to capture them. In four of these, different pieces of more indirect information suggested a pregnancy case. In the remaining three reports, the only evidence of a pregnancy exposure was the use of a medicinal product with approved indications restricted to use during pregnancy. Two reports were missed as a congenital or neonatal disorder was reported in a report with the patient age referring to the pregnant person. Thus, the reported ages fell outside the specified ranges defined in rule-in steps i7 and i8 (Fig. 1). Both reports were submitted in a non-E2B format in the year 2000 or earlier.

Baseline Comparison in the Downsampled Dataset

The pregnancy SMQ detected 157 of the 253 reports annotated as pregnancy cases in the Downsampled dataset. This is equivalent to a recall of 62% (95% CI 56–68) and represents a statistically significant (p < 0.001) reduction compared to the recall of the algorithm (75%). Restricting the analysis to E2B reports increased the recall of the pregnancy SMQ to 79% (CI 73–85), yet this represented a significantly (p < 0.001) lower recall than the algorithm (91%).

Figure 4 shows the overlap between the annotations, the pregnancy algorithm and the pregnancy SMQ. The pregnancy SMQ correctly captured three of the pregnancy cases missed by the algorithm. These reports involved congenital or neonatal events and were ruled out by the algorithm because of patient ages being outside the age conditions or not stated. All were reported in other formats than E2B. The algorithm correctly flagged 36 pregnancy cases that were not captured by the SMQ. The majority of those (32 cases) met the rule-in step “Pregnancy/foetal indication” (i5), while they did not include a reaction/event from the pregnancy SMQ.

Fig. 4.

Downsampled dataset (7874 reports) illustrating the overlap between the annotated pregnancy cases (gold standard; 253 reports), the cases flagged by the pregnancy algorithm (210 reports) and those captured by the pregnancy Standardised Medical Dictionary for Regulatory Activities Query [SMQ] (191 reports)

Precision Analysis in the Random Dataset

The algorithm flagged 344 reports as pregnancy cases in the Random dataset (316 correct, 28 incorrect) resulting in a precision of 92% (95% CI 88–95) (see Table 5). Most of the correctly flagged reports (TPs) were ruled in through the steps “Pregnancy/foetal event” (i6) or “Pregnancy/foetal indication” (i5). For the former, 255 (74%) reports met this criterion, and this was the sole rule met for 168 (49%) reports. For the latter, 96 (28%) reports met this criterion, and this was the sole rule met by 55 (16%) reports. Figure 5 shows the contribution of each rule-in step, individually and in combination with other steps.

Table 5.

Confusion matrix for the algorithm in the Random dataset, for all reports (Total) and E2B reports only (E2B). Note that in this dataset, only the reports flagged by either the algorithm or the pregnancy SMQ were annotated, consequently, the negative values are not suitable for analysis and are omitted. (TP: true positives; FP: false positives)

		According to algorithm Flagged as pregnancy case
According to annotations	Pregnancy case	TP Total: 316 E2B: 257
	Non-pregnancy case	FP Total: 28 E2B: 23
		Precision Total: 92% E2B: 92%

Fig. 5.

Distribution of algorithm rule-in steps met by reports flagged as pregnancy cases in the Random dataset, divided by true positives (TP) and false positives (FP). To the left, the horizontal bars show the numbers of reports meeting the rule-in step (regardless of other rules met). The vertical bars show the numbers of reports meeting the combination of rule-in steps marked by the nodes. The tail is cropped at combinations representing less than two reports. An additional 23 combinations are not shown

False-Positive Analysis in the Random Dataset

The 28 incorrectly flagged reports (FPs) were spread across seven of the nine rule-in steps (Fig. 5). The main reason was the ruling in of reports through “Pregnancy/foetal indication” (i5), which accounted for ten of the FPs (36%). Seven of these described a postpartum indication, such as postpartum haemorrhage, and two related to planning a pregnancy. According to our case definition, these were not considered pregnancy cases. The rule-in step “Pregnancy/foetal event” (i6) accounted for six (21%) of the FPs. Three of these included events that are not pregnancy specific but often related to pregnancy, such as chloasma, hence they were part of the pregnancy SMQ used for ruling in reports.

In seven reports, the pregnancy-related information was inferred to be miscoded. The miscoded information appeared in several different data elements, such as a foetal event term reported for an adult and parent information given in a child vaccination report. Seemingly miscoded information was also given for the seriousness criterion, route of administration and gestation period. Five reports, spread across several rules, exhibited ambiguity because of missing or unclear information and potentially represented true pregnancy cases.

Baseline Comparison in the Random Dataset

The pregnancy SMQ had a precision of 74% (95% CI 69–78), which was significantly lower (p < 0.001) than the precision of the algorithm (92%). Because lactation exposures are out of the scope of the algorithm, a sub-analysis was conducted excluding the “Lactation and related topics (incl neonatal exposure through breast milk)” sub-SMQ from the pregnancy SMQ search. This slightly increased the precision to 76% (95% CI 72–81).

Figure 6 shows the overlap between the annotations, the pregnancy algorithm and the pregnancy SMQ. The SMQ captured 98 reports that were not annotated as pregnancy cases. Of these, 91 reports were correctly not flagged by the algorithm owing to the age conditions for ruling in reports in step “Neonatal event conditioned by age” (i7) or “Congenital event conditioned by age” (i8) [72 cases] and/or the ruling out of “Patient age above 50 years” (o1) or “Patient age group elderly” (o2) [26 cases]. The SMQ correctly captured 13 cases that were missed by the algorithm. Of these, 11 were incorrectly ruled out by the algorithm through the neonatal or congenital event age conditional rules because of a patient age referring to the pregnant person or missing age information. With one exception, these were all submitted in a non-E2B format in 2005 or earlier.

Fig. 6.

Annotated part of the Random dataset (448 reports), illustrating the overlap between the annotated pregnancy cases (gold standard; 329 reports), the cases flagged by the pregnancy algorithm (344 reports) and those captured by the pregnancy Standardised Medical Dictionary for Regulatory Activities Query [SMQ] (375 reports)

Inter-Annotator Agreement

Of the 297 reports that were independently annotated by all assessors, three reports received conflicting labels. This resulted in a Fleiss’ kappa score of 83%, indicating a very good level of inter-annotator agreement.

Discussion

Overall Performance

This comprehensive evaluation of the algorithm for identifying pregnancy cases in VigiBase shows a robust performance and highlights aspects of the algorithm that enabled correct identification of reports, and where potential improvements could be made. It also provides insights into the reporting of pregnancy exposure information in case reports and demonstrates the influence of report exchange formats and coding practices in a global setting.

In our analysis, the algorithm successfully flagged 75% of the reports manually annotated as pregnancy cases. Among the pregnancy cases undetected by the algorithm, the majority were attributed to preprocessing issues arising from reporting formats or terminologies other than E2B and MedDRA^®, particularly relating to unmapped indications reported in non-English text. Restricting the analysis to E2B reports substantially increased the recall to 91%. For these reports, the primary reason for false negatives was the failure to detect reports containing pregnancy information exclusively reported in free-text fields, owing to the algorithm’s inability to process such fields. Reports were also missed because of incompleteness of information, such as the lack of explicit pregnancy-specific information, or the existence of explicit, yet sparse, pregnancy details.

The age thresholds for ruling in reports with neonatal and congenital events were implemented to allow for capturing additional relevant cases not detected through the other rules, while avoiding the erroneous flagging of cases with pre-existing congenital events or postnatal exposures. A few cases, mostly older non-E2B reports, were inadvertently missed because of these age restrictions as the patient age referred to the pregnant person or was missing. However, this cautious approach was essential to protect against numerous incorrect classifications as evidenced in our precision analysis. These age conditions, along with the ruling out of patients aged over 50 years, were the main contributors to the algorithm’s relatively higher precision compared with the pregnancy SMQ.

Most of the cases mistakenly flagged as pregnancy cases by the algorithm included post-partum indications and adverse events that were included in the pregnancy SMQ but that were not specific to pregnancy. Miscoding of pregnancy-related information as a source of false positives was spread over several different data elements and did not indicate one specific underlying issue.

Baseline Comparison

The “Pregnancy and neonatal topics (narrow)” SMQ was chosen as the baseline method to benchmark the performance of the algorithm. This was the method applied in several studies for retrieving pregnancy cases [13–15, 21, 27–30, 33] and it was the comparator used for evaluation of the algorithm developed in EudraVigilance [33]. We note that the intention of our study was not to compare the performance of the algorithm with the SMQ to decide which method is better, but to understand where they perform differently and that they may be used in different scenarios. The SMQ plays an important role in the algorithm and is the main contributor to the high recall. Also it should be noted that the comparison is restricted to coded adverse events for the SMQ. With this in mind, the algorithm captured more pregnancy cases than the SMQ and did so more precisely. The few cases that the SMQ correctly captured but the algorithm missed were primarily older non-E2B reports, captured by the SMQ because it did not apply age conditions for congenital and neonatal events. However, for the same reason, this led to more false positives for the SMQ, resulting in a significantly lower precision.

Evaluation Approach

To ensure a rigorous evaluation, we established large annotated datasets, enhancing data diversity and coverage of potential pregnancy-related scenarios. Four experienced annotators, each contributing to a large volume of annotations, adhered to a guideline that provided instructions to follow during the annotation process. Two of the annotators had been involved in developing the algorithm, hence we included two additional assessors to ensure annotation robustness. The annotated data showed high inter-annotator agreement, indicating consistent and reliable annotations.

The annotations were based only on the reported information available in VigiBase, which sometimes, because of incomplete data, limited the ability to reach definite conclusions about a pregnancy exposure. Given also that narrative information is not shared with VigiBase for all reports, it is plausible that cases with pregnancy-related information confined to the narrative were not correctly annotated as pregnancy cases. Consequently, we may have underestimated the true number of pregnancy-related reports in VigiBase.

The primary challenge in this study was estimating recall, given that most reports in VigiBase do not pertain to pregnancy. Annotating a random sample from the entire database would have demanded an unreasonable amount of resources to identify enough pregnancy cases for the required statistical power. To address this, our strategy for creating the dataset to estimate recall excluded reports less likely to be pregnancy related, thus increasing the prevalence of pregnancy cases in our sample. This allowed us to annotate the number of pregnancy reports required, while being able to demonstrate the use of the algorithm in a dataset that is sufficiently representative to be applicable to the whole of VigiBase. Consequently however, we could not directly evaluate the implications of the algorithm ruling out certain individuals, specifically patients aged over 50 years, as these were already excluded from the Downsampled dataset. Nevertheless, the comparison with the SMQ output in the Random dataset suggested that ruling out cases based on age is an effective strategy for enhancing precision without compromising recall.

To assess the algorithm’s ability to identify pregnancy cases within all of VigiBase, we intentionally refrained from imposing any restrictions on the time of reporting or the reporting formats of the test data. Hence, the reports included in the evaluation dated back to the 1970s and represented different reporting formats and practices that may have changed over the years. Our analysis showed that the algorithm performs better when applied to reports submitted in the E2B format. Not only is the E2B format more extensive and able to accommodate additional information, including data related to pregnancy, compared with other formats like the International Drug Information System - INTDIS, reports in the E2B format are also more likely to comply with current international standards for coding and reporting. While this increases the potential for well-documented reports, importantly, it does not necessarily ensure them. Additionally, of note, the algorithm was developed within VigiBase, which conforms to the E2B(R2) standard, hence, the rules were based on this data model, naturally favouring this type of report.

Applicability

The applicability of the VigiBase pregnancy algorithm is influenced by several factors, including intrinsic factors such as the exclusion of patients aged over 50 years, and extrinsic factors such as database characteristics related to regional reporting practices. Alternative methods may offer higher precision, making them suitable for studies prioritising precision over recall. These methods could, for example, rely on reported pregnancy exposure terms; importantly, we noted that less than half of the annotated pregnancy cases in our study included such a term. Conversely, if capturing all reports for a certain adverse event is crucial, a specific term search may be more appropriate, especially in studies involving a manual review to exclude false positives. Future studies could explore different applications of the algorithm and compare with other more or less inclusive approaches, to inform decision making.

The precision estimate for the pregnancy algorithm developed in EudraVigilance, which uses the E2B reporting format and MedDRA^® terminology, was very similar to that for the VigiBase algorithm. The performance was evaluated on 100 cases retrieved by the EudraVigilance algorithm and resulted in a precision of 90% (95% CI 84–96) [33]. Recall was not evaluated in the study. Importantly, the two approaches somewhat differ in scope: the EudraVigilance algorithm intends to capture only adverse outcomes and includes paternal exposures, the VigiBase algorithm intends to capture all pregnancy exposures regardless of outcome and excludes paternal exposures. Still, the methods are similar and comparing the outputs may inform future developments in this area.

Our study suggests that most of the pregnancy cases in VigiBase refer to the pregnant person while a minority refer to the foetus/prenatally exposed child. Hence, for certain studies focusing on outcomes in the foetus/child, it would be desirable to restrict the analysis to a subset of such reports. As an example, in signal detection, the statistical background is important, and pregnant person reports heavily outnumbering foetus/child reports may risk masking specific patterns in the subgroup of interest. Furthermore, as the algorithm captures both type of reports, applying it with the aim to retrieve a case series on foetal/child outcomes would potentially require manual data cleaning to remove cases referring to the pregnant person and vice versa.

Although the algorithm was initially developed and evaluated in a global database, we expect it to be applicable to other databases conforming to the E2B format and using MedDRA^® for coding. The algorithm follows an explainable and transparent rule-based method that would be relatively easy to replicate in other databases and adjust to local requirements. The current evaluation provides insights to manage expectations as the algorithm’s performance may vary across databases because of differences in their structures and regional reporting patterns.

Future Improvements

While transparency is crucial for understanding algorithm outputs and is a key strength of rule-based methods, incorporating additional report data, such as the case narrative, and using more advanced methods, like large language models, could enhance the identification of pregnancy cases, though potentially trading off transparency, reproducibility and reliability. An additional rule-based feature would be to use the age group “Foetus”, which was introduced in the E2B(R3) format. This feature was not explored as the VigiBase cohered to the E2B(R2) data model at the time the algorithm was developed. The E2B(R3) format also introduced an opportunity to extend it with additional data elements. If used in a standardised way, this could be an option for transferring a database-specific pregnancy flag. Our study also suggests that medicinal products indicated solely for use during pregnancy could provide a means for capturing exposures during pregnancy. However, regional differences and maintenance requirements limit the feasibility, at least for a global application. Investigating the potential to distinguish between reports referring to the pregnant person and the foetus/child, as well as exploring separate algorithms for detecting reports related to paternal or lactation exposures, may enhance data precision and broaden the scope of pregnancy-related investigations.

Given the findings of this study, the most crucial step for effectively retrieving pregnancy-related case reports in pharmacovigilance databases would however be to achieve global harmonisation of standards for collecting and exchanging pregnancy data. This would not only facilitate the identification of such case reports but once identified, well documented and dependable reports are key to allow for valid case assessments. The consequence of incomplete information became evident during the annotation process as case reports providing sparse or implicit pregnancy information made it difficult even for a human to determine definitively whether a case was pregnancy related or not. This highlights also the need for improved reporting and coding guidelines, as well as adherence to those that already exist. Encouragingly, recent initiatives within the ConcePTION project have suggested frameworks for data collection to increase the quality of pregnancy data and how to assess the quality of the information in pregnancy-related case reports [11, 12, 40, 41]. In addition to improving frameworks and guidelines, however, it is crucial to implement training efforts that raise awareness about how to report pregnancy information.

In summary, our study highlights the importance of harmonised global standards for reporting pregnancy exposures, and compliance with those that exist, for the retrievability of pregnancy-related case reports. Although the current standard transmission format allows for reporting pregnancy-related information through many different data elements, the necessity for a pregnancy algorithm stems from the absence of a dedicated field to report and, subsequently, retrieve pregnancy exposure cases [11, 12]. While various approaches are currently employed to identify pregnancy-related cases, thorough performance evaluations are notably absent in most studies. This underscores the importance of our study, recognising the performance of the VigiBase pregnancy algorithm in terms of recall and precision, as well as understanding its limitations and potential biases.

Conclusions

The VigiBase pregnancy algorithm demonstrates robust performance in our comprehensive evaluation using large annotated datasets, highlighting its potential to facilitate pharmacovigilance related to pregnancy exposures. The findings provide insights into the algorithm’s strengths and areas for improvement and establish a valuable benchmark for future research. Additionally, our study emphasises the need for global harmonisation of standards for reporting pregnancy exposures to enhance data consistency, completeness and retrievability.

Supplementary Information

Below is the link to the electronic supplementary material.

Declarations

Funding

The authors received no financial support for the research, authorship and/or publication of this article.

Conflict of Interest

Lovisa Sandberg, Sara Hedfors Vidlin, Levente K-Pápai, Ruth Savage, Boukje C. Raemaekers, Henric Taavola-Gustafsson, Annette Rudolph, Lucy Quirant, Tomas Bergvall, Magnus Wallberg and Johan Ellenius have no conflicts of interest that are directly related to the content of this article. Ruth Savage is an Editorial Board member of Drug Safety. She was not involved in the selection of peer reviewers nor any of the subsequent editorial decisions. Lucy Quirant, as of May 2023, is no longer employed by the Uppsala Monitoring Centre. However, her contributions to the study were completed prior to the time of departure, as part of her employment at the Uppsala Monitoring Centre.

Ethics Approval

Not applicable. This study did not use personal data.

Consent to Participate

Not applicable.

Consent for Publication

Not applicable.

Availability of Data and Material

The data that support the findings of this study are not publicly available. Access to the data is restricted based on the conditions for access within the WHO Programme for International Drug Monitoring. Subject to these conditions, data are available from the authors on reasonable request. For further inquiries, please contact Uppsala Monitoring Centre via https://who-umc.org/contact-information/.

The VigiBase pregnancy algorithm is implemented in VigiLyze, the tool for searching and analysing VigiBase data, which is available to national pharmacovigilance centres as members of the WHO Programme for International Drug Monitoring. An initial version of the algorithm was implemented in November 2022. The version evaluated in the study, which includes some minor updates, was implemented in February 2024. VigiBase data, including output of the VigiBase pregnancy algorithm, are also available through VigiBase Custom Searches.

Code Availability

The code used for this study is not provided. The implementation of the VigiBase Pregnancy Algorithm in R will be made available at https://github.com/Uppsala-Monitoring-Centre.

Authors’ Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by LS, SHV, LK-P, HT and RS. Manual annotations were performed by LS, AR, BCR and RS. The first draft of the manuscript was written by LS, SHV, LK-P and RS, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.