Understanding the variability of pharmaco‐epidemiological studies assessing the risk of appendicitis with mRNA COVID‐19 vaccines

Key Points.

• Results of cohort studies evaluating the risk of appendicitis after mRNA COVID‐19 vaccines are heterogeneous.

• Our analysis suggests that the discrepancies between results of these studies is mainly linked to the choice of control groups.

• Selection of comparator group was particularly difficult with COVID‐19 vaccines given the large proportion of population exposed, often by using the same vaccine platform.

• Using self‐controlled design could be an interesting option to addresses these issues.

• Sensitivity analyses are crucial to assess the degree of variability in the results of a study according to methodological choices.

1. INTRODUCTION

Whether mRNA COVID vaccines could increase the risk of appendicitis has been raised from the phase 3 clinical trial of BNT162b2 in which the rate of appendicitis disproportionately affected vaccinated volunteers. ¹ The US Food and Drug Administration advisory committee thus enjoined for further signal detection efforts in adding appendicitis as an adverse effect of special interest. ² This signal has then been assessed in three large nationwide cohort studies who provided very discrepant results. ³ , ⁴ , ⁵ Whether three large pharmaco‐epidemiological studies can produce widely discrepant results in using the same design for a relatively simple outcome deserve to be explored.

2. METHOD

We extracted study characteristics and incidence rates of appendicitis from the three studies.

We then performed a random‐effect meta‐analysis of incidence rate of appendicitis in control and vaccinated groups and of incidence rate ratio (IRR) in vaccinated compared to nonvaccinated groups. Statistical analysis was performed with R (version 4.1.1).

3. RESULTS

The main methodological characteristics of these studies are described in Table 1. All these studies assessed the risk of appendicitis 21 days after vaccination using a cohort design. However, outcomes definitions and appendicitis identification were slightly different due to different coding systems (ICD‐9 or ICD‐10) and selective inclusion of appendectomies. More importantly, the choice of the comparator group and matching/weighting procedures were highly heterogeneous between studies (Table 1).

TABLE 1.

Synthesis of main methodological characteristics of pharmacoepidemiological studies assessing the risk of appendicitis after mRNA COVID‐19 vaccines

	Study design	Follow‐up time	Inclusion criteria	Exclusion criteria	Outcome definition	Exposure definition	End of follow‐up
Barda N et al. NEJM 2021 3	Cohort study on Clalit Health Services (CHS) database (a 52% representative sample of Israeli population) with matching 1:1 on baseline variables. Unvaccinated patients matched on a given day could become vaccinated on a future date and newly eligible for inclusion as a vaccinated person	42 days (21 days after each dose)	‐16 years or older ‐Continuous membership in the CHS during one full year ‐No previous SARS‐CoV‐2 infection ‐No contact with the healthcare system in previous 7 days	‐Previous diagnosis of appendicitis ‐Long term care facility residents ‐Persons confined to their homes for medical reasons ‐Health care workers ‐Persons for whom data on BMI or residential area were missing	ICD‐9 diagnosis codes: ‐540: Acute appendicitis ‐541: Unspecified appendicitis ‐542: Other appendicitis ICD‐9 procedure code: ‐47: Operation on appendix	A dose of BNT162b2 vaccine	Earliest of one of the following: ‐Documentation of the appendicitis ‐21 days after a vaccine dose ‐May 24, 2021 ‐Death ‐Diagnosis of SARS‐COV2 infection in either vaccinated or unvaccinated patient will end the follow‐up for both
Klein N and al. JAMA. 2021 5	Cohort study composed of only vaccinated patients present in the 8 data‐contributing health plans, representing 3% of the US population. Comparison of appendicitis incidence during a risk interval of days 1 to 21 after vaccination with appendicitis incidence for those who were concurrently (same calendar day) in comparison interval (22 to 42 days after their most recent COVID‐19 vaccination)	42 days (21 days after each dose)	‐12 years or older ‐Present in one of the eight data‐contributing health plans ‐BNT162b2 or mRNA‐1273 vaccination	None	ICD‐10 codes: ‐K35: Acute appendicitis ‐K36: Other appendicitis ‐K37: Unspecified appendicitis ‐K38.8: Other specified diseases of appendix	A dose of BNT162b2 or mRNA‐1273 vaccine	Earliest of one of the following: ‐21 days after a vaccine dose ‐June 26, 2021
Kildegaard H et al. JAMA Internal Medicine. 2022 4	Cohort study on Danish nationwide registers. Evaluation of the risk of appendicitis during the 21 days following vaccine dose. Reference group is unvaccinated patients matched on weighting to adjust for potential confounders. Individuals with previous appendicicitis or appendicectomy were excluded	42 days (21 days after each dose)	‐12 years or older ‐BNT162b2 or mRNA‐1273 vaccination	‐Previous appendicitis or appendectomy ‐Individuals immunized with non‐mRNA COVID‐19 vaccines ‐Individuals with PCR‐confirmed SARS‐CoV‐2 before study inclusion	ICD‐10 codes: ‐K35: Acute appendicitis ‐K36: Other appendicitis ‐K37: Unspecified appendicitis Appendectomy diagnosis code in Danish Pathology Register (SNOMED codes)	A dose of BNT162b2 or mRNA‐1273 vaccine	Earliest of one of the following: ‐21 days after a vaccine dose ‐Occurrence of the appendicitis ‐November 30, 2021 ‐Death ‐SARS‐COV2 infection ‐Immunization against COVID‐19 for unvaccinated patients

The results of the meta‐analysis assessing the risk of appendicitis was IRR = 1.02 (95% CI 0.78, 1.34), with a large heterogeneity (I² = 80%) (Figure 1). The incidence of appendicitis in vaccinated groups was highly similar for two studies (Barda et al and Kildegaard et al.) with an incidence of 1487 (95% CI 1216, 1818) and 1456 (95% CI 1350, 1571) cases per 1 000 000 person years respectively. The incidence of appendicitis was inferior in the Klein et al. study, probably because of the noninclusion of appendicectomies. However, the incidence of appendicitis was more heterogeneous in control groups notably for Barda et al. and Kildegaard et al. with an incidence of 1033 (95% CI 811, 1315) and 1512 (95% CI 1404, 1630) per 1 000 000 person years, respectively. This large discrepancy in incidence rate in control groups explain the discrepant results of these two studies, IRR = 1.44 (95% CI 1.05, 1.97) for Barda et al. and 0.96 (95% CI 0.87, 1.07) for Kildegaard et al.

FIGURE 1.

Forest plot of Incidence Rate of appendicitis in vaccinated and control groups, and Incidence Rate Ratio of appendicitis with mRNA COVID vaccines compared to nonvaccinated.

4. DISCUSSION

The case of appendicitis well illustrate the range of options available to pharmacoepidemiological researchers when designing a study. ⁶ Multitudes of methods are possible which all have their pros and cons, making the evaluation of a safety signal on the basis of these results challenging. The result of the meta‐analysis of these three studies is in favor of an absence of risk of appendicitis 21 days after mRNA COVID‐19 vaccination but with a strong heterogeneity, making the unique role of sampling variance unlikely. Heterogeneity between observational studies addressing the same research question may occur from differences in data sources, set of inclusion/exclusion criteria, confounders, duration of baseline assessment, date of entry in the cohort, outcome and exposure risk algorithm and analytical strategies… ⁷ , ⁸ Our analysis suggests that the discrepancies between results of these studies is mainly linked to the choice of control groups. Comparator selection is cornerstone in observational studies to limit confounding by indication, leading to an imbalance between treatment groups in the baseline level of risk for the outcome of interest. Use of active‐comparator and new‐user design in cohort studies is currently one of the best option for increasing comparability of measured and unmeasured confounding factors between groups before further statistical adjustment and to account for time‐related bias (i.e. ‘depletion of the susceptible’ and immortal time bias). ⁹ , ¹⁰

The choice of a suited comparator group was particularly difficult for studies assessing adverse events risks after COVID‐19 vaccines given that, in western countries, a large proportion of the population have been vaccinated, often by using the same vaccine platform (i.e. mRNA COVID‐19 vaccines), thus no active‐comparator group could be found and nonvaccinated vanished rapidly. Moreover, given the dynamic nature of the pandemic most of the control and vaccinated individuals have been infected by the SARS‐CoV2 making it difficult to distinguish between the role of the vaccine or the virus.

To deal with these difficulties and increase the comparability between groups Klein et al, used post vaccinated patients as comparator. Whether this strategy may increase global mean comparability of patient's characteristics, it do not ensure individual similarity of baseline risk of appendicitis between exposed and control groups. One of the main methodological solution, that have not been used to assess the risk of appendicitis after COVID‐19 vaccination, is to use self‐controlled designs. Indeed, case only designs such as self‐controlled case series are particularly adapted to repeated intermittent drug exposure with risk periods immediately following drug use. ¹¹ Given the difficulty to identify nonvaccinated‐noninfected groups for which the exchangeability assumption could be reasonably met, these designs could be a relevant alternative. They have, for example, been recently successfully used to assess the risk of arterial and venous thromboembolic events with mRNA or adenoviral based COVID vaccines. ¹² , ¹³ , ¹⁴

These results further stress the importance of conducting sensitivity analyses in pharmaco‐epidemiological studies, which make it possible to assess the degree of variability in the results of a study according to the methodological choices made. Such sensitivity analyses must ideally be formalized a priori rationale to aid decision‐maker assessment and increase the utility of the findings. ¹⁵ Yet, recent studies have demonstrated that conducting hundreds or thousands of sensitivity analyses to evaluate the influence of design and analysis choices for studies conducted in healthcare database is becoming feasible and could be one of the future direction to appraise the robustness of pharmacoepidemiological results. ⁷

CONFLICT OF INTEREST

The author have no conflict of interest related to this study.

Jambon‐Barbara C, Bernardeau C, Cracowski J‐L, Khouri C. Understanding the variability of pharmaco‐epidemiological studies assessing the risk of appendicitis with mRNA COVID‐19 vaccines. Pharmacoepidemiol Drug Saf. 2023;32(1):87‐90. doi: 10.1002/pds.5560

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available in related articles.