A broad assessment of covid-19 vaccine safety using tree-based data-mining in the vaccine safety datalink

Abstract

Background

Except for spontaneous reporting systems, vaccine safety monitoring generally involves pre-specifying health outcomes and post-vaccination risk windows of concern. Instead, we used tree-based data-mining to look more broadly for possible adverse events after Pfizer-BioNTech, Moderna, and Janssen COVID-19 vaccination.

Methods

Vaccine Safety Datalink enrollees receiving ≥1 dose of COVID-19 vaccine in 2020–2021 were followed for 70 days after Pfizer-BioNTech or Moderna and 56 days after Janssen vaccination. Incident diagnoses in inpatient or emergency department settings were analyzed for clustering within both the hierarchical ICD-10-CM code structure and the post-vaccination follow-up period. We used the self-controlled tree-temporal scan statistic and TreeScan software. Monte Carlo simulation was used to estimate p-values; p = 0.01 was the pre-specified cut-off for statistical significance of a cluster.

Results

There were 4.1, 2.6, and 0.4 million Pfizer-BioNTech, Moderna, and Janssen vaccinees, respectively. Clusters after Pfizer-BioNTech vaccination included: (1) unspecified adverse effects, (2) common vaccine reactions, such as fever, myalgia, and headache, (3) myocarditis/pericarditis, and (4) less specific cardiac or respiratory symptoms, all with the strongest clusters generally after Dose 2; and (5) COVID-19/viral pneumonia/sepsis/respiratory failure in the first 3 weeks after Dose 1. Moderna results were similar but without a significant myocarditis/pericarditis cluster. Further investigation suggested the fifth signal group was a manifestation of mRNA vaccine effectiveness after the first 3 weeks. Janssen vaccinees had clusters of unspecified or common vaccine reactions, gait/mobility abnormalities, and muscle weakness. The latter two were deemed to have arisen from confounding related to practices at one site.

Conclusions

We detected post-vaccination clusters of unspecified adverse effects, common vaccine reactions, and, for the mRNA vaccines, chest pain and palpitations, as well as myocarditis/pericarditis after Pfizer-BioNTech Dose 2. Unique advantages of this data mining are its untargeted nature and its inherent adjustment for the multiplicity of diagnoses and risk intervals scanned.

1. Introduction

Vaccine safety monitoring in the U.S. began immediately after the Food and Drug Administration (FDA) issued the first Emergency Use Authorization for a COVID-19 vaccine in December 2020. Among the systems in place were the Centers for Disease Control and Prevention (CDC)’s Vaccine Safety Datalink (VSD), which was conducting active population-based surveillance for 23 outcomes, using sequential analysis [1]; the CDC- and FDA-managed Vaccine Adverse Event Reporting System (VAERS), a spontaneous reporting system [2], [3]; and the new opt-in smartphone-based v-safe, a cohort-based event monitoring system [2], [4]. Vaccine safety monitoring in Europe, Israel, and the U.S., to date has identified four serious but rare vaccine-associated adverse events: anaphylaxis after the Pfizer-BioNTech (BNT162b2) and Moderna (mRNA-1273) mRNA vaccines and the Janssen (Ad26.COV2.S) vaccine [5], [6], myocarditis/pericarditis after the mRNA vaccines [7], [8], and thrombosis with thrombocytopenia syndrome (TTS) and Guillain-Barré syndrome (GBS) after the Janssen vaccine [9], [10], [11]. The CDC’s Advisory Committee on Immunization Practices (ACIP) has critically assessed the risks and benefits of these three vaccines repeatedly over time and issued revised recommendations, including one issued in mid-December 2021 for the preferential use of mRNA vaccines over the Janssen vaccine [12], [13].

To complement these COVID-19 vaccine safety monitoring methods, the VSD employed a data-mining approach to look for possible associations between receipt of a COVID-19 vaccine and any of an extremely broad range of medically attended adverse events. The method involves scanning vaccinees’ diagnosis data during follow-up to check for statistically unusual temporal clustering of cases for thousands of outcome categories. This signal-detection method has been previously applied to three vaccines recommended for adolescents and young adults, the quadrivalent meningococcal vaccine and the quadrivalent and nonavalent human papillomavirus vaccines [14], [15], [16], and two vaccines recommended for older adults, the live viral and recombinant herpes zoster vaccines [17], [18]. In this work, the method has not identified unanticipated adverse events but has found known vaccine-associated adverse events and has produced few false signals. Our objective in the current study was to use the method to further assess the safety of the primary series of the three authorized or approved COVID-19 vaccines in the U.S. during 2020–2021, Pfizer-BioNTech, Moderna, and Janssen, considering it a screening tool capable of identifying unexpected adverse events (generating hypotheses), which could be further investigated in a targeted study.

2. Methods

2.1. Study population

The VSD is a collaboration between CDC and 9 integrated healthcare organizations with medical record and administrative data for a population of more than 12 million people. VSD investigators have performed population-based research on vaccine safety in the U.S. since 1990 [19]. The study population consisted of COVID-19 vaccinees who were enrolled at a VSD site from at least 400 days prior to vaccination through a pre-specified follow-up period after vaccination: 70 days after Dose 1 of Pfizer-BioNTech or Moderna and 56 days after Janssen vaccination. COVID-19 vaccines administered between December 2020 and December 2021 were included. Data were extracted from VSD sites’ up-to-date dynamic data files, which contain data from electronic medical records, claims, and state immunization registries.

Analyses were conducted for all ages combined, as well as separately for the following age groups for which the respective vaccine was authorized or approved: ages 5–11, 12–17, 18–39, 40–64, and ≥65 years.

2.2. COVID-19 vaccine exposure and follow-up

We conducted separate analyses for Pfizer-BioNTech (ascertained by CVX codes 208, 217, and 218), Moderna (CVX code 207), and Janssen (CVX code 212). For the Janssen vaccine, which had a 1-dose primary series, the follow-up period was 56 days. In view of the 2-dose primary series for mRNA vaccines and the recommended dose-spacing of 21 days for Pfizer-BioNTech and 28 days for Moderna, we used a 70-day follow-up period after the first dose for those vaccines, allowing capture of up to 6–7 weeks after the second dose. No Dose 2-specific analyses were done, as we assumed Dose 2-specific adverse events would be detectable using this longer follow-up period, in view of the close adherence to the recommended dose-spacing in the VSD. (In prior studies of multi-dose vaccines using tree-based data-mining, we either evaluated only the first dose [15], [16] or included all doses without distinguishing among them [18]).

2.3. Hierarchical diagnosis tree and outcomes of interest

We identified outcomes using International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) codes. ICD-10-CM codes have a hierarchical tree-like structure, starting with 21 broad categories of diagnoses, e.g., diseases of the musculoskeletal system and connective tissue, which progressively branch into more and more specific sets of diagnoses, culminating in highly specific diagnosis codes (Table 1 ). The ICD-10-CM tree has at most 7 levels, depending on the branch. We did not look for clustering in Levels 1, 2, or 7, so as not to expend statistical power looking for clusters that were either too broad or too specific to be of clinical significance.

Table 1.

Example of hierarchical organization of ICD-10-CM coding system; this branch of the tree does not have a seventh level.

Level	Code range or code	Description
1	M00-M99	Diseases of the musculoskeletal system and connective tissue
2	M60-M63	Disorders of muscles
3	M62	Other disorders of muscle
4	M62.8	Other specified disorders of muscle
5	M62.83	Muscle spasm
6	M62.830	Muscle spasm of back

We removed the following categories of diagnoses from consideration, as not plausibly vaccine-related, at least not within the few weeks after vaccination: neoplasms (C00–D49); congenital malformations, deformations, and chromosomal abnormalities (Q00–Q99); external causes of morbidity (V00–Y99); and factors influencing health status and contact with health services (Z00–Z99). We also removed codes for certain conditions originating in the perinatal period (P00–P96) and codes for pregnancy, childbirth, and the puerperium (O00–O9A), as self-controlled analysis of these outcomes would be subject to time-varying confounding [20] due to gestational stage likely being related to both the timing of vaccination and the timing of birth. Other monitoring efforts have been conducting surveillance for pregnancy-related COVID-19 vaccine adverse events [21], [22], [23], [24].

Included in analysis were incident diagnoses recorded in the inpatient or emergency department setting. An “incident” case was defined as one that was not preceded by another ICD-10 diagnosis code having the same first 3 characters (i.e., in the same third level of the tree) in any setting (including outpatient) during the prior 400 days. We chose 400 days to enable ascertainment of pre-existing conditions that might have been recorded at a visit roughly 1 year prior, considering that some patients seek preventive care on an approximately annual basis.

We evaluated case clustering in all risk windows of length 2 through x/2 days that started on or after Day 1 after Dose 1, where x was the follow-up period (70 days or 56 days). The self-controlled comparison period for each defined risk window consisted of the days within the follow-up period that were not in the risk window being evaluated. Events on the day of the first dose (Day 0) were excluded because a diagnosis on that date might reflect a health condition that started prior to vaccination. However, events on the day of the second dose were included, as follow-up was anchored on the first dose.

3. The tree-temporal scan statistic

We used the self-controlled tree-temporal scan statistic, which does not require pre-specifying either specific health outcomes of interest or any specific post-exposure period of putative increased risk. With the tree-temporal scan statistic, one considers many potential clusters of cases across two dimensions in combination:1) the hierarchical structure (tree) of diagnoses (ICD-10-CM codes, in this study) and 2) time, i.e., cases of each diagnosis group are checked for temporal clustering within the pre-defined post-exposure follow-up window. In using the tree-temporal scan statistic with a self-controlled design, the comparison is within person among time periods, which controls for time-invariant potential confounders such as chronic disease status or socio-economic status. The self-controlled tree-temporal scan method is described in greater detail in Yih et al. [18]. Although the method adjusts for the multiplicity of evaluations of diagnoses and risk intervals, we pre-specified the p-value cut-off for statistical significance as 0.01 rather than the more conventional 0.05 to further guard against false signals; no adjustment for the multiple testing entailed in conducting age group-specific analyses was implemented. In the tables of statistical results for all ages combined and for the specific age strata, we show all instances of clustering with p ≤ 0.05 for transparency, although we reserve the term “cluster” for those with p ≤ 0.01.

Only the strongest cluster (i.e., the one with the highest test statistic) for any diagnosis code or group of related diagnosis codes is identified.

We conducted the statistical analyses using TreeScan version 2.0 [25].

3.1. Signal investigation

We conducted signal investigation except for common vaccine-associated adverse events and other known or expected adverse events. To investigate smaller clusters, we generated and examined a list of diagnoses and procedures between 4 weeks before through 4 weeks after vaccination for each case in each cluster, using patient-level information stored at the sites. This approach permitted assessment of recent medical history without entailing more resource-intensive medical record review. It also allowed consideration of differences among sites regarding coding and other practices. For larger clusters (i.e., those with more cases), we instead generated aggregated lists and frequencies of diagnosis and procedure codes within +/-5 or 10 days of the outcome in question, to better characterize the cases. For clusters of outcomes suggestive of COVID-19, e.g., sepsis, we obtained counts of COVID-19 and pneumonia diagnoses recorded during follow-up among the cases in those clusters.

The study was approved by the Institutional Review Boards of all the VSD sites and was conducted in a manner consistent with federal law and CDC policy.

4. Results

Demographic characteristics and number of second dose recipients among the study population of first dose recipients are shown in Table 2 . For all three vaccines, the largest age group of vaccinees was the 40–64-year old group. However, there were differences in the age distribution across vaccines, with, for example, 36 %, 43 %, and 54 % of Pfizer-BioNTech, Moderna, and Janssen doses, respectively, received by the 40–64-year old group and 19 %, 29 %, and 13 % of Pfizer-BioNTech, Moderna, and Janssen doses, respectively, received by people aged 65 or older. More than half the doses of the mRNA vaccines were received by females, whereas the reverse was observed for Janssen vaccine. Ninety-seven percent of mRNA vaccinees received Dose 2 during the follow-up period.

Table 2.

Distribution of study participants by demographic characteristics, and number of second-dose recipients.

Number of vaccinees	Pfizer-BioNTech		Moderna		Janssen
	4,068,513		2,559,563		417,854
Age
5–11*	186,778	5 %	89	0.0 %	8	0.0 %
12–17*	529,234	13 %	994	0.0 %	204	0.0 %
18–39	1,141,410	28 %	713,525	28 %	137,391	33 %
40–64	1,450,828	36 %	1,104,729	43 %	227,441	54 %
65+	760,263	19 %	740,226	29 %	52,810	13 %
Ethnicity
Non-Hispanic	3,068,635	75 %	1,935,840	76 %	328,840	79 %
Hispanic	999,878	25 %	623,723	24 %	89,014	21 %
Race
Unknown	1,263,222	31 %	744,769	29 %	122,128	29 %
American Indian or Alaska Native	13,707	0.3 %	9,526	0.4 %	1,668	0.4 %
Asian	653,474	16 %	362,067	14 %	49,977	12 %
Black or African American	255,073	6 %	160,259	6 %	28,630	7 %
Native Hawaiian or other Pacific Islander	27,735	1 %	16,259	1 %	2,511	1 %
White	1,855,302	46 %	1,266,683	49 %	212,940	51 %
Sex
Female	2,192,338	54 %	1,395,720	55 %	194,405	47 %
Male	1,875,859	46 %	1,163,677	45 %	223,407	53 %
Other	316	0.0 %	166	0.0 %	42	0.0 %
Received second dose during follow-up	3,943,933	97 %	2,483,424	97 %	1,170	0.3 %

* Neither Moderna nor Janssen vaccine was authorized for use in people less than 18 years of age at the time of this study.

4.1. Pfizer-BioNTech

Among the 4,068,513 Pfizer-BioNTech vaccinees, there were 524,357 incident diagnoses during follow-up and 54 clusters detected in the all-ages-combined analysis. There were several clusters of unspecified adverse effects, all in Days 21–24 after Dose 1, presumably spanning the day of Dose 2 and the few days immediately thereafter (Table 3 ). Another outcome category with significant clustering comprised conditions we deemed related to COVID-19 disease, including codes specifically for COVID-19 (U07 and J12.82) and also viral pneumonia, sepsis, acute respiratory failure, and hypoxemia. All of these clusters were in periods within Days 3–21. In the post hoc category of local or systemic adverse effects were urticaria, myalgia, nausea and vomiting, fever, headache, malaise and fatigue, syncope, enlarged lymph nodes, chills, and unspecified allergy. Most of these clusters began on Days 21 or 22, presumably just after Dose 2. There were myocarditis/pericarditis clusters in periods within Days 23–26. Finally, there were clusters of other adverse events, namely, ‘pain in throat and chest’ in periods within Days 22–26, palpitations in Days 1–3, abnormalities of breathing in Days 21–26, and dyspnea and shortness of breath in Days 1–26.

Table 3.

Clusters of adverse events found after Pfizer-BioNTech primary series, all-ages analysis.*

ICD-10 code	Adverse Event	Total Number of Cases	Risk Window Start Day	Risk Window End Day	Number of Cases in Risk Window	Attributable Risk per 100,000 1st Doses	Log Likelihood Ratio Test Statistic	P-Value
Unspecified adverse effects
T50	Poisoning by, adverse effect of and underdosing of diuretics and other and unspecified drugs, medicaments and biological substances	2148	21	24	621	13.00	487.3	0.0001
T50.B95	Adverse effect of other viral vaccines	914	21	24	403	9.10	457.7	0.0001
T50.Z	Poisoning by, adverse effect of and underdosing of other vaccines and biological substances	309	21	24	152	3.50	187.1	0.0001
T50.Z9	Poisoning by, adverse effect of and underdosing of other vaccines and biological substances	303	21	24	152	3.50	189.7	0.0001
T50.Z95	Adverse effect of other vaccines and biological substances	302	21	24	152	3.50	190.2	0.0001
T88	Other complications of surgical and medical care, not elsewhere classified	549	21	24	223	5.00	237.5	0.0001
T88.1	Other complications following immunization, not elsewhere classified	412	21	24	208	4.80	260.8	0.0001
Conditions suggestive of COVID-19
U07	Emergency use of U07	2125	4	15	1017	20.00	427.4	0.0001
U07.1	COVID-19	2120	4	15	1016	20.00	427.9	0.0001
J12	Viral pneumonia, not elsewhere classified	1031	6	17	600	13.00	327.1	0.0001
J12.8	Other viral pneumonia	994	6	17	589	13.00	328.8	0.0001
J12.82	Pneumonia due to coronavirus disease 2019	942	6	17	559	12.00	312.6	0.0001
J12.89	Other viral pneumonia	48	3	16	34	0.76	19.9	0.0001
A41.8	Other specified sepsis	282	4	21	175	3.50	57.1	0.0001
A41.89	Other specified sepsis	242	7	16	113	2.30	58.1	0.0001
J96	Respiratory failure, not elsewhere classified	3185	7	16	628	5.40	35.6	0.0001
J96.0	Acute respiratory failure	2567	7	16	546	5.50	44.5	0.0001
J96.01	Acute respiratory failure with hypoxia	2023	6	16	498	5.50	49.4	0.0001
R09.0	Asphyxia and hypoxemia	1532	6	17	346	2.80	15.5	0.0028
R09.02	Hypoxemia	1528	6	17	345	2.70	15.5	0.0028
Local or systemic adverse effects known to be associated with some injected vaccines
L50	Urticaria	648	21	27	115	1.40	15.2	0.004
M79	Other and unspecified soft tissue disorders, not elsewhere classified	5435	22	23	275	2.70	26.9	0.0001
M79.1	Myalgia	980	22	23	128	2.50	85.3	0.0001
M79.10	Myalgia, unspecified site	771	22	23	115	2.30	88.5	0.0001
R11	Nausea and vomiting	7872	21	24	592	3.20	14.7	0.006
R50	Fever of other and unknown origin	2763	22	23	257	4.30	109.0	0.0001
R50.8	Other specified fever	329	22	23	51	1.00	40.8	0.0001
R50.83	Postvaccination fever	94	22	23	46	1.10	83.3	0.0001
R50.9	Fever, unspecified	2419	22	23	202	3.20	71.6	0.0001
R51	Headache	6004	1	29	2642	11.00	15.0	0.0047
R51.9	Headache, unspecified	5999	1	29	2638	11.00	14.8	0.0056
R53	Malaise and fatigue	4832	22	23	239	2.20	21.4	0.0001
R53.8	Other malaise and fatigue	2063	22	25	212	2.30	26.0	0.0001
R53.83	Other fatigue	1440	22	25	152	1.70	20.3	0.0001
R55	Syncope and collapse	4447	21	23	340	3.60	40.1	0.0001
R59	Enlarged lymph nodes	698	22	25	115	1.90	43.1	0.0001
R59.0	Localized enlarged lymph nodes	473	22	25	78	1.30	29.3	0.0001
R68.83	Chills (without fever)	429	21	24	64	1.00	20.2	0.0001
T78	Adverse effects, not elsewhere classified	1302	1	4	118	1.50	20.3	0.0001
T78.4	Other and unspecified allergy	807	19	27	183	2.20	24.7	0.0001
T78.40	Allergy, unspecified	753	19	27	164	1.90	19.3	0.0001
Other adverse events
R07	Pain in throat and chest	14,543	22	24	975	7.90	61.5	0.0001
R07.9	Chest pain, unspecified	7978	22	26	800	5.60	34.2	0.0001
R07.8	Other chest pain	5884	22	24	403	3.40	28.2	0.0001
R07.89	Other chest pain	5340	22	24	383	3.50	32.9	0.0001
R00.2	Palpitations	2506	1	3	152	1.60	18.5	0.0002
R06	Abnormalities of breathing	6908	21	26	746	3.80	15.5	0.0028
R06.0	Dyspnea	6132	1	26	2432	10.00	15.7	0.0027
R06.02	Shortness of breath	4689	1	26	1899	9.20	16.9	0.0004
Myo/pericarditis
I51	Complications and ill-defined descriptions of heart disease	1331	24	25	84	1.20	20.6	0.0001
I51.4	Myocarditis, unspecified	75	23	26	43	1.00	60.7	0.0001
I40	Acute myocarditis	47	24	25	23	0.55	43.7	0.0001
I40.9	Acute myocarditis, unspecified	28	24	26	18	0.43	32.1	0.0001
I31.9	Disease of pericardium, unspecified	95	24	25	16	0.34	15.1	0.0042

* For ease of review, we grouped clusters post hoc into five categories: unspecified adverse effects; local or systemic adverse effects known to be associated with injected vaccines; myo/pericarditis; other adverse events; and COVID-19, viral pneumonia, and other conditions suggestive of COVID-19. Within the table, these categories are separated by blank rows and appear in descending order of the largest test statistic in the category. Within each category, codes sharing the same first three characters are nested together. Except for one category, these nests of codes are arranged in descending order of the largest test statistic in the nest. The exception is the category of local or systemic adverse effects, a relatively large category, where we chose to organize the rows in order of their alphanumeric code, to facilitate comparison among the tables. Clusters with 0.01 < p ≤ 0.05, if any were found, appear in italics.

In age-stratified analyses (Table S1) for the 186,778 Pfizer-BioNTech vaccinees aged 5–11 years, the only cluster with p < 0.05 was for COVID-19 (U07.1) in Days 44–69, late in the follow-up period (p = 0.0001). This cluster of 87 cases constituted 0.05 % of the vaccinees in this age group. For all the other age groups, the health outcomes with clusters were a subset of those with clusters in the all-ages analysis. In the 12–17 and 18–39-year age groups, myocarditis (I51.4 and I40) clusters within Days 23–26 and clusters of chest pain in similar timeframes were detected. In the 40–64- and ≥65-year age groups, the strongest clusters were of COVID-19 and other viral pneumonia within 3 weeks of vaccination, and there were no clusters of myocarditis/pericarditis or chest pain.

4.2. Moderna

In the all-ages-combined analysis of the 2,559,563 Moderna vaccinees, there were 408,749 incident diagnoses during follow-up and 43 clusters. Clusters of unspecified adverse effects emerged, all within Days 28–32 after Dose 1, likely on the day of Dose 2 and the few days following it (Table 4 ). There were clusters of COVID-19, viral pneumonia, sepsis, and acute respiratory failure, all within Days 1–16. In the post hoc category of local or systemic adverse effects were clusters of myalgia, nausea and vomiting, fever, headache, pain, malaise and fatigue, syncope, chills, and unspecified allergy, almost all of which started on Day 28 or 29, likely after Dose 2. There were no clusters of myocarditis/pericarditis, although there was a non-statistically significant grouping of acute myocarditis (I40) in Days 31–32 and one of acute pericarditis (I30) also in Days 31–32 (Table S2). There were statistically significant clusters of ‘pain in throat and chest’ during Days 1–5, tachycardia during Days 29–30, and palpitations during Days 1–4.

Table 4.

Clusters of adverse events found after Moderna primary series, all-ages analysis.*.

ICD-10 code	Adverse Event	Total Number of Cases	Risk Window Start Day	Risk Window End Day	Number of Cases in Risk Window	Attributable Risk per 100,000 1st Doses	Log Likelihood Ratio Test Statistic	P-Value
Unspecified adverse effects
T50	Poisoning by, adverse effect of and underdosing of diuretics and other and unspecified drugs, medicaments and biological substances	1953	28	31	597	20.00	460.0	0.0001
T50.B9	Poisoning by, adverse effect of and underdosing of other viral vaccines	931	28	32	438	15.00	419.9	0.0001
T50.B95	Adverse effect of other viral vaccines	930	28	32	438	15.00	420.3	0.0001
T50.Z	Poisoning by, adverse effect of and underdosing of other vaccines and biological substances	323	28	30	138	5.00	176.0	0.0001
T50.Z9	Adverse effect of other vaccines and biological substances	319	28	30	138	5.00	177.5	0.0001
T88	Other complications of surgical and medical care, not elsewhere classified	574	28	31	234	8.20	235.3	0.0001
T88.1	Other complications following immunization, not elsewhere classified	457	28	30	193	7.00	244.1	0.0001
T80.6	Other serum reactions	33	28	30	14	0.51	17.8	0.0001
T80.62	Other serum reaction due to vaccination	25	28	30	14	0.52	21.3	0.0001
Conditions suggestive of COVID-19
U07	Emergency use of U07	1038	1	16	613	20.00	239.5	0.0001
U07.1	COVID-19	1031	1	14	563	18.00	240.9	0.0001
J12	Viral pneumonia, not elsewhere classified	555	6	16	309	10.00	180.0	0.0001
J12.8	Other viral pneumonia	530	6	15	289	9.80	183.0	0.0001
J12.82	Pneumonia due to coronavirus disease 2019	489	4	16	312	11.00	178.4	0.0001
A41.8	Other specified sepsis	235	8	14	70	2.00	31.5	0.0001
A41.89	Other specified sepsis	184	8	14	66	2.10	38.3	0.0001
J96	Respiratory failure, not elsewhere classified	2610	8	16	438	5.20	19.0	0.0001
J96.0	Acute respiratory failure	2009	8	13	251	3.60	19.1	0.0001
J96.01	Acute respiratory failure with hypoxia	1596	5	15	341	4.70	19.8	0.0001
Local or systemic adverse effects known to be associated with some injected vaccines
M79	Other and unspecified soft tissue disorders, not elsewhere classified	4073	29	30	227	3.50	22.4	0.0001
M79.1	Myalgia	803	29	30	119	3.70	82.2	0.0001
M79.10	Myalgia, unspecified site	638	29	30	110	3.60	89.0	0.0001
R11	Nausea and vomiting	5374	29	31	380	4.70	21.6	0.0001
R50	Fever of other and unknown origin	2136	29	30	261	7.60	142.8	0.0001
R50.8	Other specified fever	323	1	2	48	1.60	46.7	0.0001
R50.83	Postvaccination fever	124	1	2	40	1.50	66.4	0.0001
R50.9	Fever, unspecified	1793	29	30	211	6.00	109.8	0.0001
R51	Headache	4383	29	31	327	4.50	24.2	0.0001
R51.9	Headache, unspecified	4378	29	31	327	4.50	24.4	0.0001
R52	Pain, unspecified	355	29	30	49	1.50	31.2	0.0001
R53	Malaise and fatigue	3994	29	30	290	6.20	63.6	0.0001
R53.1	Weakness	2382	29	30	162	3.20	30.1	0.0001
R53.8	Other malaise and fatigue	1604	29	30	127	2.90	33.8	0.0001
R53.81	Other malaise	457	29	30	48	1.30	21.2	0.0001
R53.83	Other fatigue	1132	28	33	174	2.90	18.0	0.0001
R55	Syncope and collapse	3285	28	30	332	7.00	68.3	0.0001
R68.83	Chills (without fever)	323	29	30	41	1.20	23.6	0.0001
T78	Adverse effects, not elsewhere classified	861	28	32	128	2.60	21.4	0.0001
T78.4	Other and unspecified allergy	551	28	29	52	1.30	19.2	0.0001
T78.40	Allergy, unspecified	507	28	29	45	1.10	14.9	0.0042
Other adverse events
R07	Pain in throat and chest	10,808	1	5	828	7.50	22.8	0.0001
R07.8	Other chest pain	4879	1	5	383	3.70	12.7	0.0453
R00.0	Tachycardia, unspecified	1942	29	30	119	2.10	16.4	0.0005
R00.2	Palpitations	1800	1	4	142	2.30	14.8	0.0043

* See footnote under Table 3.

In the 18–39-year age group, the health outcomes with clusters were largely a subset of those with clusters in the all-ages analysis. In addition, there were clusters of abnormalities of heartbeat in Days 1–6, dyspnea/shortness of breath in Days 28–32, and unspecified chest pain in Days 29–32 (Table S3). There were non-statistically significant groupings of acute pericarditis (I30 and I30.9) in Days 31–32 and Days 26–37, respectively (Table S2).

For the 40–64- and ≥65-year age groups, the health outcomes with statistically significant clusters were a complete subset of those seen in the all-ages analysis (Table S3). COVID-19, viral pneumonia, respiratory failure, and sepsis clusters (the latter two only in the 40–64-year group) were all within Days 1–16. Other than COVID-19 and viral pneumonia, there were no clusters of any respiratory or cardiac outcomes or symptoms (e.g., chest pain, abnormalities of heartbeat, or shortness of breath) nor any of myocarditis/pericarditis.

4.3. Janssen

The 417,854 Janssen vaccinees had 57,874 incident diagnoses during follow-up and 21 clusters. There were clusters of unspecified adverse effects in Days 1–5 and clusters of local or systemic adverse effects, specifically myalgia and fever on Days 1–2 and headache on Days 1–13 (Table 5 ). In the “other” category, there were clusters of abnormalities of gait and mobility, including difficulty walking, and of other disorders of muscle, including generalized muscle weakness, all on Days 1–2. In addition, there was a cluster of dyspnea on Days 3–14 (p = 0.01). A non-statistically significant grouping of COVID-19 appeared in Days 7–15 (p = 0.0453).

Table 5.

Clusters of adverse events found after Janssen primary series, all-ages analysis.*

ICD-10 code	Adverse Event	Total Number of Cases	Risk Window Start Day	Risk Window End Day	Number of Cases in Risk Window	Attributable Risk per 100,000 1st Doses	Log Likelihood Ratio Test Statistic	P-Value
Unspecified adverse effects
T50	Poisoning by, adverse effect of and underdosing of diuretics and other and unspecified drugs, medicaments and biological substances	307	1	4	153	33.00	146.5	0.0001
T50.B95	Adverse effect of other viral vaccines	148	1	5	106	24.00	116.2	0.0001
T50.Z	Poisoning by, adverse effect of and underdosing of other vaccines and biological substances	64	1	4	48	11.00	62.9	0.0001
T50.Z9	Poisoning by, adverse effect of and underdosing of other vaccines and biological substances	62	1	4	48	11.00	64.2	0.0001
T50.Z95	Adverse effect of other vaccines and biological substances	61	1	4	48	11.00	64.9	0.0001
T88	Other complications of surgical and medical care, not elsewhere classified	82	1	4	55	13.00	66.6	0.0001
T88.1	Other complications following immunization, not elsewhere classified	64	1	4	51	12.00	69.6	0.0001
Local or systemic adverse effects known to be associated with some injected vaccines
M79.1	Myalgia	168	1	2	37	7.40	30.3	0.0001
M79.10	Myalgia, unspecified site	140	1	2	35	7.20	32.3	0.0001
R50	Fever of other and unknown origin	294	1	2	52	9.80	33.7	0.0001
R50.8	Other specified fever	36	1	2	12	2.60	14.0	0.0021
R50.83	Postvaccination fever	14	1	2	11	2.60	21.4	0.0001
R50.9	Fever, unspecified	255	1	2	38	6.70	19.9	0.0001
R51	Headache	903	1	13	359	41.00	29.4	0.0001
R51.9	Headache, unspecified	902	1	13	359	41.00	29.6	0.0001
Other adverse events
R26	Abnormalities of gait and mobility	197	1	2	42	8.40	33.3	0.0001
R26.8	Other abnormalities of gait and mobility	101	1	2	21	4.20	16.2	0.0002
R26.89	Other abnormalities of gait and mobility	69	1	2	15	3.00	12.1	0.0168
R26.2	Difficulty in walking, not elsewhere classified	66	1	2	17	3.50	16.1	0.0004
M62	Other disorders of muscle	246	1	2	32	5.30	13.7	0.0024
M62.8	Other specified disorders of muscle	226	1	2	29	4.80	12.2	0.0151
M62.81	Muscle weakness (generalized)	113	1	2	28	5.80	25.6	0.0001
R06.0	Dyspnea	779	3	14	254	22.00	12.6	0.01
Conditions suggestive of COVID-19
U07	Emergency use of U07	334	7	15	100	12.00	11.6	0.0315
U07.1	COVID-19	332	7	15	99	12.00	11.3	0.0453

* See footnote under Table 3.

In analyses of the specific age groups, the health outcomes with clusters were subsets of those with clusters in the all-ages analysis. Clusters of abnormalities of gait and mobility and disorders of muscle, all within 4 days after vaccination, were seen only in the ≥65-year age group (Table S4).

Among Janssen vaccinees of all ages, there were 58 cases of “abnormalities of gait and mobility” (R26 with or without M62) during Days 1–4 and 16 unique cases of “other disorders of muscle” (M62 without R26) on Days 1–2. One VSD site had disproportionately many cases of both R26 and M62 relative to its number of Janssen vaccinees (40 %): 43 (74 %) of the 58 cases of R26 and 13 (81 %) of the 16 cases of M62 without R26. All 43 of this site’s R26 cases in Days 1–4 had been hospitalized days earlier and received Janssen vaccine in the hospital, within a few days prior to discharge. All but one (98 %) had codes indicating they had been discharged to a nursing facility. The R26 code was, in all cases, one of many diagnosis codes appearing on the day of hospital discharge and nursing facility admission (range of number of diagnosis codes per patient: 7–33; mean and median: 15). Regarding the 13 cases of M26 without R26 on Days 1–2 at this site, 11 (85 %) had been hospitalized days earlier and received Janssen in the few days before discharge. Eight (73 %) of the 11 had codes indicating discharge to a nursing facility. For all 11 hospitalized patients, the M62 code was on the day of hospital discharge/nursing facility admission. Furthermore, for both the R26 and the M62 hospitalized cases, the primary diagnoses (prior to vaccination) were often conditions that could have produced gait/mobility problems or muscle weakness, e.g., broken fibula, tibia or femur; strokes; traumatic head injuries; cellulitis of lower limbs or diabetic ulcers on the foot; etc.

5. Discussion

We retrospectively applied a tree-based scan statistical approach to longitudinal data from approximately 4.1 million Pfizer-BioNTech vaccinees, 2.6 million Moderna vaccinees, and 0.4 million Janssen vaccinees in the VSD to look broadly for clusters of cases of any of thousands of potential adverse events during the several weeks after vaccination. We did not identify any previously unknown serious adverse events for either Pfizer-BioNTech or Moderna mRNA vaccines. For Janssen vaccine, we detected unexpected statistical signals for abnormalities of gait and mobility and for other disorders of muscle, both of which we investigated and determined not to be safety signals, as discussed below. In addition, the analyses provided some evidence of vaccine effectiveness with respect to COVID-19 and related conditions attended to in inpatient or emergency department settings.

All three vaccines were associated with codes for unspecified adverse effects, the strongest clusters of which were seen in the days immediately following the second dose of the mRNA vaccines and the one dose of the Janssen vaccine. These clusters likely reflect known, non-serious vaccine-associated adverse events, given findings of an earlier application of these methods [15]. In that earlier study, codes received in conjunction with non-specific adverse effects codes were for pain in and/or swelling of the limb, local skin reactions and/or unspecific allergic reactions, cellulitis, nausea and/or vomiting, fever, viral exanthem, dizziness and giddiness, headache, and/or unspecified myalgia and myositis, with few or no subsequent medical visits recorded [15]. In view of this previous experience, we did not investigate such clusters further in the current study.

In addition to unspecified adverse effects, there were clusters of local or systemic adverse effects for all three vaccines. Consistent with greater reactogenicity of Dose 2 of both mRNA vaccines, most of these clusters started on Day 21 or 22 after Pfizer-BioNTech and on Day 28 or 29 after Moderna. Clusters of myalgia, fever, and headache were also seen in the days immediately after vaccination among the Janssen vaccinees.

Myocarditis/pericarditis is known to be rarely associated with mRNA vaccination, observed primarily in adolescent and young adult males and particularly in the week after Dose 2 [8]. In our study, clusters of myocarditis/pericarditis were detected within Days 23–26 after Pfizer-BioNTech Dose 1 vaccination, i.e., very likely in the week after Dose 2 in most or all instances. Myocarditis/pericarditis clusters were found in the inclusive all-ages analysis as well as for the 12–17-year and 18–39- year age groups, but not for the older age groups. No clusters of myocarditis/pericarditis were seen among the Moderna vaccinees, likely due in part to sample size. However, non-statistically significant groupings of myocarditis/pericarditis were found in Days 31–32 in the all-ages and 18–39-year groups.

Other serious COVID-19 vaccine-associated adverse events that have been documented in the literature are the even rarer thrombosis with thrombocytopenia after Janssen vaccination, with an attributable risk of approximately 7 cases per million vaccinations for the highest-risk group, women aged 18–49 years [10], and Guillain-Barré syndrome after Janssen vaccination, with an attributable risk of approximately 15.5 cases per million vaccinations (using a risk window of Days 1–21) compared with mRNA vaccines [11]. The rareness of these adverse events together with the relatively small size of our Janssen-vaccinated group likely explain the lack of signals for both of these events in our analyses.

The numbers of cases in the clusters of chest pain, palpitations, tachycardia (for Moderna), and abnormalities of breathing (for Pfizer-BioNTech) were an order of magnitude greater than the numbers of myocarditis/pericarditis cases, and there was no evidence that these clusters were being driven by myocarditis/pericarditis. The conditions frequently coded within +/−5 days of chest pain in Pfizer-BioNTech vaccinees (appearing in at least 5 % of the chest pain cases) were shortness of breath, unspecified anxiety disorder, and headache. For Moderna, these conditions coded within +/−5 days of chest pain were shortness of breath, palpitations, fever, tachycardia, myalgia, and unspecified anxiety disorder. Similarly, conditions frequently coded within +/-5 days of dyspnea in Pfizer-BioNTech vaccinees included chest pain, unspecified anxiety disorder, headache, tachycardia, palpitations, and myalgia. The conditions featured in these statistical signals as well as the conditions reflected by the proximate codes are for the most part signs and symptoms rather than specific diagnoses, and the etiology of the symptoms remains unknown.

Clusters of abnormalities of gait and mobility (R26) and disorders of muscle (M62), including generalized muscle weakness, were seen in the Janssen analysis of all ages combined but were driven by the elderly, as such clusters were present only for the ≥65-year age group in age-stratified analysis (Table S4). Both the R26 and M62 clusters within a few days of vaccination appear to have resulted from one large site’s pattern of vaccinating frail hospitalized patients (many of whom were hospitalized for conditions plausibly affecting mobility) with Janssen vaccine within a few days before discharge—perhaps particularly patients heading to a nursing facility—and recording gait/mobility abnormalities and/or muscle weakness, typically along with several other diagnoses, upon transfer to a nursing facility. In synthesis, we believe these two signals are explained by (a) temporal confounding, in which hospital discharge is temporally related to both exposure and outcome, and (b) the fact that many of these patients were evidently hospitalized (prior to vaccination) for a condition that was plausibly associated with gait/mobility problems or muscle weakness.

Clusters of codes consistent with COVID-19—COVID-19 codes U07 and J12.82, viral pneumonia, sepsis, acute respiratory failure, and hypoxemia—appeared in similar time frames for both Pfizer-BioNTech and Moderna vaccinees: within Days 3–21 for Pfizer-BioNTech and Days 1–16 for Moderna. More than 80 % of the cases in the sepsis clusters and roughly half of the cases in the acute respiratory failure clusters also had diagnosis codes for COVID-19 during follow-up, suggesting that these clusters were largely manifestations of COVID-19, particularly considering that these outcomes are known complications of COVID-19 [26], [27]. There was also a non-statistically significant group of the code U07 in Days 7–15 for Janssen (p = 0.04). There were no sepsis clusters in age groups younger than 40 years of age for any of the three vaccines. We interpret these clusters as evidence that vaccination suppressed severe COVID-19 starting in the third week after Dose 1 (i.e., in the control windows of these clusters). The one exception to this temporal pattern was the observation among the 5–11-year-old Pfizer-BioNTech vaccinees of a COVID-19 cluster on Days 44–69 after vaccination, late in follow-up. This is likely due to break-through COVID-19 (in just 0.05 % of vaccinees), considering the timing of FDA’s authorization of this vaccine for use in this age group on October 29, 2021, long after authorization for older age groups, and the rapid rise in COVID-19 cases nationwide during December 2021–January 2022.

Limitations of this study include the possibility of temporal confounding. The clusters of gait/mobility abnormalities and muscle disorders after Janssen and of COVID-19 break-through disease late in follow-up in 5–11-year-old Pfizer-BioNTech vaccinees are examples of such confounding. Second, scanning tens of thousands of diagnoses and groups of diagnoses and hundreds of time intervals uses statistical power, meaning that rare health outcomes producing a signal in a more targeted study might not signal in a study like ours, even with a similar number of vaccinees. For example, the groups of myocarditis/pericarditis after Moderna did not rise to statistical significance, even in the analysis of the 18–39-year age group. Third, anchoring follow-up on Dose 1 and not doing dose-specific analyses means that true clusters of a given outcome occurring after both doses could be detected after at most one of the doses, the one with the strongest cluster. Fourth, the requirement that study subjects be enrolled for the full follow-up period means that cases of death were excluded, so we may have missed the most severe cases of any type of outcome. However, a study conducted in the VSD found no increased risk of death among COVID-19 vaccine recipients [28]. Finally, if true adverse reactions do not show strong clustering in time (e.g., because of insidious onset) or in the diagnosis tree (e.g., because they manifest across multiple body systems and might be coded differently, depending on the case and/or the clinician), they might not be detected with this tree-temporal method. We believe this would be a concern for only a small minority of potential adverse reactions, however.

Notwithstanding the limitations, this self-controlled tree-temporal scan statistical method provides unique advantages for vaccine safety monitoring by virtue of its population-based and untargeted nature—neither specific health outcomes nor specific post-exposure periods of potentially increased risk are pre-specified. Adjustment for the evaluation of the tens of thousands of diagnoses and groups of diagnoses and the hundreds of risk intervals keeps the number of false signals low. Further, the method is self-controlled, such that any potential confounding from time-invariant characteristics, including chronic disease status, is essentially eliminated. It nicely complements VSD’s sequential analysis (also known as “rapid cycle analysis”), which has greater power to detect increased risks of its prespecified health outcomes in prespecified risk windows but is not as broad an assessment of safety, given the need to prespecify outcomes and risk windows.

In summary, in this inclusive data-mining evaluation of 4.1, 2.6, and 0.4 million Pfizer-BioNTech, Moderna, and Janssen vaccinees, respectively, completely unanticipated adverse events within several weeks of vaccination could have come to light. However, no unexpected serious adverse events were identified, only known adverse events. Clusters of myocarditis in the week after Pfizer-BioNTech Dose 2 were detected in vaccinees of ages 12–17- and 18–39-years. While not ruling out the possibility of rare vaccine-associated adverse events, these results offer additional reassurance about the safety of COVID-19 vaccines, in view of the broad, untargeted nature of the study.

Disclaimer

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

Declaration of Competing Interest

WK Yih has received research funding from Pfizer in the past.  JC Nelson received grant funding to participate on the External Safety Advisory Board for Moderna’s COVID-19 vaccine program in 2020 – April 2021.  L Qian has received funding from Moderna, GlaxoSmithKline, and Dynavax for work unrelated to this manuscript.  The other authors report no conflicts.

Appendix A. Supplementary material

The following are the Supplementary data to this article:

Data availability

Researchers may request the data via the standard VSD data sharing program described on CDC’s public website: https://www.cdc.gov/vaccinesafety/ensuringsafety/monitoring/vsd/accessing-data.html.