Near Real-Time Surveillance for Influenza Vaccine Safety: Proof-of-Concept in the Vaccine Safety Datalink Project

Abstract

The emergence of pandemic H1N1 influenza in 2009 has prompted public health responses, including production and licensure of new influenza A (H1N1) 2009 monovalent vaccines. Safety monitoring is a critical component of vaccination programs. As proof-of-concept, the authors mimicked near real-time prospective surveillance for prespecified neurologic and allergic adverse events among enrollees in 8 medical care organizations (the Vaccine Safety Datalink Project) who received seasonal trivalent inactivated influenza vaccine during the 2005/06–2007/08 influenza seasons. In self-controlled case series analysis, the risk of adverse events in a prespecified exposure period following vaccination was compared with the risk in 1 control period for the same individual either before or after vaccination. In difference-in-difference analysis, the relative risk in exposed versus control periods each season was compared with the relative risk in previous seasons since 2000/01. The authors used Poisson-based analysis to compare the risk of Guillian-Barré syndrome following vaccination in each season with that in previous seasons. Maximized sequential probability ratio tests were used to adjust for repeated analyses on weekly data. With administration of 1,195,552 doses to children under age 18 years and 4,773,956 doses to adults, no elevated risk of adverse events was identified. Near real-time surveillance for selected adverse events can be implemented prospectively to rapidly assess seasonal and pandemic influenza vaccine safety.

In April 2009, the first cases of pandemic H1N1 influenza were confirmed in several countries (1). The World Health Organization declared a pandemic in June 2009 (2), and influenza A (H1N1) 2009 monovalent vaccines were licensed in September 2009 (3). The Advisory Committee on Immunization Practices recommended initial target groups for these vaccines (4), recommended annual seasonal influenza vaccination for approximately 83% of the US population (5) (255 million children and adults), and may continue to incrementally expand the recommendations (6). Given rapid deployment of 2009 H1N1 vaccines and broad recommendations for seasonal influenza vaccines, monitoring vaccine safety in near real-time is critical (7, 8).

Current systems for evaluating vaccine safety have limitations. First, passive reporting systems, such as the Vaccine Adverse Event Reporting System, allow identification of potential adverse events but tend to underreport numbers of adverse events, lack denominator data, and are not designed to distinguish coincidental associations from potentially true causal relations (9). Second, traditional postlicensure studies can rigorously evaluate potential associations between vaccines and adverse events, but they do not monitor safety in near real-time, since they often require substantial time to design and implement, resulting in delays in the ability to detect possible adverse events. If analyses are conducted after the influenza season is over, potential problems will be identified only after all doses from that season's vaccine have already been administered.

In several recent studies, investigators have addressed this limitation of timeliness by comparing adverse event risks after administration of a vaccine of interest with risks following administration of a comparison vaccine or a preventive care visit in near real-time, using weekly data and a Poisson-based maximized sequential probability ratio test to adjust for repeated analyses (10–13). However, selecting a comparison group can be particularly problematic for influenza vaccine, because of confounding by indication (14); influenza vaccine recipients are often systematically different from nonvaccinees (e.g., by functional status limitations, such as requiring assistance to bathe) in ways that may influence the incidence of adverse events and for which adjustment may not be possible using solely automated data (15). In addition, vaccination recommendations have expanded over time (5), resulting in changing characteristics of vaccinated and unvaccinated populations. Furthermore, vaccinees may be misclassified as unvaccinated in electronic data, particularly if vaccines are administered in nontraditional settings (e.g., workplaces) (16). Self-controlled case series (SCCS) (17, 18) and difference-in-difference (DID) (19) approaches compare risk and control periods within vaccinees, minimizing these potential biases and controlling for known confounders (e.g., age and sex) as well as unmeasured fixed covariates (e.g., race, ethnicity, socioeconomic status, and underlying disease).

In previous postlicensure studies, investigators have compared risk and control periods within persons receiving influenza vaccine (20–22) or have conducted sequential analyses to monitor adverse events following administration of vaccines other than influenza vaccine (10–13). In this report, we unite these methods by applying maximized sequential probability ratio testing to comparisons of risk and control periods within vaccinees. As proof-of-concept in preparation for pandemic and seasonal influenza vaccine safety monitoring, our objective was to mimic prospective surveillance for prespecified adverse events following seasonal influenza vaccine administration, using binomial-based (SCCS and DID) and Poisson-based analytic approaches.

MATERIALS AND METHODS

Study design

We used 3 approaches to conduct safety surveillance for the first seasonal dose of trivalent inactivated influenza vaccine (TIV). The SCCS approach (17, 18) was used to investigate whether a predefined period following TIV administration in a particular season was riskier than a control period. The SCCS approach may be limited by the “healthy vaccinee” effect (23), since some patients may delay or forgo routine influenza vaccination shortly after experiencing an illness classifiable as an adverse event. To minimize this effect, the control period for each adverse event category excluded the 14 days prior to vaccination. Unlike another case-only method, the case-crossover design (24), SCCS does not require the assumption that the underlying probability of vaccination is the same in all intervals (25), which is unlikely to hold for seasonally administered vaccines such as influenza.

The DID approach was used to investigate whether TIV in a particular season was riskier than TIV in cumulative previous seasons, using control periods for both current and previous seasons. It is an extension of the SCCS approach, first applied in economics research (19). The DID approach adjusts for the healthy vaccinee effect, assuming that an adverse event's effect on subsequent vaccination is the same in the current season as in previous seasons. This assumption may not always be valid—for example, if a medical care organization changes its policy for vaccinating after the occurrence of a particular adverse event between seasons. In addition, if the DID approach compares the relative risk of adverse events following pandemic influenza vaccine with the relative risk following TIV, the appropriateness of the comparison may be limited if the populations receiving pandemic vaccine and TIV are substantially different (e.g., according to underlying risk conditions). This approach differs from simply comparing risk periods in the current season with those in previous seasons without control periods; this would adjust for the healthy vaccinee effect but could be biased by secular trends in risk unrelated to vaccination (26).

Finally, a Poisson-based approach was used to compare the risk of Guillian-Barré syndrome following TIV in the current season with that in previous seasons. This analytic approach has been previously described (27) and (unlike SCCS and DID) previously applied for near real-time surveillance within the Vaccine Safety Datalink (VSD) Project (10–13).

We classified instances of adverse events as occurring within a risk period following TIV, when association with vaccination was plausible, or occurring within a control period before vaccination (“control period 1”) or after the risk period following vaccination (“control period 2”), when the adverse event was more plausibly attributable to causes other than vaccination. Only vaccinees with an adverse event were included in analysis, and only if an adverse event occurred in either the risk period or the control period but not both.

Study population

Established in 1990 to monitor and evaluate vaccine safety, the VSD Project collects medical care and vaccination data on over 9 million members annually enrolled in 8 participating medical care organizations (28, 29). Staff at each VSD site prepare data files containing demographic and medical information on the members, such as age, sex, enrollment, vaccinations, and International Classification of Diseases, Ninth Revision (ICD-9) diagnosis codes assigned to medical encounters occurring in outpatient, emergency department, and hospital settings.

Children aged 6 months–17 years and adults aged ≥18 years were eligible if they 1) received TIV during the weeks of September 1 through April 30 in the 2005/06, 2006/07, or 2007/08 influenza seasons, 2) experienced a predefined adverse event during either the risk period or the control period in any season, and 3) were enrolled at the time of vaccination in a medical care organization participating in the VSD Project. Uniformly formatted files with immunization and diagnosis data are updated weekly and include information on VSD site, age, sex, week of TIV administration, adverse event type according to ICD-9 codes, number of days between TIV administration and the adverse event, and medical setting in which the adverse event was diagnosed.

The 8 medical care organizations within the VSD Project are: Group Health Cooperative, Seattle, Washington; Harvard Pilgrim Health Care, Boston, Massachusetts; HealthPartners Research Foundation, Minneapolis, Minnesota; Kaiser Permanente of Colorado, Denver, Colorado; Kaiser Permanente of Northern California, Oakland, California; Kaiser Permanente of Southern California, Pasadena, California; Marshfield Clinic Research Foundation; Marshfield, Wisconsin; and Northwest Kaiser Permanente, Portland, Oregon. The Centers for Disease Control and Prevention and the institutional review board at each site provided approval and agreed that informed consent was not required.

Adverse events

The adverse events evaluated included seizures (30), meningoencephalitis (31–33), Bell's palsy (34, 35), other cranial nerve disorders, demyelinating disease (32, 36, 37), disorders of the peripheral nervous system and neuropathy (32, 38), ataxia (39), anaphylaxis (40, 41), allergic reaction other than anaphylaxis (including angioneurotic edema and urticaria) (42), and Guillian-Barré syndrome (43–45) (Table 1). These neurologic (46) and allergic (47) adverse events were selected because they are clinically well-defined, are serious, and either had already been observed in published studies or passive surveillance or were biologically plausible as a consequence of TIV vaccination (e.g., anaphylaxis via immunoglobulin E-mediated hypersensitivity) (48).

Table 1.

Definitions of Adverse Events Used for Influenza Vaccine Safety Surveillance, Vaccine Safety Datalink Project, 2005/06–2007/08

Adverse Event Category	ICD-9 Inclusion Code(s)a	ICD-9 Exclusion Codes (if Event Occurred ≤7 Days After Diagnosis)	Medical Setting	Risk Period Postvaccination, days	Control Period Pre- or Postvaccination, days	First Such Event in What Period and Medical Setting?
Seizure	345.0*–345.9*, 780.3, 780.31, 780.39	None	Inpatient, ED	0–7	8–15 days postvaccination, same season	8 days in outpatient, inpatient, ED
Meningoencephalitis	047.8, 047.9, 049.9, 321.2, 322*, 323.5–323.9	047.0–047.1, 048, 049.0–049.8, 053*–056*, 320*	Inpatient, ED	1–21	15–74 days prevaccination, same season	1 year in inpatient, ED
Bell's palsy	351.0	None	Outpatient, inpatient, ED	1–42	15–74 days prevaccination, same season	1 year in outpatient, inpatient, ED
Other cranial nerve disorder	350*, 351.1, 351.8, 351.9, 352*	None	Outpatient, inpatient, ED	1–42	15–74 days prevaccination, same season	1 year in outpatient, inpatient, ED
Demyelinating disease	340*, 341.0, 341.8, 341.9, 377.30, 377.31, 377.32, 377.34, 377.39	None	Outpatient, inpatient, ED	1–42	15–74 days prevaccination, same season	1 year in outpatient, inpatient, ED
Disorder of the peripheral nervous system	337.0, 337.9, 354.1–354.9, 355*, 356.4, 356.8, 357.6, 357.7, 357.8*, 357.9	None	Outpatient, inpatient, ED	1–42	15–74 days prevaccination, same season	1 year in outpatient, inpatient, ED
Ataxia	334.3	None	Outpatient, inpatient, ED	1–42	15–74 days prevaccination, same season	1 year in outpatient, inpatient, ED
Anaphylaxis	995.0, 999.4	None	Inpatient, ED	0–2	7–9 days postvaccination, same season	6 months in inpatient, ED
Allergic reaction other than anaphylaxis	995.1, 995.3, 708.0, 708.1, 708.9	None	Outpatient, inpatient, ED	1–2	8–9 days postvaccination, same season	6 months in outpatient, inpatient, ED
Guillian-Barré syndrome	357.0	None	Outpatient, inpatient, ED	1–42	1–42 days postvaccination, previous seasons	1 year in outpatient, inpatient, ED

Abbreviations: ED, emergency department; ICD-9, International Classification of Diseases, Ninth Revision.

^aAsterisks in ICD-9 codes indicate that any digits after the asterisk were included.

The length of the risk period (Table 1) was adverse event-specific and depended on the timing of adverse events relative to receipt of vaccine in published studies (30, 34, 49) and biologic plausibility. The day of vaccination was included in the risk period only for those adverse events for which a same-day diagnosis was biologically plausible (e.g., anaphylaxis) and, for improved specificity, only for adverse events diagnosed in inpatient or emergency department settings. Certain conditions (e.g., urticaria) that are listed as diagnoses in the outpatient setting are often chronic diagnoses present before vaccination (50).

For most adverse event categories, the control period was selected to occur before the risk period to facilitate timely analyses. However, control periods were selected to occur after the postvaccination risk period (control period 2) for 3 adverse event categories, because 1) a recent prior adverse event might preclude vaccination (i.e., allergic reaction and anaphylaxis) or 2) a person might have an underlying condition that is also an indication for vaccination (i.e., seizure disorder).

For most adverse event categories, an adverse event was included in analysis only if it was the first event of its type to occur in a year, irrespective of vaccination timing (Table 1). This restriction ensured that 1) the specificity of adverse event definitions was enhanced, since follow-up visits for preexisting conditions were not interpreted as new-onset events, and 2) multiple events of the same type could not be counted during a single observation period. For adverse events that could plausibly have more than 1 new-onset episode in a year (e.g., seizure), the event was included in analysis if it was the first event in a shorter time period.

Statistical analysis

In the SCCS approach, the null hypothesis assumes that the risk of an adverse event's occurrence on a day during the exposed risk period following the first TIV dose in a season is the same as the risk of its occurrence on a day during the unexposed control period. For the current season under surveillance, c represents the count of adverse events in the control period, d represents the count of adverse events in the postvaccination risk period (Figure 1, part A), and r represents the ratio of c to d expected under the null hypothesis, based on the ratio of the number of days in the control period to that in the postvaccination period. The estimate of the relative risk is dr/c, which changes as data accumulate weekly. As Kulldorff et al. (27) previously described, the likelihood under the null hypothesis is L₀ = (1/[r + 1])^d × (r/[r + 1])^c, and the likelihood under the alternative hypothesis is L₁ = (d/[c + d])^d × (c/[c + d])^c. The log-likelihood ratio is then ln(L₁/L₀) = d ln(d/[c + d]) + c ln(c/[c + d]) – d ln(1/[r + 1]) – c ln(r/[r + 1]).

Figure 1.

Risk and control periods for an example potential adverse event proximal to vaccination in the 2007/08 influenza season. The cumulative number of events occurring during each period, summed across previous seasons, is represented by a and b. The number of events occurring during the current season is represented by c and d. A) In the self-controlled case series approach, d is compared with c. B) In the difference-in-difference approach, d is compared with c, adjusting for the comparison of b with a.

In the DID approach, the null hypothesis is that the relative risk of an adverse event's occurrence in the risk period versus the control period during the current influenza season is no greater than the same relative risk across cumulative previous seasons. The vaccine in previous seasons was assumed to have an acceptable risk profile (5). The estimate of the relative risk from previous seasons was based on at least 5 seasons and was treated as known without error (51); however, DID analyses were conducted only for adverse events with 25 or more cases in the risk and control periods in previous seasons. Accumulated from vaccinated cases across previous seasons, a represents the number of adverse events in the control period and b represents the number of adverse events in the postvaccination risk period (Figure 1, part B). For DID, the estimate of the relative risk ratio is (d/c)/(b/a). The likelihood is L₀ = (b/[a + b])^d × (a/[a + b])^c, while L₁ is identical to that of the SCCS approach. Therefore, the log-likelihood ratio is ln(L₁/L₀) = d ln(d/[c + d]) + c ln(c/[c + d]) – d ln(b/[a + b]) – c ln(a/[a + b]).

For Guillian-Barré syndrome, we used a Poisson-based approach (27) to compare the risk following TIV in the current season with that in previous seasons (relative risk = [d/doses in current season]/[b/doses in previous seasons]). SCCS and DID analyses were not conducted for this adverse event; since the occurrence of Guillian-Barré syndrome within 6 weeks of TIV is a precaution for receiving future TIV (5), fewer cases of Guillian-Barré syndrome may occur among vaccinees during a control period prior to vaccination. This can bias SCCS analysis, and there may be too few events to adjust for this bias using DID analysis. In contrast, a postvaccination control period matching the length of a 42-day risk period would not elapse until 84 days after vaccination, long after most vaccinations are administered.

Prospective weekly surveillance was mimicked using sequential analysis (10), with the strong assumption of complete data accrual without delay. To account for repeated weekly looks at the data, we applied the maximized sequential probability ratio test, using a binomial probability model for all adverse events except Guillian-Barré syndrome (27). The null hypothesis was rejected if the log-likelihood ratio reached an upper bound (the “critical value” of the log-likelihood ratio). The null hypothesis was not rejected if the total number of adverse events reached a prespecified “upper limit,” or if surveillance ended without reaching this upper limit and without the log-likelihood ratio reaching the critical value.

For each adverse event, specific to each age group and season, the critical value of the log-likelihood ratio was dictated by the user-specified upper limit of expected adverse events and α-level. Upper limits were selected on the basis of the approximate number of adverse events expected under the null hypothesis, assuming that the risk in the current season was no greater than the risk in cumulative previous seasons. For example, upper limits for 2005/06 were determined using the risk of adverse events in the VSD population during risk and control periods around influenza vaccination, accumulated across the 2000/01–2004/05 influenza seasons; upper limits for the following season, 2006/07, were determined using data accumulated across 2000/01–2005/06, etc. If upper limits were reached before all cases were accumulated for the season, the power would be lower than what would have been possible with a higher upper limit. Therefore, upper limits were chosen such that they would not usually be reached. The upper limits were identical between the SCCS and DID analyses.

For very rare adverse events (e.g., anaphylaxis), surveillance was conducted for all ages together. For less rare adverse events (e.g., seizures), surveillance was conducted for children and adults separately. Because many adverse events and age groups were considered, α = 0.01 was selected. We used 1-tailed tests, looking only for risks of vaccination rather than protective effects. Surveillance ended after the risk and control periods passed for April vaccinees; if the upper limit had not been reached by that point, the cumulative α value remained below 0.01, indicating a conservative test.

Statistical power for adverse events in the 2007/08 season was calculated using previously described simulations (27) and was determined by the ratio of days in the control period to days in the risk period, the stringent selection of α = 0.01, and predefined upper limits. Since surveillance could end before upper limits were reached, statistical power could only be approximated, assuming that surveillance lasted at least halfway to the upper limit (Table 2). For the Poisson-based Guillian-Barré syndrome analysis in 2007/08, the background incidence following TIV was assumed from previous seasons to be 8.5 per million doses. With the upper limit equal to 20 and α = 0.05, the power was 45% to detect a relative risk of 1.5 and 92% to detect a relative risk of 2.0 (27).

Table 2.

Statistical Power of the Binomial-based Maximized Sequential Probability Ratio Test for Different Adverse Events and True Relative Risks, Vaccine Safety Datalink Project, 2007/08

Adverse Event Category	No. of Days in Control Period	No. of Days in Risk Period	Age Groupa	Prespecified Upper Limit of No. of Events	Rangeb of Statistical Power for α = 0.01, %
					To Detect RR = 1.5	To Detect RR = 2.0	To Detect RR = 5.0
Seizure	8	8	Children	200	18–45	64–96	>99–100
			Adults	500	57–92	99–>99	100
Meningoencephalitis	60	21	All ages	150	12–28	47–84	>99–100
Bell's palsy	60	42	Children	50	4–10	14–34	78–>99
			Adults	1,000	92–>99	100	100
Other cranial nerve disorder	60	42	Children	25	2–5	6–15	35–77
			Adults	1,200	96–>99	100	100
Demyelinating disease	60	42	Children	15	2–4	4–11	22–62
			Adults	800	83–>99	>99–100	100
Disorder of the peripheral nervous system	60	42	Children	80	6–15	23–56	97–100
			Adults	2,000	>99–100	100	100
Ataxia	60	42	All ages	80	6–15	23–56	97–100
Allergic reaction other than anaphylaxis	2	2	Children	200	18–45	64–96	>99–100
			Adults	400	44–84	96–>99	100

Abbreviation: RR, relative risk.

^aAge groups: all ages, ≥6 months; children, 6 months–17 years; adults, ≥18 years.

^bBecause the upper limit is not always reached, the exact power is unknown. Assuming that at least half of the upper limit is reached, the statistical power to detect a signal is within the ranges shown.

Critical values of the log-likelihood ratio, above which the null hypothesis would be rejected, were obtained for Poisson and SCCS analyses for r = 1 from published tables (27). Critical values for SCCS analysis for r ≠ 1 and for DID analysis (accounting for the ratio of cases in historical control periods to cases in historical risk periods) were obtained from the same program used to generate the published tables (27).

Analyses were conducted using SAS, version 9.1.3 (SAS Institute Inc., Cary, North Carolina). Sample input data, SAS code, and output are provided in the Web Appendix, which is posted on the Journal’s Web site (http://aje.oxfordjournals.org/).

RESULTS

The number of TIV doses administered to children and adults in the VSD population increased each season, consistent with increased utilization following expanded recommendations for TIV use (52–54). In 2005/06, 317,108 first TIV doses were administered to children aged 6 months–17 years, and 1,432,218 were administered to adults aged ≥18 years. In 2006/07, the first dose counts in these age groups were 415,446 and 1,598,880, respectively, and in 2007/08, the dose counts were 462,998 and 1,742,858. Vaccine administration was temporally concentrated; each season, more than 85% of vaccines were administered in October and November.

Fifteen age-specific adverse event categories were analyzed for each of 3 seasons, for a total of 45 sequential analyses each using SCCS and DID approaches and 3 additional sequential analyses using the Poisson-based approach for Guillian-Barré syndrome. Of these, the number of adverse events reached the prespecified upper limit in 1 analysis in all ages (Guillian-Barré syndrome in 2006/07) and in 5 analyses in adults (seizure in 2005/06, other cranial nerve disorder in 2005/06, and disorder of the peripheral nervous system in all 3 seasons).

The critical value of the log-likelihood ratio was not reached in any analysis, so no associations between vaccination and adverse events warranting further investigation were detected. Summarized results as of each season's final surveillance week are shown in Table 3. The DID analyses, which adjusted for the relative risk in previous seasons, yielded relative risk ratio estimates that did not always correspond to the SCCS results in terms of magnitude and direction of effect, as would be expected. The relative risk and relative risk ratio estimates obtained using both approaches were not statistically significant for any adverse event. The Poisson-based approach for Guillian-Barré syndrome showed that the risk after vaccination each season was not significantly greater than in previous seasons.

Table 3.

Results of Sequential Testing for Influenza Vaccine Safety as of Each Season's Final Surveillance Week, Vaccine Safety Datalink Project, 2005/06–2007/08

Adverse Event Category and Age Groupa	Influenza Season	No. of Vaccine Doses Administered	No. of Cases		Upper Limit of No. of Events	No. of Cases		Self-Controlled Case Series Analysis			Difference-in-Difference Analysis
			In Current Risk Period	In Current Control Period		In Historical Risk Periods	In Historical Control Periods	RR	LLRb	Critical Value of LLR	RRR	LLRb	Critical Value of LLR
Seizure
Children	2005/06	317,108	58	43	120	133	149	1.35	1.12	5.37	1.51	2.13	5.26
	2006/07	415,446	51	64	200	191	192	0.80		5.51	0.80		5.51
	2007/08	462,998	55	56	200	243	257	0.98		5.51	1.04	0.02	5.40
Adults	2005/06c	1,429,974	199	201	400	605	724	0.99		5.55	1.18	1.44	5.54
	2006/07	1,598,880	153	160	500	808	933	0.96		5.56	1.10	0.38	5.62
	2007/08	1,742,858	168	154	500	965	1,104	1.09	0.30	5.56	1.25	1.98	5.59
Meningoencephalitis, all ages	2005/06	1,749,326	23	87	120	71	319	0.76		5.40	1.19	0.26	5.11
	2006/07	2,014,326	21	53	150	94	406	1.13	0.11	5.40	1.71	1.99	5.06
	2007/08	2,205,856	22	71	150	115	459	0.89		5.40	1.24	0.37	5.17
Bell's palsy
Children	2005/06	317,108	8	17	40	28	39	0.67		5.01	0.66		5.14
	2006/07	415,446	19	15	50	36	56	1.81	1.49	5.24	1.97	1.94	4.92
	2007/08	462,998	16	18	50	55	71	1.27	0.24	5.24	1.15	0.08	4.97
Adults	2005/06	1,432,218	271	365	800	1,025	1,473	1.06	0.27	5.72	1.07	0.33	5.70
	2006/07	1,598,880	300	399	800	1,296	1,838	1.07	0.44	5.72	1.07	0.35	5.72
	2007/08	1,742,858	311	447	1,000	1,596	2,237	0.99		5.77	0.98		5.81
Other cranial nerve disorder
Children	2005/06	317,108	3	4	20	11	23	1.07	0.00	4.53	1.57	0.17	4.51
	2006/07	415,446	6	9	25	14	27	0.95		4.72	1.29	0.11	4.68
	2007/08	462,998	3	11	25	20	36	0.39		4.72	0.49		4.94
Adults	2005/06c	1,416,883	337	466	800	1,172	1,582	1.03	0.10	5.72	0.97		5.75
	2006/07	1,598,880	391	532	1,000	1,512	2,051	1.05	0.27	5.77	1.00		5.81
	2007/08	1,742,858	374	544	1,200	1,903	2,583	0.98		5.81	0.93		5.84
Demyelinating disease
Children	2005/06	317,108	3	2	15	15	11	2.14	0.36	4.49	1.10	0.01	4.41
	2006/07	415,446	1	3	20	18	13	0.48		4.53	0.24		4.81
	2007/08	462,998	3	5	15	19	16	0.86		4.49	0.51		4.28
Adults	2005/06	1,432,218	171	298	600	753	1,189	0.82		5.65	0.91		5.68
	2006/07	1,598,880	201	325	600	924	1,487	0.88		5.65	1.00		5.75
	2007/08	1,742,858	194	347	800	1,125	1,812	0.80		5.72	0.90		5.76
Disorder of the peripheral nervous system
Children	2005/06	317,108	9	22	50	46	56	0.58		5.24	0.50		5.01
	2006/07	415,446	10	20	60	55	78	0.71		5.24	0.71		5.23
	2007/08	462,998	19	27	80	65	98	1.01		5.32	1.06	0.02	5.26
Adults	2005/06c	569,039	938	1,524	2,000	7,893	11,642	0.88		5.95	0.91		5.89
	2006/07c	601,861	1,007	1,482	2,000	10,212	14,918	0.97		5.95	0.99		5.90
	2007/08c	581,324	887	1,355	2,000	12,792	18,542	0.94		5.95	0.95		5.93
Ataxia, all ages	2005/06	1,749,326	13	27	80	86	105	0.69		5.32	0.59		5.20
	2006/07	2,014,326	27	38	80	99	132	1.02	0.00	5.32	0.95		5.08
	2007/08	2,205,856	35	36	80	126	170	1.39	0.95	5.32	1.31	0.65	5.12
Anaphylaxisd, all ages	2005/06	1,749,326	3	1	12	2	6	3.00	0.52	4.27		N/A
	2006/07	2,014,326	4	1	12	5	7	4.00	0.96	4.27		N/A
	2007/08	2,205,856	1	1	12	9	8	1.00	0.00	4.27		N/A
Allergic reaction other than anaphylaxise
Children	2005/06	319,203	49	52	150	162	122	0.94		5.45	0.71		5.35
	2006/07	418,081	96	70	200	211	174	1.37	2.04	5.51	1.13	0.31	5.41
	2007/08	459,263	104	70	200	307	244	1.49	3.34	5.51	1.18	0.58	5.37
Adults	2005/06	1,466,842	137	98	300	459	376	1.40	3.25	5.55	1.15	0.53	5.44
	2006/07	1,635,562	156	125	300	596	474	1.25	1.71	5.55	0.99		5.51
	2007/08	1,746,619	196	164	400	752	599	1.20	1.42	5.55	0.95		5.61
						Expected No. of Cases Based on Historical Risk Periods	Poisson-based Analysis
							RR	LLRb	Critical Value of LLR
Guillian-Barré syndromee, all ages	2005/06	1,788,108	12		15	14.4		0.83		3.57
	2006/07c	1,960,430	17		15	15.1		1.13	0.12	3.57
	2007/08	2,214,335	23		20	16.7		1.37	1.05	3.63

Abbreviations: LLR, log-likelihood ratio; N/A, not applicable; RR, relative risk; RRR, relative risk ratio.

^aAge groups: all ages, ≥6 months; children, 6 months–17 years; adults, ≥18 years.

^bSince the statistical tests were 1-sided, the LLR was not calculated when the RR was less than 1.0.

^cCurrent season results for these adverse events are presented as of the week the upper limit was reached, not as of the end of surveillance.

^dDifference-in-difference analyses were not conducted for anaphylaxis, since there were fewer than 25 cases in the risk and control periods in previous seasons.

^eAnalyses for allergic reactions and Guillian-Barré syndrome were conducted after the analyses for the other categories. Because of the dynamic nature of the data source, the dose counts for these categories fluctuated in comparison with those for the other categories.

Weekly sequential analysis results for 1 representative outcome, seizure in adults, are highlighted for both the SCCS (Figure 2, part A) and DID (Figure 2, part B) approaches. The relative risk or relative risk ratio had a rapid increase early in the season with few doses administered, but as the season progressed and more doses were given, the relative risk or relative risk ratio stabilized and quickly trended towards 1. During surveillance, the log-likelihood ratio peaked at only 12% of the critical value for SCCS and at only 35% of the critical value for DID. In other words, the SCCS approach showed that vaccinated cases had no statistically significant elevated risk of seizure in the risk period versus the control period in 2005/06, 2006/07, or 2007/08. The DID approach showed that this relative risk was not significantly greater in 2005/06 than in 2000/01–2004/05, not significantly greater in 2006/07 than in 2000/01–2005/06, and not significantly greater in 2007/08 than in 2000/01–2006/07.

Figure 2.

Weekly sequential analysis results for seizures among trivalent inactivated influenza vaccine recipients aged ≥18 years in the 2007/08 influenza season, Vaccine Safety Datalink Project. A) Self-controlled case series approach; B) difference-in-difference approach.

DISCUSSION

Near real-time surveillance for selected adverse events following seasonal or pandemic influenza vaccine is possible in systems such as the VSD Project, where data are updated at least weekly. Implementing SCCS and DID approaches in sequential testing allows for a timely alternative to more traditional systems for evaluating influenza vaccine safety, while adjusting for important confounders. Previous randomized controlled trials (49, 55, 56), passive reporting systems (30, 57), and postlicensure safety studies (20, 21) have supported the overall safety of TIV. This multisite exercise was consistent with previous studies in showing no evidence of elevated risk following TIV for any of 10 predefined adverse event categories. Poisson-based analyses indicated that in each of 3 recent seasons, the risk of Guillian-Barré syndrome following vaccination was not significantly different from that in previous seasons. SCCS analyses indicated that the risk of adverse events during the risk period was not significantly different from that during the control period. DID analyses indicated that the relative risk in each season was not significantly different from the relative risk across cumulative previous seasons. SCCS and DID analyses compared risk periods with control periods within the same vaccinees each season, thus controlling for potential biases. Using binomial-based maximized sequential probability ratio testing, power was adequate for most adverse event categories to detect relative risks in the range of 2.0 and very good to detect high relative risks in the range of 5.0; more modest risk elevations may not have been detectable using these methods. However, if a modest elevation in risk were undetected for a very rare adverse event, the population attributable risk for vaccination would be very small. The lack of any safety signals can increase public confidence in influenza vaccine safety.

Using more than 1 analytic approach for surveillance may enhance the rigor of a safety assessment. A signal based on either method would warrant additional investigation for assessment of its validity. If both approaches were to produce a signal for the same adverse event, evidence for a safety concern requiring further evaluation would be strengthened. The relative risks and relative risk ratios produced by SCCS and DID were different, which was expected because they are calculated differently, but neither approach generated a safety signal. The application of repeated confidence intervals to maximized sequential probability ratio testing is under development (58).

At least 4 potential limitations affected the analyses. First, timely detection of a safety problem depends partially on the lag between adverse event occurrence and electronic reporting. This lag is typically 2 weeks for outpatient and emergency department data, but it ranges from 1 day to several weeks for hospitalizations, depending on the data source. The delay in ascertaining hospitalizations could be reduced with additional resources by using health plans that ascertain hospitalizations via electronic records (not claims) or by monitoring preadjudicated claims. Near real-time surveillance is most useful for adverse event categories requiring observation of only a brief risk period of less than 2–3 weeks, so that a signal could occur within a month after vaccines are given. In such a situation, surveillance may identify a safety problem before most vaccines for a season are administered, and warnings may be issued to prevent additional exposure. In contrast, using sequential analyses for adverse events with a lengthy risk window or a postvaccination control period sacrifices both power and simplicity with little potential gain in timeliness compared with a final analysis each spring.

Second, the VSD Project uses electronic data collected largely for administrative purposes. Some vaccinations are missed (16), as are telephone encounters and adverse events that are not medically attended. The sensitivity, specificity, and positive and negative predictive values of ICD-9 codes vary by adverse event and setting (59–61) and have not been thoroughly characterized. Third, binomial-based maximized sequential probability ratio testing had limited power for timely detection of a modest elevation in risk for very rare adverse events (e.g., anaphylaxis). To minimize false-negative findings, power could be improved as the VSD population expands and by investing in larger surveillance systems—for example, the US Food and Drug Administration's Sentinel Initiative, which aims to use electronic data from 100 million people by 2012 (62). Using a historical comparison group and a Poisson probability model is another alternative for rare adverse events and may have better statistical power (27). Finally, although the SCCS and DID approaches adjusted for some confounders, they did not adjust for seasonality of vaccination and adverse event risk. A case-centered approach (63) accounting for the timing of vaccination in the overall population could be used to adjust for seasonal effects.

To our knowledge, this is the first application of both SCCS and DID approaches to the sequential monitoring of adverse events. In addition to influenza vaccines, these approaches may be applicable to monitoring the safety of other vaccines for which finding an appropriate comparison group is challenging and the healthy vaccinee effect may be substantial. These approaches also may be useful in drug safety surveillance (e.g., for influenza antiviral medications) and for other data sources (e.g., Medicare or other insurers) (62, 64, 65). New adverse events of potential concern (e.g., those detected through the Vaccine Adverse Event Reporting System) can be quickly added to surveillance, and subgroup analyses (e.g., for manufacturer-specific vaccine products) can be readily undertaken, although power may be limited. If a safety signal is generated in the future, additional investigations will distinguish true positives from false positives, including evaluation of the temporal association between vaccination and adverse events, high-risk subgroup analyses, and medical chart review (12, 66).

Near real-time safety surveillance for seasonal and pandemic influenza vaccines is possible, and prospective analyses for seasonal influenza vaccines are being conducted, starting with the 2008/09 season. With a well-defined population and the ability to access medical records for review, the VSD Project will monitor the safety of influenza A (H1N1) 2009 monovalent vaccines (67), using binomial- (SCCS and DID) and Poisson-based approaches.

Glossary

Abbreviations

DID

difference-in-difference

ICD-9

International Classification of Diseases, Ninth Revision

SCCS

self-controlled case series

TIV

trivalent inactivated influenza vaccine

VSD

Vaccine Safety Datalink