Near real‐time vaccine safety surveillance using electronic health records—a systematic review of the application of statistical methods

Abstract

Purpose

Pre‐licensure studies have limited ability to detect rare adverse events (AEs) to vaccines, requiring timely post‐licensure studies. With the increasing availability of electronic health records (EHR) near real‐time vaccine safety surveillance using these data has emerged as an option. We reviewed methods currently used to inform development of similar systems for countries considering their introduction.

Methods

Medline, EMBASE and Web of Science were searched, with additional searches of conference abstract books. Questionnaires were sent to organizations worldwide to ascertain unpublished studies. Eligible studies used EHR and regularly assessed pre‐specified AE to vaccine(s). Key features of studies were compared descriptively.

Results

From 2779 studies, 31 were included from the USA (23), UK (6), and Taiwan and New Zealand (1 each). These were published/conducted between May 2005 and April 2015. Thirty‐eight different vaccines were studied, focusing mainly on influenza (47.4%), especially 2009 H1N1 vaccines. Forty‐six analytic approaches were used, reflecting frequency of EHR updates and the AE studied. Poisson‐based maximized sequential probability ratio test was the most common (43.5%), followed by its binomial (23.9%) and conditional versions (10.9%). Thirty‐seven of 49 analyses (75.5%) mentioned control for confounding, using an adjusted expected rate (51.4% of those adjusting), stratification (16.2%) or a combination of a self‐controlled design and stratification (13.5%). Guillain‐Barré syndrome (11.9%), meningitis/encephalitis/myelitis (11.9%) and seizures (10.8%) were studied most often.

Conclusions

Near real‐time vaccine safety surveillance using EHR has developed over the past decade but is not yet widely used. As more countries have access to EHR, it will be important that appropriate methods are selected, considering the data available and AE of interest. © 2016 The Authors. Pharmacoepidemiology and Drug Safety Published by John Wiley & Sons Ltd.

Introduction

Vaccines are considered to be one of the most cost‐effective interventions in public health.1, 2 As with other drugs, vaccines are not totally safe,3 but safety requirements are particularly high as vaccines are given to healthy individuals, most often children.4 All vaccines go through extensive safety assessment before licensure; however, pre‐licensure studies have limited ability to detect rare adverse events (AEs) to vaccines (with frequency <1/10 000‐1/100 000)5, AE occurring among specific sub‐populations who were not included in clinical trials, and long‐term AE.6 To overcome these limitations, timely post‐licensure studies are required. These can be broadly divided into passive (spontaneous reports) and active studies and should be followed by confirmatory epidemiologic studies. While spontaneous reporting of AE is widely implemented worldwide as a simple and low‐cost method, useful to detect new, unanticipated AE, it has limitations.2 These include difficulties in denominator calculation, potential reporting biases (e.g. over‐reporting of potential AE receiving extensive media coverage) and incomplete reporting. In contrast, active surveillance tries to identify all those experiencing (or at least seeking medical attention for) a potential AE to vaccines. This approach includes analyses of large population datasets (using electronic health records (EHR)), targeted hospital‐based surveillance (where trained health workers daily seek potential cases of conditions of interest) and recruitment of vaccinated cohorts for detection of AE (using face‐to‐face interviews, phone interviews, short‐message services or web‐based tools).7, 8 With the increased availability of large population datasets, near real‐time vaccine safety surveillance (NRTVSS) has emerged as an option.9

Near real‐time vaccine safety surveillance, also known as rapid cycle analysis, involves regular interrogation of EHR to investigate pre‐specified AE to vaccines.2 By testing these AE on a regular basis after introduction of a new vaccine, these methods ensure a timely detection of possible safety problems.10 When a signal is detected by this approach, it needs to be further analysed, including a signal refinement stage and eventual confirmatory analyses. These steps should be predetermined and will lead to the decision of whether to validate or invalidate the signal. NRTVSS is thus part of a systematic approach to signal detection, with a dual role of signalling possible AE to vaccines and reassuring authorities and populations that events are being monitored.11 For a given vaccine, NRTVSS only considers a small number of suspected AE (e.g. 5 to 10); complementary information is provided by existing methods such as spontaneous reports.12

The growing use of NRTVSS methods, along with the increasing availability of EHR, highlights the need to review studies using this approach. Such a review can provide crucial information on the development of systems for vaccine safety surveillance for countries considering their introduction.

Objective

The aim of this study was to carry out a systematic review of published and unpublished data on the methods used for NRTVSS using EHR.

Methods

Studies were included in the review if they (i) used routinely collected health data (at least for the expected number of events); (ii) studied pre‐specified outcome(s) to assess the safety of one or more vaccines; and (iii) regularly tested the outcomes. Studies (i) including only information based on spontaneous reporting systems, (ii) aimed at testing hypothesis/confirming previously generated/suspected signals or (iii) aimed at developing new methods for NRTVSS (unless a specific application of the new method was given) were excluded. No limits were imposed in terms of language or year.

Medline and EMBASE were searched for studies published until 6 January 2015, using a combination of thesaurus and free‐text terms (search strategy is provided in Supporting Information Appendix A). Titles and abstracts were reviewed to determine eligibility status, followed by the full text for those considered potentially eligible. References from the papers collected were also reviewed. Reviews of the topic were selected for reference mining. A. L. was responsible for evaluating eligibility of the identified studies. To ensure quality, eligibility of a random sample of 10% of the results was evaluated by S. T. and N. A. When eligibility was unclear, the study was discussed among the authors until a consensus was reached.

To complement the database searches, a citation search was conducted. To the best of our knowledge, the methods under study were first applied to the field of vaccine safety by the Vaccine Safety Datalink (VSD). Two key VSD papers that describe the testing and implementation of rapid cycle analysis using routinely collected health data were selected to perform a citation search.9, 13

The same search strategy was used in the Web of Science Core Collection to cover meetings and conferences, restricting the search to meeting abstracts or proceedings papers. Also, the Annual Conference on Vaccine Research and the Vaccine and ISV Congress abstract book and programme, respectively, were analysed (Supporting Information Appendix B). The Brighton Collaboration newsletter was also searched as a potential source of relevant new studies or contacts.14

A second stage of the review included contacting experts in vaccine safety, as follows:

• Specialists in vaccine safety (from the Global Advisory Committee on Vaccine Safety (GACVS),15 Brighton Collaboration16 and Accelerated Development of Vaccine benefit–risk collaboration in Europe (ADVANCE) 17) were asked if they were aware of work being conducted in the area and fulfilling our inclusion criteria.

• Authors with known work using routinely collected data and the potential to have implemented/conducted eligible studies were contacted (Medicines and Healthcare products Regulatory Agency (MHRA),18 VSD19 and Statens Serum Institute20). Further contacts were also asked for at this stage.

• Finally, authors with a previous published work but incomplete information, and those suggested by other experts, were contacted to ask for further information to characterize the methods.

An online questionnaire was used to capture information on studies conducted (Supporting Information Appendix C). When other sources of information (e.g. reports) were available and shared by the contacts these were used. Expert contacts took place from February to March 2015.

The information identified was extracted using a standardized extraction form. Data extracted included timeline, country/institutions where the study was conducted, vaccines studied, study population, outcomes assessed and their method of ascertainment, methods used to perform the analyses, frequency of assessment, confounding, data‐accrual lag (i.e. delays in the data available to perform surveillance, which may affect the results), assessment of the validity of the outcomes of interest (e.g. chart review) and main results. A descriptive summary of country/institution, vaccines, outcomes studied, confounding and data‐accrual lag handling was drawn up.

Results

A total of 29 reports were included for data extraction (including information provided by expert contacts),9, 13, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45 representing 31 studies/systems (Figure 1). A brief description of the studies/systems included by country, methods used and adjustment for confounding strategies is given in Table 1. A detailed characterization of the studies is provided in Supporting Information Appendix D.

Figure 1.

Flowchart of included studies. Studies were excluded for (i) not considering vaccines (nonvaccine), (ii) not analysing the safety of a vaccine (not safety), (iii) considering safety issues but not applying the methods of interest (other safety), (iv) only developing new methods (methods only) and (v) having no abstract available (not available)

Table 1.

Included studies according to the country, methods used and control for confounding strategies (see Supporting Information Appendix D for further details)

Study	Country, organization	Method	Confounding	Data‐accrual lag or underreporting adjustment
Retrospective
Davis9	USA, VSD	SPRT	Risk adjustment* (site, age, time, season, sex)	Retrospective
Lieu13, †	USA, VSD	PMaxSPRT	Unclear	Retrospective
Brown22	USA, i3 Drug Safety	PMaxSPRT	Expected counts (sex, age, region, month, concomitant vaccination)	Retrospective; data lags assessed during the study
Greene24, †	USA, VSD	PMaxSPRT	Expected rates (age and site)	Retrospective—data assumed to accrue without delay
		BMaxSPRT‡	SC; stratification (age, season)
Prospective
Lieu13	USA, VSD	PMaxSPRT	No adjustment	Analyses waited at least 6 weeks from the vaccination or preventive visit
		BMaxSPRT	Matching (age, week, site)
McNicholas 21	New Zealand, MoH	SPC	Stratification (age)	Daily review of databases, medical charts, discharge letters and laboratory records
Yih23	USA, VSD	PMaxSPRT	Expected counts (GBS/seizures—age; other AE—age, sex)	Analysis started at least 8 weeks from the date of vaccination46 and redone at the end of the study
Belongia25, †	USA, VSD	PMaxSPRT	Expected rates (intussusception—trend, age, site by Poisson regression; other AE—site)	Analysis started at least 8 weeks from the date of vaccination46
Bryan28	UK, MHRA	PMaxSPRT	Expected rates (age and gender)	Adjusted for underreporting (yellow‐card data)
Huang30, †	Taiwan, CDC	PMaxSPRT	Stratification (age)	Database updated daily
	BMaxSPRT‡	SC
Enger29	USA, i3 Drug Safety	Unclear	Unclear	Unclear
DMSS26, 32, 47, 48	USA, DoD	PMaxSPRT	Unclear	Unclear
VA26, 32, 48	USA, VA	PMaxSPRT	Unclear	Unclear
IHS26, 32, 48, 49	USA, IHS/FDA	PMaxSPRT	Unclear	Unclear
PRISM26, 32, 48	USA, FDA/NVPO	PMaxSPRT	Unclear	Unclear
	BMaxSPRT‡
Klein27, †	USA, VSD	BMaxSPRT	Matching (age group, site, calendar year and respiratory virus season)	Analysis delayed at least 8 weeks from date of vaccination46
Gee34	USA, VSD	PMaxSPRT	Expected rates (age, site)	Unclear
	BMaxSPRT§	Matching (age, site, vaccination date)
Lee33	USA, VSD	PMaxSPRT¶	Expected rates (age and site)	Adjusted for partially elapsed risk interval and delay in the arrival of inpatient data
	BMaxSPRT‡	SC
	Both	Stratification** (age)
Bryan 31, †	MHRA, UK	PMaxSPRT	Expected rates (age)	Adjusted for underreporting (yellow‐card data)
Burwen36	USA, FDA	USPRT	No	Critical limits adjusted for delays in the claims (based on previous seasons)
Loughlin35, †	USA, OptumInsight	Abt's modification of SPRT	No	No
Tse37	USA, VSD	PMaxSPRT¶	Stratification (age, site)	Adjusted for partially elapsed risk interval and delay in the arrival of inpatient data
	BMaxSPRT‡	SC
Donegan40, †	UK, MHRA	PMaxSPRT	Stratification (age)—first year of surveillance	Sensitivity analyses assuming various degrees of underreporting (yellow‐card data)
Nelson38	USA, VSD	GS PMaxSPRT	Expected counts (site, gender, age group, site × age—Poisson regression)	No††
Tseng39	USA, VSD	GS	Stratification (age, dose number—only for febrile seizures, urticaria/angioneurotic oedema, asthma)	No††
Daley42, †	USA, VSD	PMaxSPRT¶	Expected rates (site—except for GBS and SJS—weighted average used)	Exclusion of the most recent 14 weeks of data11
Kawai43	USA, VSD	PMaxSPRT¶	Expected rates adjusted (age, site)	Delayed analysis until estimated data lag accrual and follow‐up time was completed
	BMaxSPRT‡	SC, stratification (age)
Weintraub41, †	USA, VSD	PMaxSPRT	Expected rates (age, site)	Analysis delayed 2 weeks
Murdoch 44, †	UK, HPS	SPC	Stratification (age, site)	No
Yih 45	USA, FDA	PMaxSPRT¶	Expected rates (age for anaphylaxis and seizures and data partner for seizures)	Adjusted for partially elapsed risk interval and delay in the arrival of inpatient data
	BMaxSPRT‡	SC, stratification (seizures—age, concomitant PCV13 6–23 months)
HPS † (unpublished)	UK, HPS	SPC	Stratification (age, sex for herpes zoster, site)	No
MHRA † (unpublished)	MHRA, UK	PMaxSPRT	Expected rates (age)	Adjusted for underreporting (yellow‐card data)

Studies in italic are the ones identified from expert contacts.

AE‐Adverse event; BMaxSPRT, binomial‐based maximized sequential probability ratio test; CDC, Centers for Disease Control and Prevention; DMSS, Defense Medical Surveillance System; DoD, Department of Defense; FDA, Food and Drug Administration; HPS, Health Protection Scotland; IHS, Indian Health Service; MHRA, Medicines and Healthcare products Regulatory Agency; MoH, Ministry of Health; NVPO, National Vaccine Program Office; PCV13, 13‐valent pneumococcal conjugate vaccine; PMaxSPRT, Poisson‐based maximized sequential probability ratio test; PRISM, Post‐Licensure Rapid Immunization Safety Monitoring; SC, self‐controlled design; SJS, Stevens–Johnson syndrome; SPC, statistical process control; SPRT, sequential probability ratio test; USPRT, updating sequential probability ratio test; VA, Veterans Affairs; VSD, Vaccine Safety Datalink.

^* Each unique combination of potential confounders is identified, forming a stratum, and a baseline risk is calculated. For each stratum, a test statistic is calculated, and the test statistics are combined.

^† Additional information obtained from the authors.

^‡ Uses a self‐controlled design.

^§ Uses an exact version of the test, with flexible matching.

^¶ Uses the conditional version of the test.

^** Only for inactivated vaccines and specific outcomes (demyelinating disease of the central nervous system, disorders of the peripheral nervous system and neuropathy, seizures, Bell's palsy and other cranial nerve disorders).

^†† Analysis based on the number of doses might minimize delays for initial periods of surveillance.

Near real‐time vaccine safety surveillance using EHRs was first reported by Davis et al. in 2005, when a retrospective study assessing the feasibility of implementing such methods was published. Since this time, we identified a further 13 studies conducted by the VSD and 17 other studies in three countries (Figure 2). The first study conducted outside the VSD was conducted in New Zealand and published in 2007. The report from the last study included was published online in 2015. Four studies (all in the USA) were conducted completely or partially in a retrospective manner, to test the feasibility of implementing this kind of system (Table 1). Two of these studies attempted to replicate known signals (rotavirus vaccine and intussusception and acellular diphtheria‐tetanus‐pertussis (DTaP)/whole cell diphtheria‐tetanus‐pertussis vaccine and febrile seizures). Of the prospective studies, most were conducted in the USA (n = 20), with studies also conducted in the UK (n = 6), and Taiwan and New Zealand (n = 1 for each). The prospective studies looked mainly at influenza vaccines (n = 16), especially the 2009 H1N1 pandemic influenza vaccine (n = 7). Rotavirus (n = 5), DTaP‐based (n = 3) and human papillomavirus vaccines (n = 3) also received attention.

Figure 2.

Studies included in the review, ordered by the year of publication. Continuous sequential test are displayed in blue, group sequential in orange and statistical process control in green. Grey background indicates non‐published studies. *Results with previous published results. MaxSPRT, maximized sequential. Probability ratio test: P, Poisson version (^†use of the conditional version); B, binomial version (^‡use of self‐controlled case series or extensions of the test). DMSS, Defense Medical Surveillance System; DTaP, acellular diphtheria‐tetanus‐pertussis vaccine; DTwP, whole cell diphtheria‐tetanus‐pertussis vaccine; GBMV, group B meningococcal vaccine; HPS, Health Protection Scotland; HPV2, bivalent human papillomavirus vaccine; HPV4, quadrivalent human papillomavirus vaccine; IHS, (US) Indian Health. Service: IPV, inactivated poliovirus vaccine; MCV, meningococcal conjugate vaccine; MHRA, Medicines and Healthcare products Regulatory Agency; MMRV, measles‐mumps‐rubella‐varicella combination vaccine; PCV13, 13‐valent pneumococcal conjugate vaccine; PRISM, Post‐Licensure Rapid Immunization Safety. Monitoring: RRV, rhesus‐rotavirus vaccine; RV5, pentavalent rotavirus vaccine; VA, Veterans Affairs; VSD, Vaccine Safety Datalink

The outcomes studied were most often neurological (58.5%). Looking at specific outcomes, Guillain‐Barré syndrome (GBS) (11.9% of studied known outcomes), meningitis/encephalitis/myelitis (11.9%) and seizures (10.8%) were the most often included. Outcome ascertainment for the near real‐time analysis was, in most cases, based on automated data (with no a priori confirmation of the diagnosis). In these cases, chart review and confirmation were used whenever a potential AE was signalled. Only two studies performed this kind of confirmation for the near real‐time analysis,21, 35 and one compared the analysis considering the chart‐reviewed and non‐reviewed outcome for GBS.33 From the outcomes studied, 11 signals were identified, but only three confirmed (measles‐mumps‐rubella‐varicella combination vaccine and febrile seizures,27 2010–2011 trivalent inactivated influenza vaccine and febrile seizures,37 and monovalent rotavirus vaccine and intussusception41).

Table 2 summarizes the methods used by the studies included in this review. These can be broadly divided into continuous sequential testing, which allows examination of the data as often as desired (n = 25),9, 13, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 37, 38, 40, 41, 42, 43, 45 group sequential testing (n = 4)35, 36, 38, 39 and statistical process control (SPC; n = 3).21, 44 The choice of the group of methods has been determined by the frequency of updates to the EHR data used (Table 2).

Table 2.

Methods and respective extensions used by the eligible studies. Main advantages and challenges of each method are provided

Generic method	Version	General description	Comparator	Advantages and disadvantages	Confounding
Continuous sequential—allow examination of the data as often as desired, the various versions are described later (SPRT and MaxSPRT)
Wald's SPRT	General description	This is the generic method proposed by Wald in the 1940s.	For vaccine safety, a Poisson model would typically be used with the observed count compared with a fixed expected count.9, 50	Advantage—Easy implementation of the Poisson model.	Covariate adjusted expected levels can be obtained to allow for possible confounding.9
				Disadvantage (compared with MaxSPRT)—Fixed single alternative hypothesis (e.g. RR = 3) whose choice will usually be arbitrary.50
MaxSPRT	General description	This generically describes all SPRT methods that have a composite alternative hypothesis (RR > 1).	Depends on the version of the test (refer to succeeding data).	No need to specify a single alternative.50	Depends on the version of the test (refer to succeeding data).
	Poisson	—	This implementation assumes a Poisson distribution for observed counts and compares to a fixed expected mean.50	Advantage—Simple to implement. The use of a fixed expected level increases power.10	Covariate adjusted expected levels can be obtained to allow for possible confounding. Potential for confounding due to seasonal or temporal changes in disease incidence or coding.10
				Disadvantage—Relies on accurate data for the expected level, which may not be the case if data are limited or only historical.51
	Binomial	—	Based on a binomial distribution events occurring among vaccine exposed individuals/periods versus comparison (unexposed individuals/periods).50	Advantage—Does not rely on a fixed expected value and can match on confounders or compare to other periods within individuals.50	Can be used in different versions—matching controls (fixed or flexible matching ratio—exact sequential analysis57) or self‐controlled design (SCCS or SCRI) or considering previous seasons, avoiding the healthy vaccinee effect (DID24).
				Disadvantage—Less powerful than Poisson unless multiple unvaccinated available per vaccinated.50
				The use of a self‐controlled design with post‐exposure comparison intervals might result in delays.46	Potential for confounding depends on the version of the test used.
	Conditional	—	Assumes a Poisson process for the cumulative person‐time to observe a number of adverse events.51	Advantage—Does not assume the expected number of cases is known (as the Poisson‐based MaxSPRT).	Same as Poisson
				Accounts for uncertainty in historical data.51
				Disadvantage—Assumes constant event rates are in historical and surveillance data.51
Group sequential testing	General description	Data are examined at discrete points in time.10	Several approaches used a group sequential way (PMaxSPRT, Abt's modification of SPRT, USPRT) often implementing an alpha‐spending approach (using a function to determine how to ‘spend’ the alpha in the different tests).12	Advantage—Requires less frequent updates.	Depends on the specific version used.
				Disadvantage—Continuous tests are more powerful.58 Less explored (compared with continuous tests) in the observational setting, including adjustment for confounders. More complex designs.12
Statistical process control	General description	Graphical approach where the number of events is compared with an upper limit (the threshold is typically—mean + a certain number of SD).56	Expected count.	Advantage—Easy to implement.	Stratification can be used to handle confounding.
				Disadvantage—Less methodological work on applications to vaccine safety.
				No formal way to control for multiple test.

AE, adverse event; DID, difference‐in‐difference; (P)MaxSPRT, (Poisson‐based) maximized probability ratio test; RR, relative risk; SCCS, self‐controlled case series; SCRI, self‐controlled risk interval; SD, standard deviation; SPRT, sequential probability ratio test; USPRT, updating sequential probability ratio test; UL, upper limit.

When considering specific versions of the tests available, the choice has been guided by the increasing availability of new methods and knowledge of these methods over time, as shown in Figure 2, as well as the frequency of AE studied. In VSD, the sequential probability ratio test (SPRT) was first applied9 being subsequently replaced by its maximized version (MaxSPRT) with the advantage of not having to specify a single alternative hypothesis.13 The use of MaxSPRT and its variations also evolved over time. While in the beginning the Poisson and binomial versions were simultaneously used for the same outcome,13 from 2010, a targeted selection of the test version and its extensions, based on the strengths of each method (Table 2) and the characteristics of the outcome under study, was preferred.24, 33, 34, 42, 43 In particular, Poisson‐based MaxSPRT (PMaxSPRT) has been used when less than 50 events were anticipated and the conditional version when the ratio of observed historical events to upper limit was ≤2.5. Outside VSD, a pattern in the use of continuous sequential methods was less clear. Overall, these tests were the most often employed—PMaxSPRT (45.7%),10, 50 followed by the binomial (BMaxSPRT—23.9%)10, 50 and conditional (10.9%) versions.51

More recently, four studies used group sequential testing. Two of these used an alpha‐spending approach,38, 39 (a function controlling how much of the alpha will be ‘spent’ every time a new analysis is run52), one the Updating Sequential Probability Ratio Test53 and other the Abt's modification of SPRT.54 An alpha‐spending approach was thus preferred over the two other tests employed in a group sequential way. Both the Pocock‐type and O'Brien–Fleming‐type functions have been used.12, 55 The remaining methods did not follow a clear evolution and include use of SPC56 at different times by two non‐USA institutions (New Zealand Ministry of Health, Health Protection Scotland).21, 44

Thirty‐seven of 49 analyses (75.5%) mentioned control for confounding. Strategies chosen were often design‐based and included (alone or in combination) the following: (i) using a self‐controlled design, which automatically addresses time‐invariant confounders; (ii) matching baseline confounders, through a concurrent comparator design; (iii) adjusting the expected rate obtained from a historical comparison group based on the confounders' distribution in the study cohort (iv) stratifying the results according to relevant confounder categories. Analyses adjusting for potential confounders used mainly an expected rate adjusted for potential confounders (51.4% of those adjusting), stratification (16.2%) or a combination of a self‐controlled design and stratification (13.5%). The choice of approaches also depended on the analytical method selected. For group sequential methods and SPC, strategies to deal with confounders were even more limited. When employing group sequential methods, only expected rate calculations based on the confounders' distribution and stratification were considered. For SPC, only stratification was used. Potential confounders considered include age, sex, geographic site, concomitant vaccine administration, season and trend (Table 1).

Some of the prospective studies considered data‐accrual lags in their analysis. Most often, the analysis was delayed by some weeks (n = 7). Others adjusted for partially elapsed risk intervals and delays in the arrival of inpatient data (n = 3).46 For studies using spontaneous report for the observed number of events (and EHR for the expected number of events), sensitivity analyses with several degrees of underreporting were conducted (n = 4).28, 31, 40 Updates to the previous datasets already analysed were not considered a specific strategy to adjust for data‐accrual lags as they would not reduce the time to signal. The majority of studies did not mention ways or did not adjust for data‐accrual lags (n = 11).

Discussion

Our comprehensive systematic review has identified an increasing number of studies and systems implementing NRTVSS. All the studies identified were performed in high‐income countries/regions with most in the USA. This might reflect limited capacity in many settings to provide registry data in a timely fashion and the infra‐structure required to set up the system.

A clear effort was put into using these methods to assess pandemic influenza vaccine safety. This vaccine is a good example of the importance of post‐licensure surveillance due to potential safety concerns.32 Meningococcal group B vaccine in New Zealand21 represents a similar situation, where NRTVSS, along with enhanced passive surveillance and other active methods, was implemented after the vaccine was approved without phase III trials. Other situations where these methods have been particularly useful include vaccines/AE of concern due to experiences with previous versions of the vaccine—for example, rotavirus/intussusception25 and influenza/GBS.32 For previously suspected AE, the set of methods here reviewed has the advantage of informing in a timely manner the existence of a safety concern or reassuring regulatory authorities and the public about vaccine safety.

In this review, we have identified different methods to perform NRTVSS using EHR and the way these have been applied, both by VSD and by other institutions. All the methods identified are derived from Wald's sequential test.50, 59, 60 When choosing a particular method, it is important to be aware of its properties. Properties of the continuous and group sequential methods have been studied in the context of drug safety.12 Group sequential methods were deemed to be more appropriate when data updates are less frequent,12 but more recent work comparing these methods has found that for any group sequential design, there is a better continuous method and recommended that the data are looked at as frequently as possible.58 After selecting the methodological approach, it is necessary to choose the specific test to employ. For example, using the PMaxSPRT and BMaxSPRT simultaneously might be a more robust approach owing to complementary strengths. However, as previously suggested, BMaxSPRT might fail to identify a signal when investigating very rare events. Hence, an alternative is to use PMaxSPRT when less than 50 events are anticipated and the conditional version when the ratio of observed historical events to upper limit is ≤2.5. The use of a targeted approach has been considered in VSD's more recent work.24, 33, 34, 42, 43

On the other hand, the properties of SPC‐based methods applied to vaccine safety have not been extensively studied. Both Kulldorff et al. 50 and Musonda et al. 61 have argued that SPC‐based methods such as cumulative sum are not appropriate to perform surveillance for newly introduced products as the aim is to detect a safety problem that is already present and not a sudden change. These authors defend the use of such methods in the context of surveillance for batch‐related problems (problems arising at the time of manufacture rather than related to the product itself). However, we should consider that at the time of introduction, if there is a safety problem with that specific vaccine and an appropriate comparison group is used, a sudden change would be observable as well. Given its ease of implantation, SPC is attractive, but recommendations on the use of SPC are deferred until further research on their properties is available.

Control for potential confounders has been limited in both the strategies employed and factors adjusted for. This observation is in agreement with Nelson et al.,12 who have argued for better methods for confounder adjustment, in particular at the analysis stage. Recent work has been performed in this area, adapting group sequential methods with regression adjustment and comparing this to existing approaches.62, 63 To the best of our knowledge, these promising approaches are still at the development stage and have not yet been applied to new studies. As pointed out by Yih,11 it might not be possible to adjust for all possible confounders in this setting, which can lead to spurious signals. However, it should be noted that, as a near real‐time analysis, aimed at quickly identifying/strengthening signals, priority is given to rapid results. As such, confounding adjustment is not deemed as critical—more complete analyses can be performed at confirmatory stages.11 These might include adjusting for additional confounders or a more detailed adjustment (e.g. using finer categorization of a variable) to avoid residual confounding. The specific confounders to adjust for should be decided on the basis of the vaccine, outcome and age groups studied. In addition to those factors considered by studies, adjustments for day‐of‐the‐week effects or co‐morbidities might be required.11 Nevertheless, 12 studies13, 24, 25, 26, 27, 29, 30, 35, 36 did not refer to potential confounding in at least one of the analyses reported in their published texts.

Best practice using EHR apply equally to NRTVSS as to any study using these kind of data. For example, Lanes et al. provide an approach to identify outcomes in healthcare databases.64 One of the aspects to consider while doing so is misclassification. In some occasions, manual review of individual medical records can be used, particularly if a signal is found. In this review, only two studies21, 35 performed this confirmation before running the NRTVSS analysis, as doing so might delay the surveillance process. Alternatively, multiple algorithms might be developed, providing a trade‐off between sensitivity and positive predictive values (PPV). In the NRTVSS, an algorithm with higher sensitivity and moderate PPV is generally considered to be timelier than algorithms with moderate sensitivity algorithm and high PPV. This should be considered for the specific outcome under study, its seriousness and the data available.65 Misclassification of the exposure might also be problematic. A possible approach is to restrict the analysis to vaccinated individuals, avoiding potential biases.11

A key aspect to consider while using these methods is the availability of timely data. ‘Real‐time’ analyses are difficult to achieve, and thus, the expression ‘near real‐time’ is preferred. In fact, delays can occur at various stages, including delays in diagnosis (e.g. for conditions with more insidious onset), recording (e.g. retrospective recording of vaccination administration or diagnosis), receiving the data for analysis (due to either incomplete data accrual or partially accrued risk windows) and reporting. The timeliness of data should thus be considered. Some studies have delayed the analysis for some weeks.13, 23, 25, 27, 41, 42, 43 While this approach gives time for data to accrue, it will not reduce the time to signal. The use of group sequential methods with less frequent testing portrays a similar situation where more time has been given for data to accrue.35, 38, 39 Nevertheless, for events occurring closer to the time of testing, data‐accrual lags may still be problematic. Finally, adjustments for partially elapsed risk interval and delays in the arrival of inpatient data have been proposed (through the expected number of events)46 or integrated in the critical limits calculation36. These can decrease the time to signal, based on previously observed data‐accrual patterns. They have been applied in a few, influenza vaccine, studies. Influenza vaccines pose particular challenges when using delayed data as failure to detect a signal before the season ends will impede adequate action. Strategies proposed so far do not specifically address delays between illness onset and diagnosis.

Only three of the 11 outcomes identified in the prospective studies were confirmed as true signals. In addition to issues already raised (confounding factors that have not been considered, misclassification of the outcome), unconfirmed signals were due to (i) changes in the true incidence or coding practices; (ii) inappropriate comparison groups; (iii) uncertainty in background rates; and (iv) type I errors.11, 33 For type I errors, additional strategies to reduce the false discovery rate are available at the planning stage: these include delaying the first test,66 requiring a minimum number of events to occur before rejecting the null hypothesis67 or, in the case of group sequential tests, selecting an O'Brien–Fleming threshold. The latter spends less alpha in earlier tests and was used by Nelson et al. 38 During the surveillance period, it is important to update the critical limits as data arrive, as the observed data might differ from those planned.66 As in the case of outcome identification, these considerations should be balanced against the importance of detecting signals in a timely manner. Even after careful consideration of all these aspects before and during surveillance, possible spurious signals may still arise. This emphasizes the need for a predetermined plan of action for signal refinement if a signal is found.11 The plan should include a careful decision on the data source to use to test the hypothesis in subsequent analyses if needed, owing to potential biases with the use of the same data to identify and test the signals. NRTVSS is thus not a stand‐alone method but part of the signal detection and evaluation process.

This review aimed at capturing studies and systems worldwide using EHR to perform NRTVSS. Our rigorous search strategy and further contacts with many experts on vaccine safety from different countries and institutions (with a satisfactory response rate, 70.6%) should have minimized the risk of missing systems currently in use. However, we cannot exclude the existence of similar systems elsewhere. Furthermore, some information was missing from the studies included, which we have tried to reduce by contacting the authors. The missing information most often related to confounding control strategies and the data‐accrual lag adjustment employed. This might reflect the limited options to address these issues, especially for the earlier studies.

Countries considering introduction of these methods should benefit from the work developed so far and from strategies under development. There should be a cautious reflection on the availability of timely data and their characteristics (including discussion with the data providers), the vaccine(s) and outcome(s) to be studied and the infra‐structure needed in case a signal is detected. Future directions for research might include further development and application of strategies for adjustment for confounding and data‐accrual lag, as well as consideration of other methods not yet applied to observational settings but in use in clinical trials, for example, Bayesian approaches to group sequential tests.68 Bayesian methods can incorporate previous information (such as the data generated by pre‐licensure studies) and potentially provide a more flexible approach.

In conclusion, NRTVSS using EHR to assess the safety of newly introduced vaccines is being increasingly used in the USA, with limited introduction in a few other countries. These methods ensure timely detection of safety signals. New methods have been integrated over time, but strategies to account for potential confounders and data‐accrual lags have received less attention. As new vaccines are expected to be introduced and the public questions vaccine safety, the demand for strong post‐licensure surveillance systems will increase.

Conflict of Interest

The authors declare no conflict of interest.

Key points.

• Near real‐time vaccine safety surveillance using electronic health records (EHR) is one of the options available to identify vaccine safety signals.

• Use of near real‐time vaccine safety surveillance using EHR has been increasing in the USA but to date has only been considered in a few other countries.

• Methods available have developed over time and have been integrated into systems using this kind of surveillance. Continuous sequential testing has been the preferred approach.

• Strategies to address potential confounding factors are currently limited, but further developments may address this in the near future.

• Timeliness and allowing for data‐accrual lag are important factors for consideration when implementing near real‐time surveillance using EHR. Lags have only been addressed in a few studies.

Ethics statement

The authors state that no ethical approval was needed.

Supporting information

Supporting info item

Leite, A. , Andrews, N. J. , and Thomas, S. L. (2016) Near real‐time vaccine safety surveillance using electronic health records—a systematic review of the application of statistical methods. Pharmacoepidemiol Drug Saf, 25: 225–237. doi: 10.1002/pds.3966.