Assessing misclassification of vaccination status: Implications for studies of the safety of the childhood immunization schedule

Abstract

Background:

To address public concern about the safety of the childhood immunization schedule, the Institute of Medicine recommended observational studies comparing adverse health outcomes of fully vaccinated children to children under-vaccinated due to parental choice. Misclassification of vaccination status could bias such studies.

Objective:

To assess risk of misclassification of vaccination status within the Vaccine Safety Datalink (VSD).

Design/methods:

A retrospective cohort study was conducted in three phases. In phase 1, electronic health record (EHR) data were used to identify patterns of under-vaccination during the first 24 months of life potentially due to parental choice. In phase 2, a random sample of records of under-vaccinated children was manually reviewed. In phase 3, a separate sample of parents were surveyed to assess whether EHR data accurately reflected their child’s vaccination status. Phases 1 and 2 were conducted at 6 VSD sites, phase 3 at 1 site.

Results:

The study cohort included 361,901 children born 2004 through 2012. By 24 months of age, 198,249 (54.8%) were fully vaccinated with no delays, 84,698 (23.4%) experienced delays but were fully vaccinated by 24 months of age, 4865 (1.3%) received no vaccines, 3789 (1.0%) delayed starting vaccination until ≤4 months of age, 4781 (1.3%) had consistent vaccine-limiting (≤2 vaccines per visit), and the remaining 65,519 (18.1%) were missing vaccine series or doses. When a diagnosis code for vaccine refusal was present in EHR data, encounter notes confirmed vaccine refusal as the reason for under-vaccination for nearly 100% of sampled records. Parent surveys confirmed these findings. Parents of under-vaccinated children were more likely to report visiting an alternative medical provider than parents of fully vaccinated children.

Conclusions:

Specific groups of children, under-vaccinated due to parental choice, can be identified with relatively low likelihood of misclassification of vaccination status using EHR-based vaccine data and diagnosis codes.

1. Introduction

Vaccination is regarded as one of the greatest public health achievements of the past century [1], and vaccination coverage for young children in the U.S. remains high relative to historical benchmarks [2]. However, survey data indicate that more than 10% of parents have intentionally refused or delayed vaccines for their children, with vaccine safety reported as a primary concern [3–6]. Some parents have questioned the safety of the immunization schedule as a whole, expressing the opinion that children receive too many vaccines at too young an age, and that early childhood immunization ‘‘overwhelms” the immune system [7–9].

In response to these concerns, in 2012 an Institute of Medicine (IOM) committee reviewed scientific evidence regarding the safety of the recommended childhood immunization schedule, and concluded that available evidence strongly supported the safety of the schedule [10]. The committee also identified limitations with existing safety data, asserting that ‘‘most vaccine-related research focuses on the outcomes of single immunizations or combinations of vaccines administered at a single visit,” and consequently ‘‘key elements of the entire schedule-the number, frequency, timing, order, and age at administration of vaccines-have not been systematically examined in research studies” [10]. The committee advocated for new observational studies of the safety of the schedule, and suggested that the Vaccine Safety Datalink (VSD) project [11,12] was an important resource for conducting such studies.

To evaluate the safety of the schedule, the IOM committee recommended comparing adverse health outcomes between fully vaccinated children, completely unvaccinated children, and those on a delayed or alternative schedule [10]. Using observational data to make these comparisons creates significant methodological challenges [10,13,14], in part because health- and health care-related behaviors may differ in systematic ways between parents of fully vaccinated children and those under-vaccinated due to parental choice [15,16]. In addition, vaccination status can be misclassified; in the VSD, this occurs when children who appear under-vaccinated within VSD data have actually received vaccines elsewhere. As detailed in a VSD White Paper, such misclassification could bias studies of the safety of the schedule [17].

The objective of the current investigation was to determine the degree of misclassification of vaccination status within the VSD, specifically in the context of designing future studies of the safety of the recommended childhood immunization schedule. This process included: (1) developing an algorithm to identify specific patterns of under-vaccination that were likely due to parental choice; (2) conducting a manual review of electronic health record (EHR) data to verify vaccination status and reasons for under-vaccination; and (3) surveying a sample of parents of under-vaccinated and fully vaccinated children to assess vaccination status, reasons for under-vaccination when present, and reported receipt of vaccines and health care elsewhere than at their VSD site.

2. Methods

2.1. Study setting

A retrospective cohort study was conducted in the VSD, a collaboration between the Centers for Disease Control and Prevention (CDC) and 9 large medical care organizations (referred to as ‘‘sites”) from across the U.S [11,12,18]. Six VSD sites participated: Group Health Cooperative; Marshfield Clinic; Kaiser Permanente (KP) Northwest; KP Northern California; KP Southern California; and KP Colorado.

This investigation was accomplished in three phases. In phase 1, vaccination and diagnosis data from the EHR were used to identify patterns of under-vaccination potentially due to parental choice. In phase 2, manual medical record review was performed on a sample of records within each pattern of under-vaccination. In phase 3, a survey was conducted among a separate sample of parents to assess whether vaccine data in the EHR accurately reflected a child’s true vaccination status. Phases 1 and 2 were conducted at six VSD sites; phase 3 was conducted at KP Colorado. The study was approved by institutional review boards at all study sites. Written consent was not required for survey administration, and parents could opt out of the survey verbally or in writing.

2.2. Study population

We identified all children within participating VSD sites born January 1, 2004 through December 31, 2012. Within this cohort, we evaluated all vaccines received in the first 24 months of life. We required children to have continuous health insurance enrollment in their respective VSD site from 6 weeks of age through their second birthday. We excluded 91 children (0.02%) with contraindications to vaccination, 288 (0.08%) with obvious vaccine data errors (e.g. vaccine dates prior to date of birth), 2305 (0.63%) with an uncertain vaccine type, and 1150 (0.31%) with vaccines not routinely given under 2 years of age. Four VSD sites (Group Health Cooperative, Marshfield Clinic, KP Northwest, KP Colorado) had access to vaccine data from their statewide immunization information system (IIS) [19], while two VSD sites (KP Northern California, KP Southern California) did not. For VSD sites with IIS linkages, an estimated 1% or less of all vaccines were identified in IIS but not internal site data.

2.3. Phase 1: Identifying under-vaccination due to parental choice

In this phase, EHR vaccination and diagnosis data were used to identify specific patterns of under-vaccination likely due to parental choice. First, an algorithm developed by Luman [20] and modified by Glanz [15] was used to calculate the average days under-vaccinated (ADU) for each child in the study population. The algorithm assesses all vaccines routinely recommended during the first 24 months of life by the Advisory Committee on Immunization Practices (ACIP) [21–23] except influenza and hepatitis A vaccines, and incorporates information on minimum ages, minimum intervals between doses, different dose requirements based on different vaccine products, national vaccine shortages, and changes in vaccination recommendations over time (all historical U.S. immunization schedules are available at https://www.cdc.gov/vaccines/schedules/past.html). Children with ADU = 0 had received all recommended vaccines with no delays.

Next, we examined vaccination patterns for all children with ADU ≥ 1 day. Based upon prior work [15,24] as well as published ‘‘alternative” vaccination schedules [25], we grouped the observed vaccination patterns into one of six hierarchical, mutually exclusive categories of under-vaccination: (1) no vaccines (completely unvaccinated); (2) first vaccine at ≥4 months of age; (3) consistent vaccine-limiting, defined as 2 or fewer vaccines per visit at 2 or more vaccine visits within the first year of life [24]; (4) any vaccine series not received, including not receiving measles-mumps-rubella vaccine or varicella vaccine; (5) vaccine doses not received; and (6) fully vaccinated by 24 months of age, but with some vaccines delayed prior to 24 months. In this context, delay was defined as receiving one or more vaccines ≥30 days after the recommended age of administration [21,22,26].

We also assessed the use, at any time between 3 days and 24 months of age, of diagnosis codes indicating ‘‘vaccination not carried out because of caregiver/patient refusal” (International Classification of Diseases, 9th Revision, Clinical Modification codes V64.05 and V64.06). In earlier work these codes had high specificity but poor sensitivity as a marker for under-vaccination due to parental choice [15]. We also examined codes for preventive pediatric health care (i.e. well-child care) visits (V20.1, V20.2, V70.x), under the hypothesis that parents of children receiving well-child care but limited or no vaccines were more likely declining vaccination as opposed to encountering barriers to care.

2.4. Phase 2: Manual review of electronic health records

Using trained record abstractors, we conducted a manual review of EHR encounter notes for a sample of children within each category of under-vaccination. The goal of the manual record review was to assess whether under-vaccination was present, and if so whether EHR encounter notes confirmed that under-vaccination was due to parental choice, rather than due to other reasons (such as vaccines having been given elsewhere, vaccines being contraindicated, or vaccination being temporarily deferred due to a child being sick during a routine visit). A stratified random sample of records was selected for review, with sampling stratified by VSD site and category of under-vaccination. A total of 429 records were reviewed.

2.5. Phase 3: Survey of parents

At KP Colorado, we conducted a survey of parents of under-vaccinated and fully vaccinated children. To develop a population for survey administration, we identified children 9 through 24 months of age with continuous health insurance enrollment, and used the methods described above to calculate ADUs and assign categories of under-vaccination. We surveyed all 329 parents of children with under-vaccination, vaccine delay, vaccine refusal, or missed well-child care, and a random sample of 71 parents of fully vaccinated children, for a total survey sample of 400.

The survey was used to assess whether the child’s vaccination status (under-vaccinated or fully vaccinated) matched EHR vaccine data, and if under-vaccinated, whether the child was under-vaccinated due to parental choice. The survey also assessed whether the child had received vaccines or other health care outside the VSD site. The survey instrument was pilot-tested via cognitive interviews with 10 parents, and revised accordingly.

Surveys were administered by postal mail with up to three attempts, with a reminder postcard between the 2nd and 3rd mailing, and with two automated telephone call reminders after the 3rd mailing. For non-respondents following mail and telephone contact, two reminders were sent by email (email addresses were available for 80% of subjects). Finally, for subjects without an email address, two additional personal telephone calls were made. The survey was fielded November 1, 2015 through April 15, 2016.

2.6. Analytic methods

For manual record review, the primary outcome was whether record review confirmed under-vaccination due to parental choice. Because of the sampling approach, the probability of a record being selected varied across VSD sites, based upon the number of records selected compared to the overall size of the under-vaccination pattern at that site. For each stratum of under-vaccination, a weighted confirmation rate was calculated, with records weighted by the inverse probability of selection. Because subjects were clustered by VSD site, this design effect was specified in the calculation of Clopper-Pearson 95% exact confidence intervals (CIs) [27].

Regarding the survey, respondents were compared to non-respondents using Student’s t-tests. Among respondents, survey answers were compared between parents whose children were fully vaccinated and those under-vaccinated, with and without a diagnosis code for parental vaccine refusal. Exact Clopper-Pearson 95% CIs were calculated within each vaccination stratum [27]. No adjustment was made to account for survey non-response. Analyses were conducted using SAS 9.4 (SAS Institute, Cary, NC).

3. Results

3.1. Patterns of under-vaccination identified

The study population consisted of 361,901 children born during January 1, 2004 through December 31, 2012. At 24 months of age, 198,249 (54.8%) were fully vaccinated with no vaccination delays, and 84,698 (23.4%) had experienced some delays but were fully vaccinated by 24 months of age (i.e. they had ‘‘caught up”). As shown in Table 1, the remainder were categorized into specific patterns of under-vaccination, including 4865 (1.3%) who had received no vaccines. The degree of under-vaccination, as measured by ADU, declined in a step-wise fashion for each of the hierarchal categories of under-vaccination presented in Table 1. Among under-vaccinated children, the use of a diagnosis code for parent vaccine refusal was common, particularly among children with no vaccines, those whose first vaccine was at ≥4 months of age, and those with consistent vaccine-limiting.

Table 1.

Number and percent of children within specific hierarchical patterns of vaccination in the first 24 months of life, Vaccine Safety Datalink, 2004–2014,^a n = 361,901.

			Average days under-vaccinated (ADU)d
Vaccination patternb	n (% of total cohort)	Diagnosis codec for vaccine refusal,%	Mean (SD)	Median (IQR)
No vaccines (completely unvaccinated)	4865 (1.3)	51.7	484.0 (13.7)	479.1 (479.1–479.1)
First vaccine ≥4 months of age	3789 (1.0)	45.3	360.4 (131.0)	414.5 (256.5–479.1)
Consistent vaccine-limiting ( ≤2 vaccines per visit)	4781 (1.3)	53.5	239.8 (126.8)	222.8 (142.9–343.8)
Vaccine series not receivede	28,112 (7.8)	18.3	121.2 (122.5)	74.1 (30.3–157.1)
Vaccine doses not received	37,407 (10.3)	6.6	68.3 (69.0)	43.3 (19.9–88.4)
Fully vaccinated by age 24 months, but with delayf	84,698 (23.4)	5.5	17.5 (22.3)	10.4 (4.1–20.9)
Fully vaccinated by age 24 months, with no delayf	198,249 (54.8)	5.1	0.0 (0.0)	0.0 (0.0–0.0)

Abbreviations: ADU, average number of days under-vaccinated; IQR, inter-quartile range; ICD-9-CM, International Classification of Diseases, 9th Revision, Clinical Modification; SD, standard deviation.

^aChildren born between January 1, 2004 and December 31, 2012.

^bPatterns or categories of vaccination were mutually exclusive and hierarchical, with vaccination status assessed at 24 months of age.

^cICD-9-CM codes V64.05 and V65.06.

^dCalculated as the total number of days each vaccine dose was delayed beyond the recommended age divided by the total number of vaccines recommended, assessed at 730 days of age.

^eIncludes, in order of frequency, no rotavirus vaccines (n = 14,478), no varicella vaccine (n = 12,967), no measles-mumps-rubella vaccine (n = 11,636), all other series (n = 5032); totals more than 28,112 because children may have been missing more than one series.

^fDelay defined as receiving one or more vaccines ≤30 days after the recommended age of administration.

3.2. Manual review of electronic health records

A total of 429 records were reviewed. Table 2 presents record review findings, stratified by presence of a diagnosis code for vaccine refusal and use of well-child care, for 3 patterns of under-vaccination (no vaccines; first vaccine ≥4 months of age; consistent vaccine-limiting).When a diagnosis code for vaccine refusal was present, EHR encounter notes confirmed vaccine refusal as the reason for under-vaccination for 100% of sampled records within each pattern. When a diagnosis code was absent, but the child had one or more well-child visits, vaccine refusal was confirmed in the majority (72.0–94.7%) of sampled records. However, when a diagnosis code was absent, and the child did not have any well-child visits, encounter notes were less likely to confirm vaccine refusal as the reason for under-vaccination. As shown in the last column of Table 2, the percent of reviewed medical records with evidence that vaccines had been received elsewhere (i.e. outside of their VSD site) ranged from 0.0% to 24.0% across different strata of under-vaccination.

Table 2.

Medical record confirmation of specific patterns of under-vaccination due to parental choice, stratified by presence of a diagnosis code for vaccine refusal and use of well-child care, Vaccine Safety Datalink, 2004–2014.^a

Vaccination patternb	Diagnosis code for vaccine refusalc	One or more well-child visits	Total number observed within VSD cohort	Total number of medical records sampled	Vaccine refusal confirmed in medical record,% (95% CI)d	Vaccines received elsewhere, per medical record,% (95% CI)d
No vaccines (completely unvaccinated)	Yes	NAe	2514	37	100.0 (90.5–100.0)f	0.0 (0.0–9.5)f
	No	Yes	903	68	72.0 (58.3–83.2)	12.6 (5.8–22.9)
	No	No	1448	34	5.8 (2.8–36.3)	0.0 (0.0–10.3)f
First vaccine ≥4 months of age	Yes	NAe	1718	9	100.0 (66.4–100.0)f	0.0 (0.0–33.6)f
	No	Yes	1100	25	92.4 (65.7–99.8)	24.0 (9.4–45.2)
	No	No	971	33	53.6 (23.6–81.8)	9.7 (1.8–27.1)
Consistent vaccine-limiting (2 or fewer vaccines per visit)	Yes	NAe	2559	20	100.0 (83.2–100.0)f	10.9 (0.4–44.3)
	No	Yes	2185	53	94.7 (69.2–100.0)	5.8 (0.0–36.9)
	No	No	37	0	Not reviewedg	Not reviewedg

Abbreviations: CI, confidence interval; ICD-9-CM, International Classification of Diseases, 9th Revision, Clinical Modification; NA, not applicable; VSD, Vaccine Safety Datalink.

^aChildren born between January 1, 2004 and December 31, 2012.

^bThese patterns or categories of vaccination are mutually exclusive and hierarchical, with vaccination status assessed at 24 months of age.

^cICD-9-CM codes V64.05 and V65.06, which indicate ‘‘vaccination not carried out because of caregiver/patient refusal”.

^dFor each stratum of under-vaccination, a weighted confirmation rate was calculated, with records weighted by the inverse probability of selection; subjects were clustered by VSD site, and this design effect was specified in the calculation of Clopper-Pearson 95% exact confidence intervals.

^eNoted as not applicable, because virtually all individuals with a diagnosis code for vaccine refusal had one or more well-child visits.

^fClopper-Pearson 95% exact confidence intervals; observations not weighted because all (i.e. 100%) or none (i.e. 0%) confirmed across sampled medical records.

^gNo manual medical record reviews were performed, because of the rarity of this vaccination pattern.

A sample of medical records were also reviewed for children missing vaccine series, and for those missing vaccine doses but not entire series. When a diagnosis code for vaccine refusal was present, encounter notes confirmed parental refusal as the reason for under-vaccination in 100% (95% CI 87.2–100.0%) of sampled records for children missing vaccine series, and for 99.3% (95% CI 85.9–100.0%) among children missing vaccine doses.

In sampled records, it was uncommon to find documentation of contraindications to vaccination or care outside of the VSD site. Combining results across all patterns of under-vaccination, children had a true contraindication to some or all vaccines in 0.1% (95% CI 0.0–0.3%) of sampled records. For 13.8% (95% CI 5.9–27.5%) of sampled records, encounter notes indicated that one or more vaccines had been temporarily deferred due to the child being sick on a day vaccines were due (which was not considered intentional under-vaccination due to parental choice). Finally, for 2.4% (95% CI 0.0–5.7%) of sampled records, encounter notes indicated that the patient had received routine non-urgent healthcare outside of their VSD site.

3.3. Survey of parents

Surveys were administered to 400 parents, of whom 185 (46.2%) responded. The survey response rate was lower among parents of under-vaccinated children (102 of 246 [41.1%] responded) than fully vaccinated children (83 of 154 [53.9%] responded). Comparing survey respondents to non-respondents, the age of their child was similar (mean age in months 14.7 among respondents versus 15.0 among non-respondents, p = 0.33), as was the total number of outpatient visits per child prior to the survey period (mean visits 9.9 versus 9.0, p = 0.17). However, survey respondents’ children had more well-child visits between birth and the survey period than non-respondents (mean well-child visits 4.0 versus 3.5, p < 0.01).

As shown in Table 3, survey responses differed between parents of fully vaccinated children and those who were under-vaccinated, particularly based on whether a diagnosis code for vaccine refusal was present or absent in the child’s medical record. For under-vaccinated children with a diagnosis code present, 85% of parents confirmed that they had refused or delayed vaccines, and none reported having received vaccines someplace other than KP Colorado. In contrast, for under-vaccinated children with no diagnosis code, 24.4% reported refusing and 54.9% reported delaying vaccines, and 6.1% reported having received vaccines someplace other than KP Colorado. With respect to seeking health care outside KP Colorado, 35.0% of parents of under-vaccinated children with a diagnosis code present reported having taken their child to an alternative medical provider.

Table 3.

Survey responses of parents of fully vaccinated and under-vaccinated children at Kaiser Permanente Colorado, n = 185.^a

Survey response	Fully vaccinatedb n = 83	Under-vaccinated, with diagnosis code for vaccine refusalb n = 20	Under-vaccinated, no diagnosis code for vaccine refusalb n = 82
Reported having decided not to get a vaccine for child	5.3 (1.3–13.7)	85.0 (62.1–96.8)	24.4 (15.6–35.1)
Reported having delayed getting a vaccine for child	15.9 (7.7–27.84)	85.0 (62.1–96.8)	54.9 (43.5–65.9)
Reported having received any vaccines some place other than at Kaiser Colorado	7.6 (2.5–17.1)	0.0 (0.0–16.8)	6.1 (2.0–13.7)
Reported having additional health insurance other than with Kaiser Colorado	10.6 (3.9–21.8)	5.0 (0.1–24.9)	4.9 (1.3–12.0)
Taken child for care some place other than Kaiser Colorado	19.4 (10.3–31.6)	10.0 (1.2–31.7)	29.3 (19.7–40.4)
Taken child to an alternative medical provider (such as a chiropractor, naturalist, homeopath, or acupuncturist)	9.2 (3.2–19.5)	35.0 (15.4–59.2)	14.6 (7.8–24.2)
Reported that if had urgent health care need, would take child to Kaiser Colorado	78.4 (64.9–88.5)	75.0 (50.9–91.3)	75.6 (64.9–84.4)

^aSurveyed parents had children that were between 9 and 24 months of age, with age and vaccination status determined as of November 1, 2015.

^bNumbers represent percentages, with 95% exact Clopper-Pearson confidence intervals.

4. Discussion

In this investigation we utilized EHR data, manual record review, and parent surveys to quantify misclassification of vaccination status for children who appeared under-vaccinated due to parental choice. We found that vaccine and diagnosis data could be used to identify specific cohorts of children (e.g. fully vaccinated; completely unvaccinated; consistent vaccine-limiting) with relatively low likelihood of misclassification of their vaccination status. As recommended by the IOM [10], adverse child health outcomes such as allergic or autoimmune diseases could be compared between these cohorts.

However, it is important to recognize that misclassification of vaccination status is only one of many potential sources of bias in observational studies of the safety of the childhood immunization schedule [10,13]. Observational studies of the schedule are also susceptible to confounding, because children under-vaccinated due to parental choice may differ from fully vaccinated children by characteristics not routinely captured in EHR data [16]. Such unmeasured factors that could be associated with vaccination status and a health outcome of interest include parental education, household income, family medical history, and extent of breast-feeding. Consequently, observational studies of the safety of the schedule will need to explore established (e.g. propensity scores, stratification, and matching) and novel methods to address confounding, even if misclassification of vaccination status is minimized by design.

When we observed specific patterns of under-vaccination in combination with a diagnosis code for vaccine refusal, vaccination status was confirmed by medical record review and parent report nearly 100% of the time. Nonetheless, because the diagnosis code was not used consistently, and these patterns were uncommon, only 1.9% of children (6791 of 361,901) in the VSD population met these criteria. A safety study using a cohort this size may be adequately powered to examine common conditions (e.g. asthma [17,28]) but not rarer conditions (e.g. acute demyelinating events [17,29]). Even when a diagnosis code for vaccine refusal was absent, children with delayed start to vaccination or consistent vaccine-limiting who were receiving well-child care were likely under-vaccinated due to parental choice. Representing 0.9% of children (3285 of 361,901) in the VSD, these children could also be included in future vaccine safety studies with relatively low likelihood of misclassification of their vaccination status.

In safety monitoring systems such as the VSD, EHR-based vaccine and diagnosis information provide the foundation for vaccine safety studies [11]. Diagnoses made by providers working outside the system are unlikely to be evident within VSD data, unless a claim was generated or clinical information was transferred electronically. Therefore, conditions such as environmental allergies or developmental disorders may be difficult to study within the VSD, because diagnoses may be made by alternative medical providers or outside the health care system entirely. In contrast, conditions such as type 1 diabetes or acute demyelinating events may be less prone to this type of information bias, under the assumption that parents of fully vaccinated and under-vaccinated children would be equally likely to present for care within their VSD site for these types of conditions.

In this study we focused primarily on under-vaccination due to parental choice, not under-vaccination due to barriers to care. This was a deliberate decision, based on recommendations from the IOM committee [10] and VSD White Paper [17]. Children completely unvaccinated or consistently vaccine-limiting due to parental choice represent well-defined cohorts with clearly defined vaccination patterns. In contrast, in children under-vaccinated due to barriers, hundreds of different patterns of vaccination can be observed [15]. This heterogeneity of vaccination patterns would create significant methodological challenges when designing studies of the safety of the immunization schedule.

This investigation has important limitations. The algorithm used to identify under-vaccination was not able to incorporate factors, such as local vaccine shortages, that may have contributed to under-vaccination. The EHR review and parent survey focused primarily on under-vaccination due to parental choice, rather than other causes of under-vaccination. Additionally, because reasons for under-vaccination can be missing in EHR encounter notes, medical record abstractors were not always able to differentiate between under-vaccination due to parental choice and under-vaccination due to other reasons. Despite intensive survey administration methods, with multiple surveys sent by various modalities over a 5-month period, the survey response rate was only 46%, with a lower response rate among parents of under-vaccinated children. This survey response was not surprising, given that survey response rates have generally fallen over time [30], and parents who refuse vaccines may be less willing to engage the health care system by completing a survey [16]. Consequently, response bias could have affected survey results. Finally, it is possible some parents could not accurately recall whether or not their child had been vaccinated or been seen for care outside their VSD site.

In summary, we found that specific groups of children, under-vaccinated due to parental choice, could be identified with relatively low likelihood of misclassification of vaccination status using EHR-based vaccine and diagnosis data. As recommended by a recent IOM report [10], such groups could be used in observational studies to evaluate the safety of the recommended childhood immunization schedule. However, even when misclassification of vaccination status can be minimized, other threats to internal validity will need to be carefully considered in such studies.

Abbreviations:

ACIP

Advisory Committee on Immunization Practices

ADU

average days under-vaccinated

CI

confidence interval

IIS

immunization information system

KP

Kaiser Permanente

VSD

Vaccine Safety Datalink