Abstract
Immunizations are crucial to the prevention of disease, thus, having an accurate measure of vaccination status for a population is an important guide in targeting prevention efforts. In order to comprehensively assess the validity of self-reported adult vaccination status for the eight most common adult vaccines we conducted a survey of vaccination receipt and compared it to the electronic medical record (EMR), which was used as the criterion standard, in a population of community-dwelling patients in a large healthcare system. In addition, we assessed whether validity varied by demographic factors. The vaccines included: pneumococcal (PPSV), influenza (Flu), tetanus diphtheria (Td), tetanus diphtheria pertussis (Tdap), Human PapillomaVirus (HPV), hepatitis A (HepA), hepatitis B (HepB) and herpes zoster (shingles). Telephone surveys were conducted with 11,760 individuals, ≥ 18, half with documented receipt of vaccination and half without. We measured sensitivity, specificity, positive and negative predictive value, net bias and over- and under-reporting of vaccination. Variation was found across vaccines, however, sensitivity and specificity did not vary substantially by either age or race/ethnicity. Sensitivity ranged between 63% for HepA to over 90% (tetanus, HPV, shingles and Flu). Hispanics were 2.7 times more likely to claim receipt of vaccination compared to whites. For PPSV and Flu those 65+ had low specificity compared to patients of younger ages while those in the youngest age group had lowest specificity for HepA and HepB. In addition to racial/ethnic differences, over-reporting was more frequent in those retired and those with household income less than $75,000. Accurate information for vaccination surveillance is important to estimate progress toward vaccination coverage goals and ensure appropriate policy decisions and allocation of resources for public health. It was clear from our findings that EMR and self-report do not always agree. Finding approaches to improve both EMR data capture and patient awareness would be beneficial.
Keywords: Immunization, Vaccination surveillance, Self-report
1. Introduction
Because immunization is crucial to the prevention of disease, having an accurate measure of vaccination status for a population can serve as an important guide in targeting prevention efforts.[1,2] To monitor vaccination status, the United States conducts population-based vaccination coverage surveys, [3,4] however, obtaining accurate assessment is difficult. Most people have attended multiple medical practices, leaving records scattered or incomplete. Time may also result in lapses in memory [5–7]. Several vaccinations, such as tetanus/diphtheria (Td), pneumococcal polysaccharide vaccine (PPSV), hepatitis A (HepA) and hepatitis B (HepB) series, may have been administered years before a survey is conducted [6]. Also, patients may affirm receipt of vaccines they believe they should have obtained or deny obtaining a vaccination that might indicate risky behavior [8,9]. Lack of accurate data decreases the ability to interpret estimated coverage levels and may cause providers to miss opportunities to provide needed vaccines. Validity of self-report has been extensively studied for Influenza (Flu) and PPSV [9–15], but there is a paucity of literature on other vaccines (e.g., HepA, HepB) and relatively new vaccines such as Human Papillomavirus (HPV). Further, information on validity that is age and race/ethnicity specific has also had limited study [5,7,16]. In order to comprehensively assess the validity of self-reported adult vaccination status for the eight most common adult vaccines, we conducted a survey of vaccination receipt and compared it to the electronic medical record (EMR) in a population of community-dwelling patients in a large healthcare system. In addition, we assessed whether validity varied by demographic factors. The purpose of this paper is to report the concordance of data obtained through both methods of data collection.
2. Methods
2.1. Study setting and population
This study was conducted in an integrated health care delivery system with 21 primary care clinics, 30 specialty clinics and over 700 practicing physicians. The plan insures over one million people in an open-access system, allowing patients to obtain care within the medical group or the larger network. The vast majority of care is obtained within the network as nearly all services are covered. The majority of the population is white, employed, with education of high school or beyond. Eligible patients were 18 years or older as of January 1, 2007, and seen in one of the medical clinics during 2007.
Health plan Institutional Review Board (IRB) approval was granted for the conduct of this study.
The study was conducted in two phases. The first, was a retrospective review of the EMR data to determine documented receipt of vaccine. In the second phase, a telephone survey was conducted. Concordance of vaccination status between the EMR and self-report from survey results was assessed.
In Phase 1, EMR data were examined for patients seen in 2007. For HepA this timeframe was expanded back to 2001 to ensure adequate numbers of potential subjects. All vaccination information was obtained for each patient as far back as it was available. The denominator of those eligible for each vaccine was determined and a vaccine specific database was created. The eight vaccines studied included: PPSV, Flu, tetanus/diphtheria Td), tetanus/diphtheria/acellular pertussis: (Tdap), HPV, HepA, HepB and herpes zoster vaccine (shingles). Data were stratified by age group for most vaccines and by race/ethnicity for PPV and Flu.
Vaccination history was retrieved from information obtained from patients when they entered the health system, which was entered into the EMR as were vaccinations obtained within the system. A vaccination procedure code as well as the facility where it was obtained, lot number and vaccine manufacturer were considered evidence of vaccination. For each vaccine, the date of the most recent vaccination was recorded ensure we were using the most relevant information.
EMR data were sorted by vaccine and within each vaccine for the ages and racial/ethnic specific groups of interest to the Centers for Disease Control and Prevention (CDC). For surveys of PPSV, tetanus, HepA, and Flu vaccination, the age groups sampled were for 18–49 years, 50–64 and 65+; for shingles (50–64 and 65+); HepB (18–49 and 50–64) and HPV (females 18–26). Specific racial/ethnic groups (White, Black, Hispanic) were targeted for those 65+ for the PPSV vaccine and all three age categories for Flu. For each group we determined the underlying EMR vaccination rate. We then sorted based on documented evidence of receipt of vaccination: those with and without. After creating all age and race/ethnicity groups by receipt or no receipt of vaccine, there were a total of 56 sampling strata (Appendix A).
2.2. Patient survey
For Phase 2, the patient survey, we randomly selected 300 individuals from each strata, to ensure 200 completed surveys. Two weeks prior to initiating the telephone calls for a given strata, the EMR data was refreshed to be certain all vaccination information was current. Anyone with modified information was reclassified. Letters of invitation for participation were then sent to the 300 randomly selected individuals. There were up to 15 attempts to reach each patient by telephone. Surveys were conducted between January 2009 and March 2011. The intention was to ask any individual about just one vaccine. However, in order to fill some strata (i.e. Hispanic 65+, Black 65+) 339 subjects were surveyed for more than one vaccine.
2.3. Survey content
Surveys were created for each vaccine. Subjects were asked if they “ever” received the vaccine in question. Responses were “yes”, “no”, “don’t know”, and “refused to answer.” Follow-up questions varied by vaccine regarding how long ago the immunization was obtained. Subjects surveyed about tetanus vaccination were also asked if they were told if the vaccine contained pertussis (whooping cough). Specifics on all vaccines can be found in Table 2. The core content of all surveys was similar. Demographics included: age, sex, race/ethnicity, marital status, education, employment and annual household income. We asked whether reported information was based on recall or if participants had records of vaccination status. Patients with no EMR documentation, claiming to have been vaccinated, were asked if they had any evidence. Some immigrants (<5%) had vaccination history cards.
Table 2.
Agreement statistics and 95% confidence intervals for 7 common vaccinesa.
| PPV | Tetanus | HPV | Shingles | HepA | HepB | Flu | |
|---|---|---|---|---|---|---|---|
| Source population (N) | 195,792 | 233,113 | 22,466 | 98,719 | 15,804 | 40,042 | 200,818 |
| Coverage in source populationb (%) | 28.6 | 56.9 | 55.7 | 9.3 | 15.7 | 27.7 | 41.7 |
| Survey population, n | 1939 | 2549 | 405 | 846 | 1704 | 818 | 3499 |
| Self-reported vaccinationc (%) | 27.8 | 95.9 | 57.9 | 17.8 | 23.3 | 44.3 | 60.6 |
| Kappa | 0.65 (0.63–0.67) | 0.03 (0.03–0.04) | 0.67 (0.63–0.72) | 0.57 (0.54–0.60) | 0.39(0.37–0.41) | 0.33 (0.30–0.36) | 0.56 (0.54–0.57) |
| Sensitivityd | 73.8 (70.3–77.2) | 92.1 (90.4–93.8) | 91.2(87.3–95.1) | 90.7 (87.8–93.5) | 62.5 (59.0–66.0) | 72.6 (67.6–77.6) | 93.0(91.3–94.8) |
| Specificityd | 90.7 (88.4–93.0) | 11.0(9.0–13.0) | 76.1 (70.2–82.1) | 89.7 (86.7–92.6) | 84.0(81.6–86.4) | 66.6(62.0–71.1) | 65.7 (61.9–69.5) |
| Positive predictive valued | 76.0 (76.0–76.0) | 57.7(57.7–57.7) | 79.6 (79.6–79.6) | 47.4 (47.4–47.4) | 42.1 (42.1–42.1) | 45.5 (45.5–45.5) | 66.0 (66.0–66.0) |
| Negative predictive valued | 90.9 (90.9–90.9) | 55.0(55.0–55.0) | 92.5 (92.5–92.5) | 99.1 (99.1–99.1) | 94.0(94.0–94.0) | 90.6 (90.6–90.6) | 93.1 (93.1–93.1) |
| Net bias estimate | −0.8 | 39 | 2.2 | 8.5 | 7.6 | 16.6 | 18.9 |
| Net bias relative (%) | −2.9% | 68.5% | 3.9% | 47.8% | 48.4% | 59.9% | 45.3% |
For each vaccine, we asked if the participant knew which year he/she had received the last vaccination. Where exact year was not known, we prompted participants asking roughly how long ago they received the vaccination. Prompts varied by vaccine. For HepA and HepB, prompts were: 12 months, last 5 years, last 10 years, more than 10 years ago. For PPV and tetanus, prompts were: last 12 months, last 2–5 years, last 6–10 years, more than 10 years ago. For flu, we asked not only if they received the vaccination during the past year but if they could give the month they received the flu vaccination and for shingles and HPV we asked only the year of vaccination.
Coverage based on EMR evidence in source population.
Based on survey results.
Reported as percentage.
2.4. Statistical analysis
Demographic characteristics were reported overall and by self-reported vaccination status. In order to assess the validity of self-reported vaccination status, we measured sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). In addition, we assessed net bias and lack of concordance indicated by over-and under-reporting of vaccination.
We calculated agreement statistics sensitivity, specificity, positive predictive value, and negative predictive value.[17] Additionally, we calculated a Kappa statistic,[18] to measure agreement between self-report and EMR (agreement was categorized as follows: almost perfect 0.81–1.00, substantial agreement 0.61–.80, moderate agreement 0.41–0.60, fair 0.21–0.40, and poor <0.21). These validity parameters were calculated for all vaccines and for each vaccine separately. Clinically, if a patient can’t affirm that they have had the vaccine, a provider may offer the vaccine, thus we considered the lowest coverage scenario where all “don’t knows” for self-reported vaccination status were considered “no”.
Biased estimates can occur as a result of unequal sampling rates across strata. To correct for the unequal sampling rates of vaccinated and unvaccinated persons across sampling strata, all study data were weighted to reflect the actual distribution of EMR vaccination status among study-eligibles within each of the age and race/ethnicity-specific strata. Sampling weights were computed as the reciprocal of the achieved sampling fraction for each stratum. The weighted analysis results in numbers that sum to that of the original population. Statistical analyses of validity measures accounted for the complex sampling methods and weighting. All statistical analyses were conducted using SAS version 9.2 (SAS Institute, Inc., Cary, NC).
Net bias was calculated as the estimated vaccination coverage estimate (self-reported vaccination) minus the actual vaccination coverage estimate (EMR based). In order to compare vaccines with varying coverage levels, we also calculated net bias relative difference (estimated vaccination – actual vaccination)/actual vaccination) and multiplied by 100 to get percent. In addition, we calculated proportions under- and over-reporting (1-sensitivity and 1-specificity, respectively).
To test whether over- or under-reporting differed by patient characteristics (self-reported race/ethnicity, sex, age, marital status, employment status, education status, and income level) we conducted a series of logistic regression models using the entire survey sample for all eight vaccines to estimate odds ratios and 95% confidence intervals. For under-reporting, we restricted the analysis to those who had EMR documentation of vaccination and modeled the likelihood of self-report of no vaccination as a function of patient characteristics one at a time. Because 339 subjects (<3%) were surveyed for two vaccines and all others were surveyed for only vaccine, we conducted a sensitivity analysis to see if the assumption of independence was violated. The logistic regression models examining over- and under-reporting were run excluding the second observation recorded for each subject with more than one survey.
2.5. Additional follow up
To further assess the completeness of our data, we conducted a substudy. There were two groups where misclassification would be most likely to exist: those reporting having had the vaccination where we had no EMR record (the more obvious group) and those who had neither an EMR record nor self-report of vaccination. In the first group, we asked for any evidence of receipt of vaccination and if evidence was available, the patient was reclassified. For the latter group, we inquired if they might have any additional medical history elsewhere. We then asked for consent to allow us to check medical records outside our health care system. Resources were available to conduct this additional check for unrecorded vaccination on 330 patients. We were able to identify and follow up on 279 (85%) who met the required eligibility (no record in EMR, self-report of no vaccination, attended a clinic outside of those within the health system and gave permission to contact the outside clinic).
3. Results
The goal of 11,200 completed surveys was based on 200 each from the 56 strata. We ended with 11,760 completed surveys: 10,670 toward goal and 1090 over. The overage occurred due to an initial flaw in the tracking program affecting all HepA (overage = 713). Overage was also due to multiple interviewers continuing to survey until informed that the goal was achieved. For non-Hispanic blacks 65+ and Hispanics with no evidence of PPSV or Flu vaccination, numbers in the base population were insufficient to achieve our goal. Thus, although our intention was to survey each individual about only one vaccine, for certain strata we used an individual more than once. There were 339 (<3%) individuals who were surveyed about more than one vaccine. Over half of these were Hispanic.
Table 1 presents the demographic characteristics of the study sample. Because certain ages and race/ethnicities were dictated by the study, the balance is as expected. The fields dictated by the study protocol are shaded. The other variables are reflective of the health plan’s population.
Table 1.
Demographic characteristics of study sample.
| Overall | PPV | Tetanus | HPV | Shingles | HepA | HepB | Flu | |
|---|---|---|---|---|---|---|---|---|
| N | 11,760 | 1939 | 2549 | 405 | 846 | 1704 | 818 | 3499 |
| Male (%) | 38 | 37 | 41 | – | 40 | 41 | 40 | 40 |
| Age (%) | ||||||||
| 18–49 | 33 | 21a | 34a | 100a | 33a | 49a | 36a | |
| 50–65 | 34 | 21a | 34a | – | 50a | 32a | 51a | 37a |
| 65+ | 33 | 58a | 32a | 50a | 35a | – | 27a | |
| Race/ethnicity | ||||||||
| Non-Hispanic white | 68 | 59a | 87 | 77 | 95 | 85 | 83 | 37a |
| Non-Hispanic black | 15 | 20a | 6 | 5 | 2 | 6 | 8 | 30a |
| Hispanic | 11 | 16a | 0 | 0 | 0 | 0 | 0 | 29a |
| Other | 16 | 4a | 8 | 13 | 3 | 9 | 9 | 4a |
| Educational attainment | ||||||||
| HS grad or less | 27 | 38 | 23 | 15 | 21 | 21 | 17 | 33 |
| Some college or college grad | 52 | 44 | 56 | 75 | 50 | 52 | 58 | 49 |
| More than college | 20 | 17 | 21 | 10 | 29 | 27 | 24 | 16 |
| No data | 1 | <1 | <1 | <1 | <1 | <1 | <1 | 2 |
| Employment status | ||||||||
| Working/homemaker/student | 57 | 44 | 62 | 92 | 29 | 53 | 72 | 61 |
| Unable to work, can’t find work | 9 | 9 | 7 | 7 | 4 | 10 | 14 | 11 |
| Retired | 33 | 46 | 30 | 0 | 67 | 37 | 15 | 27 |
| No work data | <1 | <1 | <1 | <1 | <1 | <1 | <1 | 1 |
| Marital status | ||||||||
| Married/couple | 58 | 53 | 63 | 29 | 65 | 61 | 63 | 57 |
| No longer married | 25 | 34 | 22 | 1 | 30 | 25 | 15 | 26 |
| Single | 16 | 12 | 14 | 70 | 5 | 13 | 21 | 16 |
| No data | 1 | <1 | 1 | 0 | <1 | <1 | <1 | 2 |
| Household income | ||||||||
| <$75,000 | 53 | 58 | 47 | 62 | 53 | 51 | 48 | 56 |
| ≥$75000 | 25 | 19 | 30 | 15 | 22 | 31 | 35 | 22 |
| No data | 22 | 23 | 23 | 24 | 25 | 19 | 17 | 22 |
Indicates targeted sampling with regard to strata.
3.1. Agreement statistics
The sensitivity of self-reported vaccination status for true vaccination status, based on EMR data, ranged between 63% (HepA) and over 90% (Td, Tdap, HPV, shingles and Flu). Specificity varied widely from 11% for Tdap and Td to 91% for PPSV. PPV varied from below 50% for Shingles, HepA and HepB to 80% for HPV. NPV was over 90% for all vaccines except tetanus (55%). When asked if their tetanus shot included the pertussis vaccine, 10% of those who claimed to receive tetanus vaccine said they received the pertussis vaccine, 11% said no, 17% said the provider did not say, and the remaining 61% did not know.
EMR vaccine coverage based on the source population is shown in Table 2. Table 2 also presents net bias and net bias relative difference. Net bias estimates ranged from −0.8 to 39.0 for PPSV and tetanus, respectively. Values for net bias relative difference ranged considerably across vaccines; from low (PPSV and HPV) to moderate (HepA, Flu) to high (tetanus, shingles and HepB).
Table 3 reports the agreement per vaccine for race/ethnicity and age strata. With the exception of PPSV, sensitivity did not vary considerably by age and/or race/ethnicity within vaccine strata. With regard to specificity, for PPSV, and Flu, those over 65 had low specificity compared to patients of younger ages. In contrast for HepA and HepB, those in the youngest group had the lowest specificity. Low specificity was also found with Hispanics for Flu vaccine. With some exceptions, NPV generally was high; however, for PPSV those over 65 had low NPV as did tetanus for all ages. PPV varied by age for PPSV, with those 65+ with the highest values and for shingles, those 50–64 were highest.
Table 3.
Agreement statistics and 95% confidence intervals by targeted strata (reported as percentage).
| Vaccine | n | Kappa | 95% confidence interval |
Sensitivitya | 95% confidence interval |
Specificitya | 95% confidence interval |
PPVa | 95% confidence interval |
NPVa | 95% confidence interval |
|---|---|---|---|---|---|---|---|---|---|---|---|
| PPV | |||||||||||
| Race/ethnicity | |||||||||||
| Non-Hispanic white | 1155 | 0.67 | 0.65–0.70 | 76.3 | 72.6–80.0 | 90.7 | 88.3–93.1 | 77.4 | 77.4–77.4 | 91.7 | 91.7–91.7 |
| Non-Hispanic black | 431 | 0.43 | 0.38–0.48 | 56.9 | 46.0–67.9 | 85.3 | 70.8–99.9 | 60.6 | 60.6–60.6 | 84.4 | 84.4–84.4 |
| Hispanic | 315 | 0.53 | 0.47–0.58 | 46.5 | 28.5–64.5 | 97.8 | 97.5–98.0 | 85.9 | 85.9–85.9 | 88.2 | 88.2–88.2 |
| Age | |||||||||||
| 18–49 | 404 | 0.59 | 0.54–0.64 | 63.2 | 56.5–69.9 | 96.1 | 93.4–98.8 | 62.8 | 62.8–62.8 | 96.4 | 96.4–96.4 |
| 50–65 | 413 | 0.55 | 0.51–0.60 | 72.0 | 65.8–78.1 | 84.5 | 79.5–89.5 | 67.6 | 67.6–67.6 | 89.4 | 89.4–89.4 |
| 65+ | 1122 | 0.24 | 0.21–0.26 | 78.6 | 73.4–83.6 | 48.9 | 42.7–55.1 | 87.1 | 87.1–87.1 | 38.8 | 38.8–38.8 |
| Tetanus | |||||||||||
| Age | |||||||||||
| 18–49 | 439 | 0.04 | 0.02–0.05 | 91.8 | 89.2–94.4 | 11.4 | 8.4–14.4 | 57.2 | 57.2–57.2 | 47.3 | 47.3–47.3 |
| 50–65 | 406 | 0.02 | 0.01–0.03 | 93.2 | 90.7–95.7 | 8.5 | 5.8–11.1 | 62.8 | 62.8–62.8 | 58.8 | 58.8–58.8 |
| 65+ | 404 | 0.04 | 0.03–0.05 | 91.1 | 87.8–94.5 | 12.8 | 8.8–16.7 | 50.0 | 50.0–50.0 | 70.3 | 70.3–70.3 |
| Shingles | |||||||||||
| Age | |||||||||||
| 50–65 | 419 | 0.49 | 0.44–0.54 | 88.7 | 84.3–93.1 | 90.7 | 86.8–94.6 | 79.6 | 79.6–79.6 | 99.3 | 99.3–99.3 |
| 65+ | 427 | 0.64 | 0.59–0.68 | 92.1 | 88.3–95.8 | 87.6 | 83.2–91.9 | 57.0 | 57.0–57.0 | 98.5 | 98.5–98.5 |
| HepA | |||||||||||
| Age | |||||||||||
| 18–49 | 555 | 0.35 | 0.31–0.39 | 67.7 | 61.9–73.6 | 78.2 | 73.5–82.8 | 38.4 | 38.4–38.4 | 92.2 | 92.2–92.2 |
| 50–65 | 456 | 0.43 | 0.39–0.47 | 65.2 | 59.4–71.1 | 83.8 | 79.5–88.1 | 47.9 | 47.9–47.9 | 94.3 | 94.3–94.3 |
| 65+ | 603 | 0.34 | 0.30–0.38 | 46.5 | 40.6–52.3 | 90.9 | 87.7–94.1 | 37.9 | 37.9–37.9 | 94.8 | 94.8–94.8 |
| HepB | |||||||||||
| Age | |||||||||||
| 18–49 | 400 | 0.26 | 0.22–0.30 | 70.0 | 63.6–76.4 | 58.5 | 51.6–65.4 | 49.1 | 49.1–49.1 | 82.3 | 82.3–82.3 |
| 50–65 | 418 | 0.38 | 0.34–0.43 | 80.8 | 75.5–86.2 | 75.0 | 69.0–81.0 | 37.9 | 37.9–37.9 | 96.1 | 96.1–96.1 |
| Flu | |||||||||||
| Race/ethnicity | |||||||||||
| Non-Hispanic white | 1376 | 0.57 | 0.54–0.59 | 94.0 | 92.1–96.0 | 65.1 | 60.7–69.6 | 67.4 | 67.4–67.4 | 93.7 | 93.7–93.7 |
| Non-Hispanic black | 1089 | 0.54 | 0.51–0.57 | 82.7 | 79.2–86.3 | 72.1 | 67.2–77.0 | 58.7 | 58.7–58.7 | 89.8 | 89.8–89.8 |
| Hispanic | 1000 | 0.38 | 0.35–0.41 | 89.8 | 87.2–92.4 | 55.4 | 51.1–59.6 | 50.4 | 50.4–50.4 | 91.7 | 91.7–91.7 |
| Age | |||||||||||
| 18–49 | 1246 | 0.52 | 0.50–0.55 | 90.1 | 86.9–93.4 | 68.1 | 62.7–73.4 | 57.2 | 57.2–57.2 | 93.9 | 93.9–93.9 |
| 50–65 | 1308 | 0.62 | 0.59–0.65 | 94.8 | 92.3–97.3 | 66.8 | 61.2–72.4 | 74.7 | 74.7–74.7 | 92.5 | 92.5–92.5 |
| 65+ | 945 | 0.49 | 0.45–0.52 | 95.8 | 93.2–98.4 | 49.5 | 43.1–56.0 | 72.9 | 72.9–72.9 | 89.3 | 89.3–89.3 |
Reported as percentage.
In order to examine differences in over- and under-reporting as a function of patient characteristics combined across all vaccines, we conducted logistic regression adjusted for age and sex (Table 4). Compared with those ages 18–49, those 50–64 had a slightly lower odds of under-reporting. Black subjects and those reporting other race/ethnicity had 3-fold higher odds of over-reporting, but no association was observed with under-reporting. Hispanic subjects had a 4-fold greater odds of over-reporting and had 60% lower odds of under-reporting compared with whites. Retired subjects had higher odds of over-reporting and lower odds of under-reporting compared to those working/homemaker/or student. Those with household incomes below $75,000 were more likely to over-report and less likely to under report compared with those making $75,000 or more. Compared with those who reported some college, those with a high school diploma or less were 2.3-fold more likely to over-report. We did not observe strong associations of over- or under-reporting by marital status.
Table 4.
Odds ratios and 95% confidence intervals in the relationship of patient characteristics with over-reporting and under-reporting.
| Over-reportinga,c N = 6074 | Under-reportingb,c n=5686 | |
|---|---|---|
| Gender | ||
| Female | 1.0 REF | 1.00 REF |
| Male | 1.56 (1.22–2.01) | 1.16 (0.99–1.36) |
| Age(%) | ||
| 18–49 | 1.00 REF | 1.00 REF |
| 50–64 | 0.85 (0.64–1.14) | 0.61 (0.54–0.70) |
| 65+ | 1.10 (0.83–1.47) | 1.13 (0.99–1.30) |
| Race/ethnicity | ||
| Non-Hispanic white | 1.00 REF | 1.00 REF |
| Non-Hispanic black | 3.37 (2.26–5.01) | 0.92 (0.70–1.21) |
| Hispanic | 4.00 (2.53–6.32) | 0.40 (0.21–0.74) |
| Other | 2.90 (1.80–14.69) | 1.67 (1.21–2.30) |
| Educational attainment | ||
| HS grad or less | 2.28 (1.68–3.09) | 0.92 (0.76–1.12) |
| Some college or college grad | 1.0 REF | 1.00 REF |
| More than college | 0.83 (0.60–1.15) | 0.98 (0.80–1.20) |
| No data | 1.61 (0.32–8.19) | 2.17 (0.88–5.40) |
| Employment status | ||
| Working/homemaker/student | 1.00 REF | 1.00 REF |
| Unable to work, can’t find work | 2.10 (1.39–3.19) | 1.04 (0.78–1.38) |
| Retired | 1.17 (0.78–1.75) | 0.65 (0.51–0.82) |
| No work data | 0.68 (0.12–3.91) | 2.23 (0.75–6.59) |
| Marital status | ||
| Married/couple | 1.0 REF | 1.00 REF |
| No longer married | 1.40 (1.00–1.94) | 0.71 (0.48–1.06) |
| Single | 1.42 (0.99–2.04) | 0.73 (0.47–1.14) |
| No data | 1.93 (0.56–6.64) | 0.44 (0.11–1.72) |
| Household income | ||
| <$75,000 | 1.74 (1.28–2.38) | 0.80 (0.66–0.96) |
| ≥$75,000 | 1.00 REF | 1.00 REF |
| No data | 2.09 (1.44–3.01) | 0.85 (0.68–1.07) |
Models restricted to those with EMR documentation of vaccination.
Models restricted to those with no EMR documentation of vaccination.
All models adjusted for sex and age.
In addition, we conducted a sensitivity analysis to see if over-or under-reporting differed when we excluded the second observation for the subjects who were surveyed for more than one vaccine. For this series of analyses, the odds ratios for each demographic characteristic were similar except for gender, where in the sensitivity analysis for over-reporting males had higher odds of over-reporting compared to females.
3.1.1. Additional follow up patients
For the 279 patients who had no record in EMR of vaccination, self-report of no vaccination, attended a clinic outside the system and gave permission to contact the outside clinic, we were able to obtain information on 246 (88%). Of the 246 total patients with information obtained outside of our system, 8 were found to have received vaccinations (3%). Three of the 8 vaccinations were for PPSV, two for shingles and one each for Tdap, Td and HPV.
3.2. Sensitivity analyses
While we report the results of the lowest coverage scenario by counting “don’t know” responses as “no”, we conducted additional sensitivity analyses (data not shown). We compared the effects on validity measures when all “don’t know” responses were considered “yes” and found sensitivity improved slightly and specificity measures decreased slightly for all vaccines except Flu, for which measures of sensitivity and specificity were virtually the same.
4. Discussion
The purpose of this study was to examine self-reported vaccination status compared to EMR data. Considerable variation was found by vaccine, age and race/ethnicity. We also assessed rates of under-and over-reporting and net bias. Under-reporting was relatively low, except in the Hispanic strata, while over-reporting varied by vaccine. The net bias relative difference also varied widely. We found more favorable agreement statistics for PPSV and HPV. Tetanus had the largest net bias relative difference.
Findings on sensitivity, specificity, PPV and NPV, were consistent with the literature [9–15]. Flu studies have reported higher sensitivity with lower specificity [9–11,13]. In a recent study, however, examining the accuracy of Flu vaccination from four Vaccine Safety Datalink sites, sensitivity ranged from 0.51–0.89, highest in those ages 65–79 and those at high risk [8]. While our findings for those age 65+ were also highest (96% versus 89% for age 18–49), specificity was the lowest of all age groups (50% versus 68% for those 18–49). Findings for PPSV vaccination have been reported as more variable, perhaps due to more distant recall [14,19].
The self-report accuracy of HepB vaccination has been poorly studied. One study using serological data found self-report to be unreliable [20]. However, others examining self-report of Flu, tetanus booster, and HepB found patient questionnaire nearly as sensitive and specific (hepatitis B: sensitivity 80%, specificity 100%) as the medical record [12]. Our findings were a bit lower for HepB specificity (sensitivity 0.73, specificity 0.67). HepB is administered primarily for employment or as a travel vaccine with a number of other vaccines, and no boosters, so recall may be difficult.
Inaccurate patient reporting has implications for both clinical care and estimates of vaccination coverage [1,2,8,21]. A positive net bias in vaccine coverage level estimates may indicate better progress toward disease control and achievement of national and local health objectives than is true.
Our patients were not asked to distinguish between Td/Tdap so we collapsed them for examination; the reason for the low specificity is unclear. Perhaps the 10-year duration between vaccinations forTd and newness of Tdap is part of the explanation. If a tetanus vaccination was given several years ago it is possible it was administered before the patient entered our healthcare system and they did not report it upon intake into the system.
4.1. Limitations
This study has several potential limitations. First it was conducted in a single health plan so findings may not be representative of other populations. Further, survey participants may differ from non-participants, although we did select a random sample from each strata. For some vaccines, the EMR may not always have gone back as far as needed, leading to some misclassification. In addition, it is possible that patients may have obtained vaccinations outside of the health system, particularly for the Flu vaccination which is commonly available in work and retail settings [22,23]. If patients did not report obtaining these to a provider, they would be missed. Our overall findings for the shingles vaccine should be taken with caution, as we surveyed ages 50 and over and the Adsvisory Committee for Immunization Practices recommendation for obtaining this vaccine starts at age 60 [24]. Thus, because of differences in study population, survey question wording, recall period and other methodological differences, results from this study may not be comparable to self-reported data from other surveys such as the National Health Interview Survey [25].
Despite its limitations, the assessment of the accuracy of self-report using telephone survey for each of the vaccines studied is a major strength of the study. The population was large, with demographic data and robust vaccination histories. This enabled us to examine self-report by age and race/ethnicity. The comprehensive medical record allowed the team to readily identify the underlying populations of interest, obtain documented vaccination data on all and compare self-report to EMR. The presence of vaccination in the EMR provided strong evidence that vaccination had occurred, although lack of documentation did not guarantee that no vaccination had been obtained. Few studies have tested the accuracy of telephone survey self-report for many of the vaccines studied. While Flu and PPSV have had extensive study, limited data exists on the newer shingles or the HPV vaccinations. Our study adds to our understanding of well-reported vaccination coverage as well as reporting on lesser-studied vaccinations.
Accurate information for vaccination surveillance is important to be able to estimate progress toward Healthy People 2020 vaccination coverage objectives and to ensure appropriate policy decisions and allocation of resources for public health. It was clear from our findings that EMR and self-report do not always agree. Finding approaches to improve both EMR data capture and patient awareness would be beneficial. The EMR may not always be accurate if vaccination occurred in the past. It is also possible that vaccines obtained outside the system are not getting captured. With growing number of pharmacies offering vaccinations, health system records may become less trustworthy. According to Sy, if there is no record in data base, there is a 20% chance that vaccination was obtained in a different setting [2]. Assisting patients to improve recall would also be advantageous. Like others, we found higher rates of over-reporting compared to under-reporting [5]. With today’s technology it is possible to have patients record their vaccinations into personal devices and electronic applications. Information from such applications could be part of inquiry at office visits, similar to updating address and telephone number information. The portability of the EMR and immunization registries could also facilitate improved accuracy of vaccination status.
5. Conclusions and future directions
Health system electronic databases provide ready capture of data on vaccinations, yet potential for misclassification cannot be ignored. As more vaccines are available in non-traditional settings, this problem is likely to become more common. We must continue to find ways for both health plans and patients to capture and communicate information on vaccination status to ensure surveillance efforts are as robust as possible.
Acknowledgments
The authors wish to thank Colleen King, Manager, and David Butani, Research Study Coordinator, from the HealthPartners Institute for Education and Research Data Collection Center, who trained and monitored the numerous interviewers and oversaw the data collection for over 11,000 surveys. We also thank the Centers for Disease Control and Prevention (CDC) for funding of the study and James Singleton for his editorial comments. The authors wish to acknowledge Sandy De Quesada, BA, HealthPartners Institute for Education and Research, for her assistance in the preparation of this manuscript. The findings and conclusions in this paper are those of the authors and do not necessarily represent the views of the CDC.
Appendix A. Vaccine target groups and eligibility status
| Vaccine | Age groups | Race/ethnicity groups | Additional eligibility criteria |
|---|---|---|---|
| Pneumococcal (PPV) | 65+ | White Black Hispanic |
|
| 50–64 | All races | Forages 18–64: patients with risk factors: American Indian or Alaska Native; history of diabetes, lung disease, heart disease, kidney disease, liver disease, anemia spleen (including sickle-cell disease), cancer, immunodeficiency (HIV/AIDS), organ transplant, rheumatologic diseases (treatment involved taking steroids), alcoholism, splenectomy, spinal fluid leak | |
| 18–49 | |||
| Influenza (LAIV/TIV) | 65+ 50–64 18–49 |
White Black Hispanic |
|
| Tetanus (Td) Tetanus (Tdap) |
65+ 50–64 18–49 |
All races | |
| HPV | 18–26 | All races | Females only |
| Hepatitis A | 65+ | In addition to those seen in 2007, we included those who had a diagnosis of liver disease or Hemophilia or took a Hepatitis A laboratory test at one of owned clinics during 2001–2007. | |
| 50–64 | All races | ||
| 18–49 | |||
| Hepatitis B | 50–64 | All races | Patients with risk factors which include: Asian or Pacific Islander ethnicity, history of a sexually-transmitted disease, kidney disease, liver disease, hemophilia, Immunodeficiency (including HIV/AIDS). |
| 18–49 | |||
| Shingles | 65+ 50–64 |
All races |
Contributor Information
E.D. Parker, Email: emily.d.parker@healthpartners.com.
J.D. Nordin, Email: jim.d.nordin@healthpartners.com.
B.D. Hedblom, Email: brita.d.hedblom@healthpartners.com.
F. Wei, Email: fwei@uams.edu.
T. Kerby, Email: tessa.j.kerby@healthpartners.com.
J.M. Jackson, Email: jody.m.jackson@healthpartners.com.
A.L Crain, Email: lauren.a.crain@healthpartners.com.
G. Euler, Email: cdceuler1@comcast.net.
References
- 1.Ozasa K. The effect of misclassification on evaluating the effectiveness of influenza vaccines. Vaccine. 2008;26:6462–5. doi: 10.1016/j.vaccine.2008.06.039. [DOI] [PubMed] [Google Scholar]
- 2.Sy LS, Liu IL, Solano Z, Cheetham TC, Lugg MM, Greene SK, et al. Accuracy of influenza vaccination status in a computer-based immunization tracking system of a managed care organization. Vaccine. 2010;28:5254–9. doi: 10.1016/j.vaccine.2010.05.061. [DOI] [PubMed] [Google Scholar]
- 3.Centers for Disease C. Prevention. Adult vaccination coverage – United States, 2010. MMWR Morbidity and mortality weekly report. 2012;61:66–72. [PubMed] [Google Scholar]
- 4.Flu Vaccination Coverage. United States, 2011–12 Influenza Season. Seasonal Influenza: Centers for Disease Control and Prevention; 2012. [Google Scholar]
- 5.Gordon NP, Wortley PM, Singleton JA, Lin TY, Bardenheier BH. Race/ethnicity and validity of self-reported pneumococcal vaccination. BMC Public Health. 2008;8:227. doi: 10.1186/1471-2458-8-227. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Shenson D, Dimartino D, Bolen J, Campbell M, Lu PJ, Singleton JA. Validation of self-reported pneumococcal vaccination in behavioral risk factor surveillance surveys: experience from the sickness prevention achieved through regional collaboration (SPARC) program. Vaccine. 2005;23:1015–20. doi: 10.1016/j.vaccine.2004.07.039. [DOI] [PubMed] [Google Scholar]
- 7.Singleton JA, Santibanez TA, Wortley PM. Influenza and pneumococcal vaccination of adults aged > or = 65: racial/ethnic differences. Am J Prev Med. 2005;29:412–20. doi: 10.1016/j.amepre.2005.08.012. [DOI] [PubMed] [Google Scholar]
- 8.Greene SK, Shi P, Dutta-Linn MM, Shoup JA, Hinrichsen VL, Ray P, et al. Accuracy of data on influenza vaccination status at four Vaccine Safety Datalink sites. Am J Prev Med. 2009;37:552–5. doi: 10.1016/j.amepre.2009.08.022. [DOI] [PubMed] [Google Scholar]
- 9.Mac Donald R, Baken L, Nelson A, Nichol KL. Validation of self-report of influenza and pneumococcal vaccination status in elderly outpatients. Am J Prev Med. 1999;16:173–7. doi: 10.1016/s0749-3797(98)00159-7. [DOI] [PubMed] [Google Scholar]
- 10.Andrews RM. Assessment of vaccine coverage following the introduction of a publicly funded pneumococcal vaccine program for the elderly in Victoria, Australia. Vaccine. 2005;23:2756–61. doi: 10.1016/j.vaccine.2004.11.039. [DOI] [PubMed] [Google Scholar]
- 11.Martin LM, Leff M, Calonge N, Garrett C, Nelson DE. Validation of self-reported chronic conditions and health services in a managed care population. Am J Prev Med. 2000;18:215–8. doi: 10.1016/s0749-3797(99)00158-0. [DOI] [PubMed] [Google Scholar]
- 12.Stange KC, Zyzanski SJ, Smith TF, Kelly R, Langa DM, Flocke SA, et al. How valid are medical records and patient questionnaires for physician profiling and health services research? A comparison with direct observation of patients visits. Med Care. 1998;36:851–67. doi: 10.1097/00005650-199806000-00009. [DOI] [PubMed] [Google Scholar]
- 13.Skull SA, Andrews RM, Byrnes GB, Kelly HA, Nolan TM, Brown GV, et al. Validity of self-reported influenza and pneumococcal vaccination status among a cohort of hospitalized elderly inpatients. Vaccine. 2007;25:4775–83. doi: 10.1016/j.vaccine.2007.04.015. [DOI] [PubMed] [Google Scholar]
- 14.Mangtani P, Shah A, Roberts JA. Validation of influenza and pneumococcal vaccine status in adults based on self-report. Epidemiol Infect. 2007;135:139–43. doi: 10.1017/S0950268806006479. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Zimmerman RK, Raymund M, Janosky JE, Nowalk MP, Fine MJ. Sensitivity and specificity of patient self-report of influenza and pneumococcal polysaccharide vaccinations among elderly outpatients in diverse patient care strata. Vaccine. 2003;21:1486–91. doi: 10.1016/s0264-410x(02)00700-4. [DOI] [PubMed] [Google Scholar]
- 16.Hebert PL, Frick KD, Kane RL, McBean AM. The causes of racial and ethnic differences in influenza vaccination rates among elderly Medicare beneficiaries. Health Serv Res. 2005;40:517–37. doi: 10.1111/j.1475-6773.2005.00370.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Gordis L. Epidemiology. Third. Philadelphia, PA: Elsevier Saunders; 2004. [Google Scholar]
- 18.Norman GR, Streiner DL. Biostatistics: The Bare Essentials. 2nd. Hamilton: B. C. Decker; 2000. [Google Scholar]
- 19.Mac Donald R. In response to the April 1999 article entitled: “validation of self report of influenza and pneumococcal vaccination status in elderly outpatients”. Am J Prev Med. 1999;17:249. [PubMed] [Google Scholar]
- 20.Trevisan A, Frasson C, Morandin M, Beggio M, Bruno A, Davanzo E, et al. Immunity against infectious diseases: predictive value of self-reported history of vaccination and disease. Infect Control Hosp Epidemiol. 2007;28:564–9. doi: 10.1086/516657. [DOI] [PubMed] [Google Scholar]
- 21.Placzek H, Madoff LC. The use of immunization registry-based data in vaccine effectiveness studies. Vaccine. 2011;29:399–411. doi: 10.1016/j.vaccine.2010.11.007. [DOI] [PubMed] [Google Scholar]
- 22.Singleton JA, Poel AJ, Lu PJ, Nichol KL, Iwane MK. Where adults reported receiving influenza vaccination in the United States. Am J Infect Control. 2005;33:563–70. doi: 10.1016/j.ajic.2005.03.016. [DOI] [PubMed] [Google Scholar]
- 23.Lee BY, Mehrotra A, Burns RM, Harris KM. Alternative vaccination locations: who uses them and can they increase flu vaccination rates. Vaccine. 2009;27:4252–6. doi: 10.1016/j.vaccine.2009.04.055. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Harpaz R, Ortega-Sanchez IR, Seward JF, Advisory Committee on Immunization Practices Centers for Disease C, prevention Prevention of herpes zoster: recommendations of the Advisory Committee on Immunization Practices (ACIP) MMWR Recomm Rep. 2008;57:1–30. quiz CE2-4. [PubMed] [Google Scholar]
- 25.Lu PJ, Dorell C, Yankey D, Santibanez TA, Singleton JA. A comparison of parent and provider reported influenza vaccination status of adolescents. Vaccine. 2012;30:3278–85. doi: 10.1016/j.vaccine.2012.03.015. [DOI] [PubMed] [Google Scholar]