Addressing Immunization Registry Population Inflation in Adolescent Immunization Rates

Abstract

Objective

While U.S. adolescent immunization rates are available annually at national and state levels, finding pockets of need may require county or sub-county information. Immunization information systems (IISs) are one tool for assessing local immunization rates. However, the presence of IIS records dating back to early childhood and challenges in capturing mobility out of IIS areas typically leads to denominator inflation. We examined the feasibility of weighting adolescent immunization records by length of time since last report to produce more accurate county adolescent counts and immunization rates.

Methods

We compared weighted and unweighted adolescent denominators from the Oregon ALERT IIS, along with county-level Census Bureau estimates, with school enrollment counts from Oregon's annual review of seventh-grade school immunization compliance for public and private schools. Adolescent immunization rates calculated using weighted data, for the state as a whole, were also checked against comparable National Immunization Survey (NIS) rates.

Results

Weighting individual records by the length of time since last activity substantially improved the fit of IIS data to county populations for adolescents. A nonlinear logarithmic (ogive) weight produced the best fit to the school count data of all examined estimates. Overall, the ogive weighted results matched NIS adolescent rates for Oregon.

Conclusion

The problem of mobility-inflated counts of teenagers can be addressed by weighting individual records based on time since last immunization. Well-populated IISs can rely on their own data to produce adolescent immunization rates and find pockets of need.

Immunization of children, including teenagers, against vaccine-preventable diseases is among the most important public health activities in the United States.^1,2 However, maintaining adequate levels of protection herd immunity levels against diseases such as measles and pertussis is a constant challenge, as witnessed by recent pertussis outbreaks in the U.S.³ Improving immunization rates on a local level is a key need in public health.⁴ The expansion of immunizations beyond early childhood and into adolescence has added a number of challenges to this public health need.⁵ Switching between providers and health plans along with geographic mobility also leads to immunization records that are scattered across multiple clinics and even states.^6,7 Consolidating records, providing accurate clinical decision support, performing surveillance of population immunization levels, and finding geographic pockets of need are essential functions of immunization registries, also called immunization information systems (IISs). For these reasons, the use of IISs is a supported public health intervention according to the U.S. Community Services Task Force.^8,9

A feature of U.S. IISs is that they are located at state or local levels rather than regional or national levels. While a few IISs have limited data exchange agreements with their neighbors, such as between Oregon and Washington, on the whole, barriers such as incompatible data systems, hard-to-reconcile local IISs enabling legislation, and lack of funding have left most IISs isolated. Thus, IISs are useful for consolidating scattered records from providers within a state or local area but are challenged by mobility in and out of their local area. This challenge from potential mobility also carries over to the local use of IISs in assessing immunization levels or conducting reminder-recall interventions.¹⁰

IISs began with a focus on supporting providers in making immunization decisions for infants.¹¹ However, by the mid-1990s, it was recognized that IISs could also support efforts to find immunization pockets of need and more efficiently direct public health resources.¹² Subsequent Centers for Disease Control and Prevention (CDC) funding of state and local IISs was accompanied by initiatives for their use in calculating clinic and population immunization rates.¹³ Immunization delivery for children falls into three general epochs: early delivery around the well-baby visits recommended by 2 years of age, school entry requirements for primary school attendance, and teenage-recommended and school-required immunizations. The majority of immunizations are given by 2 years of age, and population immunization rates at this age have received the most attention, although systems such as the Healthcare Effectiveness Data and Information Set or the National Immunization Survey (NIS) have added teenage assessment components to reflect the increasing span of immunization.¹⁴

Starting with birth records from vital records, birth hospital reports, and other early sources, IIS populations continue to grow by adding more children over time as a consequence of mobility into the IIS area. Success in populating IISs, however, is balanced by the challenge that mobility out of the area is commonly unobserved or unreported.¹⁰ Children who leave the area may still be included in rate denominators, although their immunizations will no longer be reported. Thus, IIS population counts are at risk of ending up as a cumulative history of all the children who have ever lived in an area, rather than of the current population. This cumulative history effect adds an upward bias to population counts and a downward bias to calculated immunization rates when IIS-based denominators are used. Less-than-perfect levels of population capture can mask this population inflation factor to some extent. Over time, the effects of mobility add up; by teenage years, it is common for well-populated IISs to have rate denominators that exceed population counts. As an example of this denominator inflation, from 2012 annual state and local U.S. IIS reporting activity to CDC in 2012, the median ratio of IIS teenage population counts, irrespective of immunizations levels, to Census-estimated populations was 117%. This percentage implies that the majority of IISs contain substantially more teenagers than likely exist in their areas.¹⁵

While there is a public health need for local area assessments of immunization rates and disease vulnerability such as IISs can produce, to date, the literature provides little guidance on how to account for potential mobility issues. Standards from the American Immunization Registry Association Modeling of Immunization Registry Operations Workgroup (AIRA-MIROW) specify excluding children from rate calculations if ≥10 years have passed since any activity on their record.¹⁶ Recent research has also noted that different approaches to defining IIS populations can lead to substantial variance in county-level estimates of teen populations.¹⁷

The purpose of this study was to assess how different methods perform in estimating local teen populations in light of unseen mobility. Regardless of the overall fit of the different methods to local data, another question we examined was how well each method explains patterns of county-to-county variation. Along with Census Bureau county estimates and AIRA-MIROW 10-year exclusions, two methods based on weighting individual records by the amount of time since their last reported immunization are also presented. The time-weighting approach is based on the premise that those with more time since last activity are more likely to have unobserved mobility than those with more recent activity. Results from all methods are compared on a county level with Oregon's Annual School Review of compliance with school immunization requirements. Oregon students not in compliance with school immunizations or who do not have an exemption are excluded as of February 19 of each year. This review counts all seventh-grade students, both in public and private settings, and also captures many teenagers in nontraditional educational settings. The Census Bureau relies on the U.S. Department of Education's Common Core Dataset (CCD) for school count data to validate and correct their estimation process.¹⁸ For Oregon, the Annual Review is similar to the CCD but covers more educational settings and is verified by external review.

METHODS

The population for this study was drawn from the Oregon ALERT Immunization Information System (ALERT) for Oregon teenagers aged 13 years as of April 30, 2013. The target assessment date of April 30 was chosen so that the age cohort in ALERT would closely match the seventh-grade school cohort. ALERT is a well-populated IIS, receiving immunization records from 89% of Oregon providers who immunize patients of all ages and 95% of childhood providers, as well as from most payors and health plans. ALERT is initially populated from Oregon birth records and hospital reports and contains demographic information for children even if no immunizations were given. County of residence was determined by last reported ALERT ZIP Code. Length of time since last report was determined as the length of time in days from assessment date to the child's last reported immunization. Last reported immunization date was chosen rather than any type of reporting date, as health plans and large clinic data systems occasionally backload older demographic files into ALERT along with newer records.

We based the use of Annual School Review as a referent in this study rather than local Census estimates on the principle that administrative count data, when drawn from a whole population, is preferable to estimates. This preference for administrative count data is reflected in the Census practice of using CCD counts, adjusted for private schools not counted in the CCD, for validating and correcting their own estimates.¹⁸ Other reasons included the known undercounting of children in the Census, reports that Census estimates for two of the school-aged cohorts in Oregon differ substantially from the CCD data, and prior Census reports that showed issues in matching their estimates to school district populations with a reported mean percentage error of 15% in one study.^18–20

Estimates of county 13-year-old populations were made using five methods. The first method used Census Bureau yearly population estimates. Census Bureau-estimated county populations of 13-year-olds are not available on a yearly basis as an individual age band, so they were imputed from county 10- to 14-year age bands available as of May 2013. The second estimate of county-level 13-year-old populations was based on raw counts of all Oregon 13-year-olds in ALERT as of the assessment date. The third set of estimations was extracted from ALERT based on current AIRA-MIROW recommendations of excluding any record for rate calculations that has not had activity for the last 10 years. In the case of 13-year-olds, the application of the AIRA-MIROW recommendation implied excluding those with no immunizations reported after age 2 years.

The fourth and fifth estimation types were based on weighting individual records in ALERT by the amount of time since last reported immunization. For the fourth type, this weight was applied linearly based on the number of days between the assessment date and last reported immunization. The linear weight varied from 1 at 0 days to 0 at 10 years. The fifth estimation type was similar to the aforementioned except that a nonlinear weight was applied. For this study, we applied a logarithmic ogive weighting function. The formula for this weight was:

$${\text{w}}_{\text{i}}={\text{e}}_{\text{i}}^{\text{L}}/\left(1+{\text{e}}_{\text{i}}^{\text{L}}\right)$$

where

$${\text{L}}_{\text{i}}=\text{A}*\left[\text{B}-\left(\text{assessment}\hspace{0.17em}\text{date}-\text{last}\hspace{0.17em}\text{immunization}\hspace{0.17em}{\text{date}}_{\text{i}}\right)/365)\right]$$

In this equation, w_i is the weight to apply to the i^th individual in the IIS, and the parameters A and B modify the shape of the curve. For comparability in testing to the linear form, in this study the parameters A and B were a priori set at A = 0.75 and B = 6.00. These a priori choices for the A and B parameters were set so that the chances of including a teen without an immunization report for five years would be the same for both the ogive and linear models at probability equal to one-half.

We evaluated each denominator estimate against the total of school-reported seventh-grade students on a county level. While there are many possible methods for calculating goodness-of-fit indices, there is little substantive difference for models that are approximately true.²¹ Here, we calculated a normed fit index as model chi-square value divided by chi-square determined from base ALERT counts. This fit index was chosen in place of chi-square under independence, as the concern was with the improved fit of estimates beyond base ALERT data. We expected this method to be relatively free of particular distribution concerns.²² Index values varied from 1 (no improvement over base ALERT) to 0 (best fit) and were used to rank estimates for goodness of fit. The primary evaluation method for comparing denominator estimates was by pairwise likelihood ratios between each pair of estimates, calculated as a ratio of chi-square statistics with an F-distribution. We made a secondary analysis to determine how well each estimate type explained patterns of county variation in the population regardless of overall goodness of fit.

While the primary concern of this study was county adolescent immunization rates, for each model we calculated an aggregate tetanus-diphtheria-acellular pertussis (Tdap) rate for the entire state by matching an ALERT Tdap numerator with denominators from each model. In addition, we applied the ogive weighted method to ALERT 13- to 17-year-old populations for Tdap, meningococcal, and human papillomavirus (HPV) immunizations. For all immunizations, the ogive weighted estimates were then compared with NIS results for Oregon.

RESULTS

According to school review data for 2013, there were 45,364 seventh-grade students in Oregon as of February 19, 2013; of these students, 45,290 were assignable to one of Oregon's 36 counties. Overall, 93% were in public schools and 7% were in private schools. As a population reference, the Oregon birth cohort size for 1999 was 45,193 children, indicating that net mobility was low even though substantial in-and-out migration occurs. Two small rural counties where both school and health functions are integrated into a larger, neighboring county were merged with their neighbor for analysis. In addition, several school districts and education programs crossed county lines and were proportionately assigned. As expected from mobility inflation of teen population counts, the unadjusted count of 13-year-olds in ALERT as of the assessment date was 126% of the school review count (data not shown).

The rank order of estimates from best to worst fit was: ogive weighting (index = 0.021), linear weighting (index = 0.041), Census Bureau estimates (index = 0.221), AIRA-MIROW exclusion (index = 0.372), and base ALERT counts (index = 1.0) (data not shown). Likelihood ratios for comparing the different types of estimates are presented in the Table. Overall, the ogive weight of IIS records had the best fit of the five different estimate types, with a 10.6 times better fit than the Census estimates and a 17.8 times better fit than the AIRA-MIROW standard (Table). For the secondary purpose of how different estimate types explained county-to-county variation independent of overall goodness of fit, we observed the same rank order. In this secondary analysis, the likelihood ratio between the ogive and linear weights was reduced to a nonsignificant 1.12 (p=0.37)(data not shown). However, the likelihood ratio comparison between the ogive weight and Census Bureau estimates remained significant at 5.95 (p<0.001) (data not shown).

Table.

Likelihood ratios for comparing estimators of 13-year-old populations (n=45,290) in 34 counties in Oregon for potential Oregon ALERT IIS immunization rate denominators, 2013^a

^aBased on comparison with Oregon annual review of school immunization compliance data

^bSignificant at p<0.05

^cSignificant at p<0.001

^dRatio inverses are not presented.

IIS = Immunization Information System

AIRA-MIROW = American Immunization Registry Association Modeling of Immunization Registry Operations Workgroup

NA = not applicable

We calculated Tdap rates in aggregate for 13-year-olds in the state for each model. Using a numerator from ALERT and model denominators produced the following: for the unadjusted ALERT denominator, the Tdap rate was 70.7%; for Census denominators, the Tdap rate was 82.0%; for the AIRA-MIROW 10-year exclusion, the rate was 77.5%; for the linear ALERT weight, the Tdap rate was 91.7%; and for the ogive ALERT weight, the Tdap rate was 88.4% (data not shown).

In comparing ALERT ogive weighted rates for 13- to 17-year-olds with NIS Teen rates for 2013, the NIS Tdap rate for Oregon was 87.0% (95% confidence interval [CI] 64.3), while the ALERT ogive weighted rate was 89.8%. For meningococcal, the NIS rate was 65.3% (95% CI 65.8), and the ALERT ogive rate was 65.3%. For female 1+ HPV, the NIS rate was 66.3% (95% CI 68.4), while the ALERT ogive rate was 61.3%; for 3+ female HPV, the NIS rate was 39.5% (95% CI 68.8), while the ALERT ogive rate was 32.7% (data not shown).

DISCUSSION

The results of this study support the feasibility of addressing denominator inflation issues in IIS data by weighting individual records for the time between assessment and last reported immunization. Applying a nonlinear, logarithmic (ogive) weight in this study provided the best results. In the context of Oregon data, both linear and nonlinear weighting of ALERT data provided a better match to county-level school populations than did Census Bureau estimates. The ogive weighting method performed better than the other examined methods for predicting local teen population counts using Oregon ALERT IIS data, regardless of whether the total population count was known or only the variation between counties was known.

Most U.S. states have adolescent school requirements for Tdap.²³ However, there is substantial state-to-state variability in how school requirements are administered, so that other IISs may not have such a readily available source of population validity, or have the near-complete IIS reporting levels that allow comparisons with whole population sources. Further studies on the applicability of the weighting used in this study may be needed in other areas with less complete data or lower reporting levels.

While the focus of the proposed ogive weighting is its use at the county level, the weighted results also matched to aggregate state-level results from the NIS. This match of the NIS results to weighted ALERT data provides further validation of the weighting approach's feasibility. For school-required Tdap immunization, the weighted results were slightly above NIS rates but not significantly different. As the NIS questionnaire for Tdap first screens for tetanus-diphtheria (Td) and may contain a slight bias to reporting Tdap as Td, it was expected that our weighted results could be slightly above the NIS rate. All ALERT ogive weighted rates for 13- to 17-year-olds fell within the CIs for NIS rates.

An advantage of using IIS data to estimate adolescent immunization coverage is the basis in centralized, provider-supplied records. IIS data use avoids issues of record scattering or inaccuracy in survey recall of immunizations.^24–26 Having IIS data for both numerators and denominators rather than hybrid methods using IIS numerators and external denominators may provide a benefit in ensuring that numerator and denominator biases are matched. A further issue, however, is ensuring that IIS populations are representative of the entire population.

Limitations

This study was subject to several limitations. The first was that matching seventh-grade to 13-year-old populations is not perfect; despite picking an assessment date that should lead the two to be the same, a small number of children in school will be either younger or older than their grade cohort. Another concern was with gap populations that are outside of the review process. By seventh grade, a small number of teens are likely to have already dropped out of school. In addition, the degree of homeschooled teenagers captured in the review process was uncertain. Those who participate in optional activities sponsored through their school districts, such as online education, music, and sports programs, were included, but comprehensive data for assessing this participation are not currently available. A further limitation was that evaluated immunizations were limited to those due in adolescence. The applicability of this method for assessing adolescent rates for immunizations started in early childhood, such as measles, is currently in need of study and may require weighting not only by time to last immunization but also weighting by first reported immunization.

CONCLUSION

Weighting IIS records was shown in this study to increase the accuracy of denominators and by extension the immunization rates produced by IISs. The overall best fit to external population data for 13-year-olds was obtained by weighting individual records using a nonlinear ogive function based on the time since last reported immunization. This nonlinear ogive weight approach performed well in correcting problems of IIS count inflation regardless of whether or not reliable external sources of local population data were available.