Evaluation of the frequency of immunization information system use for public health research

Abstract

Immunization information systems (IIS) have been useful for consolidating immunization data and increasing coverage, and have the potential to be a valuable resource for immunization research, but the extent which IIS data are used for research purposes has not been evaluated. We reviewed studies conducted using data from federally supported state and city immunization program IIS, and categorized research type based on study objectives to evaluate patterns in the types of research conducted. Research papers using IIS data published between 1999 and July 3, 2012 were identified by searching the CDC IIS publication database and PubMed. These searches produced 304 and 884 papers, respectively, 44 of which were eligible to be included in this evaluation. The most common research category was evaluation of factors associated with vaccine coverage and vaccine coverage estimates (n = 20). This study shows that IIS may not be used to their full potential with regards to research. Further research is needed to determine barriers to using IIS data for research purposes.

Introduction

The Centers for Disease Control and Prevention (CDC) defines immunization information systems (IIS) as “confidential, population-based, computerized databases that record all immunization doses administered by participating providers to persons residing within a given geopolitical area”.¹ Development of IIS largely began in the 1980s but was primarily focused in managed care organizations. Starting in 1993 federal funding was provided for the creation of population-based IIS.² By 1999, 61 of the 64 state and local immunization programs were using federal funds to implement these systems, and 84% of all children in the United States under the age of 6 had two or more vaccinations documented in an IIS in 2011.³

The use of IIS has been suggested as a way to address the fact that children with multiple healthcare providers are less likely to be up to date with their immunizations.^4,5 In addition to improved vaccination coverage, there are other important outcomes of IIS use, such as cost savings, generating vaccination recall notices, vaccination reminders and providing official vaccination history forms for use in meeting school entry immunization requirements.⁶

IIS are especially useful because they provide population-based data and thus are less prone to bias introduced by only including people who are able to seek out medical services,⁷ though the extent to which this bias is reduced depends on the completeness of provider participation in the IIS. Such reduction of bias makes IIS data a valuable research tool for creating new immunization schedule recommendations,² or monitoring the impact of vaccine shortages.⁸ Compiling such comprehensive data opens novel avenues for research including analyzing immunization accessibility, quality and disparities.⁷ Research into these areas with the accuracy that IIS can provide has the potential to increase vaccination coverage and lower the rate of vaccine preventable diseases, especially in vulnerable populations. In fact, South Carolina law states that the use of IIS “will enable research on the causes, distribution and prevention of vaccine preventable diseases,” and New York law states that IIS data may be used “for the purposes of outreach, quality improvement and vaccine accountability, research, epidemiological studies and disease control.”^9,10 In order to maximize the potential of this powerful tool, it important to determine what type of research is being done with data produced by IIS.

In 2010, the Guide to Community Preventive Services conducted a systematic review of papers published using IIS data. Using the 71 published papers and 123 conference abstracts they found, they concluded that IIS are useful for surveillance and investigation of vaccination rates, provider assessment and feedback, providing vaccine reminders and recalls, assisting during outbreaks of vaccine-preventable diseases, facilitating management of vaccines and identifying missed opportunities, invalid dose administration and disparities in coverage.¹¹ This review included conference abstracts, papers that included data from multiple sites (such as Sentinel Site data) and was not restricted to one country. In addition, this review did not examine the use of IIS data specifically for research purposes. Therefore, though findings from this review are important for understanding the utility of IIS, it does not examine the use of individual IIS associated with the 64immunization programs in the United States for research purposes, and it is possible that the full potential of this powerful tool is not being realized. We examined patterns in the use of individual IIS data for research by reviewing all papers published since 1999 that used IIS data.

Results

We identified 304 and 884 papers from the CDC IIS publication database and PubMed search results, respectively. No additional new manuscripts were found through reference list review. Following removal of duplicates and applying inclusion and exclusion criteria, 44 papers were available for analysis (Fig. 1).^12-55 These 44 manuscripts were produced through research at 18 IIS locations. The most number of manuscripts were affiliated with a university (n = 16),^{12,16,17,19,20,24,25,27,32-34,36,40,41,48,55} followed by affiliations with a health department (n = 9).^{14,20,22,30,31,38,46,47,49} Other affiliations included the CDC (n = 8),^{18,28,29,37,39,42,44,45} a hospital (n = 8),^{12,15,26,35,41,53} an HMO (n = 1),²³ a consulting service (n = 1)⁵⁴ and an independent research group (n = 1). Data from the Michigan IIS (Michigan Care Improvement Registry) were used to generate 9 manuscripts^{12,14,15,30,39-41,43,48} the most from any IIS, followed by Philadelphia (8)^{27,28,33,35,37,44,45,55} and New York City (6)^{17,18,32,46,49,51} (Table 1). Other than these immunization programs, no IIS produced more than three manuscripts. The number of IIS manuscripts published each year followed an increasing trajectory between 1999 and 2011, with the majority of publications data in the year 2008 or later (Fig. 2).

Figure 1. Research papers published between 1999 and July 3, 2012 utilizing data from immunization information systems, identified through a review of a CDC database [54] and PubMed, with searches conducted between February 13, 2012 and July 3, 2012.

Table 1. Number of papers published using IIS data from individual Immunization Programs and the number of minimum functional standards met by those Immunization Programs as of 2009.

Immunization Program	Number of Papers
Michigan	9
Philadelphia	8
New York City	6
Arizona	3
North Carolina	2
Oregon	2
Houston	2
Connecticut	2
Wisconsin	1
District of Columbia	1
Minnesota	1
Utah	1
Washington State	1
San Antonio	1
Chicago	1
Colorado	1
Delaware	1
North Dakota	1

Figure 2. Number of published research papers using data from an immunization information system, by year of publication, 1999–2011. Note that only January–July was included for 2012, so the three publications from 2012 are not included in this figure

The largest group, was coverage associations and estimate evaluations (n = 22 papers), including 11 describing associations with coverage,^{12,17,21,27,38-40,51,54,55} 2 estimating coverage rates,^21,22 2 evaluating completion vaccine series completion,^23,24 3 evaluating the accuracy of coverage estimates from parents^25-27 and 4 describing the completeness of data in the IIS.^28,29,43,44 The Policy implementation/change category included 11 papers, including one that described the coverage of a new vaccine compared with an older vaccine,³⁰ one that examined recommended ages,³⁵ two that examined the impact of a policy change,^36,37 and 7 that evaluated an intervention.^31-34,45-47 Two manuscripts described a response to short-term vaccine supply issues, including the impact of an outbreak³⁸ and a shortage.³⁹ There were six papers in the reminder/recall group, with three papers evaluating the use of an IIS in a vaccine recall^40,48,49 and three evaluating the use of IIS in vaccination reminders.^41,50,51 Three papers evaluated vaccine effectiveness,^42,52,53 all of which focused on the rotavirus vaccine.

Discussion and Conclusion

We conducted the first assessment, to our knowledge, on the extent of use of publicly funded IIS in the US for research purposes. While IIS have been in place for many immunization programs for over a decade, there have been relatively few research reports utilizing these data. Additionally, the IIS locations conducting these research studies are limited, with more than half of the published research papers coming from three immunization programs. While IIS have served many purposes to aid public health practice (e.g., reminder/recall systems, generation of immunization reports for school entry and surveillance for immunization coverage), it appears that we are not currently using IIS to their full potential. On the other hand only 2 manuscripts used data collected after 2009,^12,25 indicating that there is a lag between when data are collected and when research is published. Therefore, it is possible that now that IIS have matured and include more data they are more useful to researchers and manuscripts using current data will be published in the near future.

The largest groups of studies dealt with factors associated with coverage and evaluation of an intervention. Using IIS to answer these questions not only takes advantage of the reduced bias in the population-based data available with IIS, but also helps with one of the main purposes of IIS; increased coverage. Though IIS are useful for researching factors associated with vaccine coverage, they can also be used for other vaccine-related research, such as vaccine effectiveness, or adverse events. Only one study examined recommended ages for vaccination, and all three of the studies examining vaccine effectiveness focused on rotavirus vaccine (though a study published after the time period covered by this review used IIS data to research influenza vaccine effectiveness, showing that these data are being utilized to study multiple vaccine preventable diseases).⁵⁶ There were no studies on adverse events. On the other hand, the systematic review conducted by the Guide to Community Preventive Services found more papers and abstracts than included in this review, so it is possible that research into these areas is being conducted, but did not fit our inclusion criteria (e.g., multi-site research or research done with HMO databases).

There are many barriers to research, including possible issues with data sharing and confidentiality, as well as staffing limitations due to recent cuts in the public health workforce.⁵⁷ In addition, it is possible that immunization program staff have other priorities regarding IIS (such as generating vaccination reports or managing vaccines), and are under time constraints. Though IIS have been shown to be useful in immunization research, such barriers may prevent them from being used in this manner. Our results imply that partnerships with academic institutions may be one way to overcome these barriers and use the data from IIS most effectively. Future research is needed to understand these issues. In response to this need we have conducted a survey of Immunization Program Managers and future direction of work includes analysis of possible barriers to research with IIS research and data sharing and usage.

This study has some limitations. It was assumed that regional registries that covered an area different from a federally funded state or local registry. Since regional registries have been known to combine to form what we now consider state or local registries, it is possible that there were some studies used data from regional registries that later joined to become what are now known as the 64 federally funded state and local registries⁵⁸. However, as we were interested in how those immunization programs in particular were using their IIS data, the resulting bias is likely minimal. Only published studies included in the CDC website and PubMed were reviewed. Therefore, it is possible that gray literature or studies that have been completed but not published were missed. On the other hand, most high quality research is published in peer reviewed databases. Our study shows that IIS are not being used to their full potential with regard to research. Since the largest number of studies were affiliated with a university, it is possible that lack of a relationship with a university could be seen as a barrier to research, and immunization programs that want to use the IIS data for research purposes could be advised to seek such a relationship. There are other possible barriers to the use of IIS data, including concerns regarding confidentiality, data quality and budget constrictions, but further research is needed in this area. Our study highlights the need for future research, both with IIS data itself and barriers to such research.

Methods

Literature search

We identified published IIS research manuscripts using two systems. First, we searched the CDC IIS publication database⁵⁹ for papers published from 1999- July 3, 2012. Next, We searched Pubmed during the period April 9, 2012 to July 3, 2012 using the search terms “(immunization OR vaccination) AND [(information system*) OR registry]” with results limited to papers written in English and published after January 1, 1999. Titles and abstracts were reviewed for possible IIS data usage for research purposes, and the full article was reviewed for those studies that reported research from an IIS. Papers were included if they described using data from an IIS affiliated with an immunization program registry or a “regional registry” that covered the same geographic area as an IIS affiliated with an immunization program registry.⁶⁰ For example, a regional registry that covered Philadelphia was assumed to be the Philadelphia citywide IIS, and was included. However, if a “regional registry” was referred to and the study area differed from that of a federal registry, for example a regional registry that covered Boston, MA, the paper was excluded.⁶¹ Papers were also excluded if they were not from the United States, used a managed care organization, hospital or other IIS. In addition, to see how individual immunization program registries were using their data rather than how it was being used as part of a research consortium, we only included studies that covered the area of one IIS, thereby further excluding analyses reported using National Immunization Survey, the Vaccine Safety Datalink, the CDC Sentinel sites, and other studies that used data from multiple IIS. Our focus for this evaluation was on research activities using IIS data, as defined by an activity that “contribute[s] to generalizable knowledge to improve public health practice,” the results of which can be used to benefit a population beyond the scope of the study;⁶² therefore papers specifically addressing IIS implementation, methodology, or cost issues were excluded. In accordance with this objective, we excluded gray literature, such as information posted on websites or reports, and only included papers published in journals. The reference lists of the IIS papers we had included were then searched for any additional research manuscripts.

Analysis

For each research manuscript, information on publication date, IIS location, study objective and author affiliation (e.g., university, health department etc.) was extracted. For papers with more than one author affiliation reported, only the affiliation of the corresponding author was included. The number of times each IIS was used was totaled, as was the total number of publications in each year. Categories for qualitative grouping were created based on study objectives to assess patterns in the type of research being conducted through IIS. The manuscripts were grouped into five main categories: coverage associations and estimate evaluations, policy implementation/change, response to short-term vaccine supply issues, reminder/recall and vaccine effectiveness.

IRB

Since this was a review and used no human subjects, no IRB approval or informed consent was needed.

Disclosure of Potential Conflicts of Interest

No potential conflicts of interest were disclosed.