Effects of Automated Immunization Registry Reporting via an Electronic Health Record Deployed in Community Practice Settings

Abstract

Background

Among the expected benefits of electronic health records (EHRs) is increased reporting of public health information, such as immunization status. State and local immunization registries aid control of vaccine-preventable diseases and help offset fragmentation in healthcare, but reporting is often slow and incomplete. The Primary Care Information Project (PCIP), an initiative of the NYC Department of Health and Mental Hygiene, has implemented EHRs with immunization reporting capability in community settings.

Objective and Methods

To evaluate the effect of automated reporting via an EHR on use and efficiency of reporting to the NY Citywide Immunization Registry, we conducted a secondary analysis of 1.7 million de-identified records submitted between January 2007 and June 2011 by 217 primary care practices enrolled in PCIP, pre and post launch of automated reporting via an EHR. We examined differences in records submitted per day, lag time, and documentation of eligibility for subsidized vaccines.

Results

Mean submissions per day did not change. Automated submissions of new and historical records increased by 18% and 98% respectively. Submissions within 14 days increased from 84% to 87%, and within 2 days increased from 60% to 77%. Median lag time decreased from 13 to 10 days. Documentation of eligibility decreased. Results are significant at p<0.001.

Conclusions

Significant improvements in registry use and efficiency of reporting were found after launch of automated reporting via an EHR. A decrease in eligibility documentation was attributed to EHR workflow. The limitations to comprehensive evaluation found in these data, which were extracted from a registry initiated prior to widespread EHR implementation suggests that reliable evaluation of immunization reporting via the EHR may require modifications to legacy registry databases.

1. Introduction

Immunization registries are confidential, population-based, computerized databases that record all immunization doses administered by participating providers to persons residing within a given geopolitical area [1, 2]. State and local registries offset fragmentation in healthcare by consolidating scattered records, by reducing unnecessary vaccination among patients who have lost records or visit multiple providers, and by tracking vaccine formulations, supply, and schedules [3]. In 2010, the Task Force on Community Preventive Services recommended registries to increase vaccination coverage based on strong evidence of their effectiveness for managing immunizations [4]. Registry data also are used to evaluate vaccine programs and population coverage [5-10]. However, accuracy, timeliness and completeness of data submitted are highly variable [9, 11-22], reducing the impact of these registries. Electronic health records (EHRs) provide a potential solution when immunizations documented at the point of care are reported automatically, directly to the registry [5, 23].

In 1997 the Citywide Immunization Registry (CIR) was established as a centralized repository of immunization records for people vaccinated in NYC. Over 2,000 medical practices and other venues are mandated to report immunizations administered to children 19 and older who reside in NYC. [19, 24]. There are three ways to submit immunization records to CIR:

• 1.manual entry in CIR’s online registry;
• 2.electronic transfer as a batch file, either automatically from an EHR or using data from a practice billing system; and
• 3.direct real-time HL7 messaging, which currently is being implemented and not examined here [25].

Starting in 2007 EHRs equipped with an automated immunization reporting module were implemented in community practice settings by the Primary Care Information Project (PCIP). This initiative of the NYC Department of Health and Mental Hygiene subsidized adoption of EHRs, targeting practices serving communities at risk for health disparities determined by Medicaid visits per zip code [26]. Within the EHR a customized application, or module, automates the process of reporting immunization records via an Internet connection to CIR established when a practice first deploys the module. Each evening the module automatically aggregates the day’s records of immunizations administered and uploads a batch file directly to the CIR Web File Registry (WFR) [27]. This nightly process extracts immunization data directly from the native EHR database with no additional workflow for staff. The specifications for the module were developed collaboratively by PCIP and CIR and implemented by the vendor eClinicalWorks (eCW). At the time of the study immunization reports used a legacy reporting format, and CIR was planning transition to the national HL7 standard. To date, there are approximately 300 practices reporting to CIR via an automated immunization module.

We conducted a secondary analysis of immunization records submitted to CIR by a cohort of these practices to evaluate the effect of automated EHR-based reporting on use of the registry and efficiency of reporting. To our knowledge there are no previous studies of registry use and efficiency related to EHR-based reporting.

2. Methods

Practices were eligible if they were enrolled in PCIP, had implemented the immunization module in their EHR, and were certified by CIR to use the module to submit immunization records. In general, a practice is certified after approximately 30 days of rigorous system testing by CIR staff.

We obtained a de-identified, limited data set of vaccination records submitted to CIR by practices that met the study criteria. The data covered children ages 19 and under and represented all immunizations, including influenza, submitted to the CIR between January 1, 2007 and May 31, 2011. Records submitted in the 1.25 years before the CIR certification date for each practice were considered pre-implementation records. Records submitted in the 1.25 years after were considered post-implementation records. This time period was selected to reflect a full year of reporting before and after certification, including additional time (0.25 years) for transition pre/post. In all cases daily submission counts above the 99th percentile were removed to minimize skew in the distribution. MySQL was used to extract relevant data sets for analysis. SAS was used for statistical tests and to generate regression plots. The study was approved by Columbia University Institutional Review Board.

To measure effect on use of the registry we tested pre and post implementation differences in the mean number of records submitted per day via all sources, via the online registry, and via automatic file transfer. In the sub set of records submitted via automatic file transfer we examined new records (vaccines submitted ≤ 1 year of delivery) and historical records (submitted >1 year post delivery, or flagged as historical by the provider). These data were normally distributed and the student’s t-test was performed.

To evaluate the efficiency of reporting we excluded historical records from the analysis to eliminate the effect of large batches of backlogged records submitted by individual practices shortly after EHR immunization module deployment. We tested pre and post implementation differences in the median percentage of records from all sources uploaded within the 14-day requirement and within 2 days. Due to the nightly record upload a 2-day window represents immediate reporting via the module and reflects an increased use of the module. The 14-day reporting period is the legal reporting requirement and represents a broader measure of reporting efficiency by study practices. We tested pre and post implementation lag time, defined as the difference in days between the vaccine administration date and the date it was reported to the CIR. To approximate record completeness pre and post implementation we determined the median percentage of records submitted from all sources which had a non-null value in the field indicating eligibility for the Vaccine for Children program (VFC), a federal program that subsidizes vaccines. CIR tracks vaccines administered through this program. These data were asymmetrically distributed and the Wilcoxon rank sum test was performed. We also performed segmented regressions to allow visual inspection of pre/post data plots and to test for significant differences in slope.

3. Results

There were 217 practices that met the study criteria. Of these 82% (n = 178) had submitted records to CIR before implementation of automated reporting via the EHR. About 18% (n = 39) began submitting to CIR after implementation, despite the NYC reporting requirement. The data obtained consisted of 1.7 million immunization records representing 220,000 unique individuals. It should be noted that the population of children was stable during the study period, as shown in ► Table 1 [28].

Table 1.

US census population estimates of children in NYC during the study period.

Group	2007*	2008*	2009*	2010	Mean	SD
Under 5	565,649	575,742	580,330	517,724	559,861	28,753
Age 5 – 9	499,273	519,022	526,440	473,159	504,474	23,818
Age10 – 14	507,573	498,542	475.648	468,154	487,479	18,615
Age 15– 19	548,635	551,139	516,709	535,833	538,079	15,745
Total	1,994,870	2,099,127	2,144,445	2,121,130	2,089,893	65,996

*2007 – 2009 intercensal figures are forward estimates based on the 2000 census

Although the mean number of records submitted daily to CIR from all sources remained constant, the pattern of submissions changed significantly. The results are shown in ► Table 2.

Table 2.

Immunization records submitted to NYC Immunization Registry before and after by a set of primary care practices that were certified to submit immunization records automatically from their EHR. N = 178 practices reporting before, 217 practices reporting after certification by CIR. Differences are significant at p<0.001 unless noted.

Use Case Goal	Observation1	Before Mean (SD or IQR)	After Mean (SD or IQR)
Increased registry use	Mean records submitted to CIR per day via all sources2	2442 (618)*	2419 (662)*
	Mean records submitted manually via the on-line registry per day	1061 (264)	369 (112)
	Mean records submitted via automated file transfer3 per day	1415 (454)	2051 (615)
	Mean new records submitted via automated file transfer per day	917 (298)	1085 (320)
	Mean historical4 records submitted automated file transfer per day	497 (238)	985 (426)
More efficient reporting	Median percent of all records submitted to CIR via all sources within 14 days5	84.1	86.6
	Median percent all records submitted to CIR via all sources within 2 days	60.2	76.7
	Median lag time (in days) of records submitted via automated file transfer	13.3 (10.7, 17.2)	9.8 (6.4, 15.9)
	Median lag time (in days) of records submitted manually via the on-line registry	32.7 (25.9, 39.9)	80.3 (61.6, 105.6)
	Median percent records with VFC6 field completed	70.9	62.7

* not significant

¹ outliers above 99th percentile for each element of the dependent variable (records submitted per day) were excluded

² includes records from all sources (on-line registry and automated file transfer)

³ automated file transfer; records can be submitted by file transfer from a practice management/billing system or an EHR

⁴ historical records: vaccine administration date more than one year from the submission date, or record flagged by provider as historical

⁵ immunizations must be reported within 14 days pursuant to NYC regulations

⁶ Vaccines for Children, a federal subsidy program for which eligibility is tracked by CIR

Submissions via the online registry decreased from a mean of 1061 to 369 per day. Submissions via automated file transfer increased from mean of 1451 to 2051 per day. For new records automated file transfers increased from a daily mean of 917 to 1085 and for historical records from 497 to 985. Median lag time before implementation was 13.3 days [IQR = 10.7, 17.2], and after was 9.8 days [IQR = 6.4, 15.9]. Median lag time for manual submissions before implementation was 32.7 days [IQR = 25.9, 39.9], and after was 80.3 days [IQR = 61.6, 105.6]. The percentage of records submitted to CIR within the 14-day legal limit improved from 84% to 87%. The percentage of records submitted within 2 days increased from 60% to 77%. The proportion of records with the Vaccine for Children field completed decreased significantly after implementation, from 70% to 62%. These differences are significant at p<0.001.

A segmented regression plot in ► Figure 1 shows the percent of immunization records reported within 2 days of vaccine administration. The certification dates of each practice were aligned and outliers above the 99th percentile were removed. There is a visually notable decrease in dispersion of submissions after automated reporting with a significant pre/post difference in slope of the regression line (p<0.001).

Fig. 1.

A segmented regression plot showing the percent of immunization records reported to the NYC Immunization Registry within 2 days of vaccine administration by 217 primary care practices. The certification dates for each practice were aligned. Outliers above the 99th percentile were removed. There is a significant difference in the slope (p<0.001).

4. Discussion

The combined evidence suggests that automated reporting via the EHR has facilitated registry use for these 217 practices. The overall mean number of records submitted per day remained stable, suggesting already high compliance by providers with reporting mandates as well as suggesting a stable number of immunizations during the study. Automatic submissions (both new and historical) did increase in proportion to all submissions, and records from 39 previously non-compliant practices were captured. The increases in the proportion of submissions within 2 and within 14 days may underestimate the true efficiencies achieved since 18% of practices had not previously reported. The decreases in 2 and 14 day lag time are corroborated by the decreased data dispersion visualized post certification.

We did not assess how automated reporting affected workflow, and cannot speak to whether the gains in efficiency for CIR came at the expense of any clinician time required to use the reporting module. We might argue the greater lag in manual reporting that we found post implementation suggests a preference for automated reporting, but it could also suggest over reliance on automated reporting, which would represent an unintended consequence. As always, further research is needed to fully comprehend the effects of this, or any, implementation.

The decline in completeness of the VFC eligibility field may be due to multiple factors. Records submitted via the EHR did not contain a field for Medicaid numbers, a criterion from which eligibility may be determined. Further, a provider could document VFC eligibility in more than one field, but automated file transfer captured only one of those fields, suggesting that the EHR workflow may present difficulties for recording eligibility. CIR staff is evaluating these issues.

The study was greatly limited by the registry database structure: records submitted electronically via a non-EHR practice management system versus the EHR immunization module could not be dis-aggregated. This fact limited our analysis to pre- and post-certification of electronic reporting aggregated at the practice level. We were unable to discern if practices may have used non-EHR systems after certification to submit records electronically. We believe that situation is unlikely for two reasons: reporting via the EHR was automated to deliver records in a more timely way, and our findings clearly documented such a change. The data structure also limited our ability to ascertain a rate of immunization based on the patient population of the study practices. Significant changes in the underlying patient population could impact the interpretation of the study results. We don’t believe this to be the case as the the overall patient population was stable during the study period (► Table 1) and no significant changes in the mean immunizations reporting pre and post implementation were found (► Table 2). The structural limitation presents a cautionary lesson for those considering analysis of data from legacy immunization registries to evaluate reporting via the EHR. Perhaps our strongest finding is to illustrate that evaluation of the impact of automated immunization reporting will require the collection and reporting of additional data elements by EHR systems and registries to improve the accuracy and reliability of future studies.

5. Conclusion

Immunization recommendations in the US currently target 17 vaccine-preventable diseases across the lifespan [29]. As EHRs proliferate, automated reporting of immunizations can help public health officials and individual providers better determine who in the patient population has been adequately immunized, reduce provider paperwork and staff time, provide easy access and reliable immunization histories, and reinforce the concept of the medical home [30]. Improved efficiency of immunization registry reporting via an EHR is of increased importance for underserved and at risk populations where transition of care between multiple providers is not uncommon [32]. This study demonstrated a positive impact of EHR adoption on immunization reporting to a citywide immunization registry. Benefits such as these are among the reasons that immunization reporting is included in Stage I requirements for meaningful use of EHRs [31]. NYC CIR has recently implemented bi-directional HL7 messaging standards that will allow providers to attest to this meaningful use criterion. The enhanced interoperability between CIR and EHR systems that results will provide greater consistency in data access and exchange, and increase the availability of health information for other public health functions to assure community health (e.g. assessment of vaccine coverage during emergencies, spikes in communicable disease incidence, or outbreaks of seasonal illness) [5].

Clinical Relevance Statement

This study demonstrated efficiencies in automated reporting via an EHR to an immunization registry. Improvements included reduction in reporting lag time and an increase in automated reporting versus other methods. The ability of EHRs to improve reporting of health data to registries has implications for achieving public health goals for many at risk populations.

Conflicts of Interest

The authors declare no conflicts of interest in the research.

Human Subject Research Approval

The study was approved by the Columbia University Institutional Review Board.