Improving the Quality of Immunization Delivery to an At-Risk Population: A Comprehensive Approach

Abstract

OBJECTIVE:

Immunization quality improvement (QI) interventions are rarely tested as multicomponent interventions within the context of a theoretical framework proven to improve outcomes. Our goal was to study a comprehensive QI program to increase immunization rates for underserved children that relied on recommendations from the Centers for Disease Control and Prevention’s Task Force on Community Preventive Services and the framework of the Chronic Care Model.

METHODS:

QI activities occurred from September 2007 to May 2008 at 6 health centers serving a low-income, minority population in Washington, DC. Interventions included family reminders, education, expanding immunization access, reminders and feedback for providers, and coordination of activities with community stakeholders. We determined project effectiveness in improving the 4:3:1:3:3:1:3 vaccination series (4 diphtheria-tetanus-pertussis vaccines, 3 poliovirus vaccines, 1 measles-mumps-rubella vaccine, 3 Haemophilus influenzae type b vaccines, 3 hepatitis B vaccines, 1 varicella vaccine, and three 7-valent pneumococcal conjugate vaccines) compliance.

RESULTS:

We found a 16% increase in immunization rates overall and a 14% increase in on-time immunization by 24 months of age. Improvement was achieved at all 6 health centers and maintained beyond 18 months.

CONCLUSION:

We were able to implement a comprehensive immunization QI program that was sustainable over time.

Nearly 1 in 4 young children in the United States are not appropriately immunized.¹ Underimmunized children are more likely to be poor,^2,3 from single-parent households,² African American,^3–6 and from inner cities.^7–9 The Centers for Disease Control and Prevention’s (CDC’s) Task Force on Community Preventive Services has recommended 13 interventions with the potential to improve immunization rates based on a literature review of their efficacy.^10,11 In 2008, the recommended interventions related to outpatient pediatric practice included client reminder/recall systems, reduction of out-of-pocket costs, expansion of access in clinical settings, provider reminder/recall, multicomponent educational strategies, assessment and feedback for providers, standing orders, and vaccination programs in Women, Infants and Children (WIC) settings. To our knowledge, no study has evaluated the impact of implementing all the Task Force recommendations related to ambulatory pediatric practice jointly.

Because the Task Force recommendations only include proven effective strategies, they potentially miss beneficial but untested interventions. To examine this possibility, it is useful to consider the fit of the Task Force recommendations to a theoretical framework designed to improve patient outcomes such as the Chronic Care Model (CCM). The CCM includes 6 elements: community resources and policies, patient self-management support, health care organization, delivery system design, decision support, and clinical information systems.^12,13 Implementation of the model has been shown to improve patient outcomes for various populations, including children, and various health conditions.^14,15 The CCM is applicable to improving immunization coverage because, similar to chronic care, immunization occurs in the outpatient setting and requires a sustained commitment to preventive services by the community and health care system.

Our 6 health centers instituted an immunization quality improvement (QI) program in which we implemented all of the Task Force’s recommended interventions relevant to outpatient pediatric practice. We fit the recommendations to the CCM framework and decided to add another intervention we felt was missing, yet important for success. The aim of this study was to determine the effectiveness of our immunization QI program for increasing compliance with the 4:3:1:3:3:1:3 vaccination series (which includes 4 diphtheria-tetanus-pertussis vaccines, 3 poliovirus vaccines, 1 measles-mumps-rubella vaccine, 3 Haemophilus influenzae type b vaccines, 3 hepatitis B vaccines, 1 varicella vaccine, and 3 doses of 7-valent pneumococcal conjugate vaccine [PCV7]) among children 19 to 35 months old.

Methods

Setting

Activities occurred from September 2007 to May 2008 at 6 health centers associated with a pediatric hospital in Washington, DC. The largest center, center C, is colocated with the hospital, whereas the others are community-based. Center F includes a permanent site and a mobile van. All centers serve patients who are predominantly poor and publicly insured (Table 1). The majority of patients are African American at all centers, with the exception of center B, which is predominantly Hispanic. Collectively, ∼30 physicians, 3 nurse practitioners, 90 residents, and 42 nurses immunize ∼37% of DC’s pediatric population annually.

TABLE 1.

Overall Health Center Sociodemographic Characteristics in 2008

	Health Center
	Overall	A	B	C	D	E	F
Annual patient visits	71 936	5507 (8%)	11 068 (15%)	30 156 (42%)	8323 (12%)	6337 (9%)	10 544 (15%)
Publicly insured, n (%)	60 921 (85%)	4538 (82%)	9684 (87%)	25 104 (83%)	7302 (88%)	5532 (87%)	8761 (83%)
Race/ethnicity, n (%)
African American Hispanic White Asian Other/unknown	53 264 (74%) 10 885 (15%) 380 (1%) 602 (1%) 6805 (9%)	4721 (86%) 163 (3%) 34 (1%) 266 (1%) 323 (6%)	2132 (19%) 8454 (76%) 62 (1%) 140 (1%) 280 (3%)	22 766 (75%) 2120 (7%) 221 (7%) 185 (1%) 4860 (16%)	7992 (96%) 10 (0%) 6 (0%) 0 (0%) 315 (4%)	5603 (88%) 14 0%) 27 (0%) 6 (0%) 687 (11%)	10 046 (95%) 123 (1%) 30 (0%) 5 (0%) 340 (3%)

Our QI program paid staff consisted of 1 physician 30% time, and 2 full-time administrators. The study protocol was approved by the Children’s National Medical Center institutional review board.

Planning the QI Initiative

We first determined the fit of the Task Force recommendations with the CCM (Table 2).¹⁶ Although the fit was good, there remains a deficiency of studies examining the interaction between health care institutions and the community. According to the CCM, health care organization (defined as its goals and values and relationships with other community stakeholders) is a vital pillar of the chronic care edifice. Therefore, we chose to add community partnership to our activities.

TABLE 2.

The Determinants of Patient Outcome as They Relate to Different Types of Recommended Immunization QI Interventions and the Changes We Implemented

Theoretical Determinant of Patient Outcome (from the Chronic Care Model)	Evidence-Based Recommendation(from the CDC Task Force on Community Preventive Services)	Actual Changes Implemented	How Frequently the Intervention Was Used at Each Health Center
			A	B	C	D	E	F
Health care organization	• Collaboration with community stakeholdersa	• Monthly meetings were held with DC Department of Health and Medicaid MCO representatives to discuss any issues with ordering vaccines, documenting or transferring vaccination data to the IIS, coordination of outreach efforts, etcb	N/A	N/A	N/A	N/A	N/A	N/A
		• Names of children overdue for immunizations who we were unable to contact were transferred to their MCOs of record for more extensive outreach (eg, home visits).	●	●	●	●	●	◯
Clinical information systems	• Provider reminder/recall	• IIS vaccination status report was generated for every patient encounter as a reminder/recall for providers to order due or overdue vaccines.	●	●	●	●	●	●
Decision support	• Assessment and feedback for vaccination providers	• Three training sessions for all providers were held during staff meetings and resident conferences reviewing the vaccination schedules, reducing missed opportunities for vaccination and improving documentation of vaccines administered.	●	●	●	●	●	●
		• Immunization coverage for each health center’s patient population was graphed monthly and shared with all centers to chart progress.	●	●	●	●	●	●
	• Provider reminder/recall	• See above description	●	●	●	●	●	●
Delivery system design	• Expanding access in clinical settings	• Vaccination status was assessed and overdue immunizations administered at sick and well visits (versus just well visits).	●	●	●	●	●	●
		• Immunization-only clinics	◯	◯	⊙	◯	◯	●
	• Standing orders	• Nurses verified all physician vaccine orders with IIS recommendations, and discrepancies were discussed.	●	●	●	●	●	●
Self-management support	• Client reminder/recall systems	• Autodialer messages were sent monthly to all families with 2-, 4-, 6-, 12-, 18-, and 24-mo-old children to remind them of due immunizations.	●	●	●	●	●	●
		• Staff made two attempts at live phone calls to reach families with children ages 0-4 y old who were overdue for immunizations. A postcard was mailed to families unreachable by phone. If no appointment was scheduled 1 mo after the postcard was mailed (or postcard was returned to sender), the child’s name was forwarded to his health plan for further outreach.	●	●	●	●	●	●
	• Multicomponent interventions that include education	• Handouts with the immunization schedule and space for providers to document the due date for the next set of immunizations were given to all parents of newbornsb	⊙	⊙	⊙	⊙	⊙	◯
		• Posters were placed in waiting areas and examination rooms with pictorial examples of vaccine-preventable diseases, school immunization requirements, as well as immunization myths and facts.	⊙	⊙	●	●	●	◯
Community resources and policies	• Vaccination programs in WIC settings	• Vaccination status was assessed for WIC clients at a center colocated with one of our health centers. WIC clients ≤5 y old overdue for vaccines were walked across the hallway to the health center to schedule an appointment or to wait to see a physician that dayc	◯	◯	●	◯	◯	◯

●, always or very frequently; ⊙, occasionally or rarely; ◯, very rarely or never

^aIntervention type in addition to the 7 recommended by the Task Force on Community Preventive Services.

^bMonthly meetings with community stakeholders were attended by immunization QI leadership on behalf of all health centers.

^cThis intervention was also discontinued at health center C after a 3-week trial because of minimal effect.

We chose not to implement 1 Task Force recommendation—reducing out-of-pocket costs. This intervention did not fit well into the CCM framework and was not an important consideration for our patients because the majority was Medicaid enrollees who did not incur personal costs for immunization.

Implementing the QI Initiative

Table 2 describes the changes we implemented and evaluated. All changes were first implemented at health center C where our program director was located. We implemented each change on a small scale for 1 to 3 weeks before seeking feedback from staff and/or families. With feedback, we altered the interventions as needed. Project leaders met monthly with immunization champions (including physicians, nurses, and administrators) from each health center, representatives of the DC Department of Health (DOH) Immunization Program, and the four DC Medicaid health plans to share ideas and discuss intervention implementation strategies. Afterward, immunization champions introduced the changes at their respective centers, tailoring them to enhance performance improvement for their setting.

Collaboration With Community Stakeholders

The DC DOH helped us to identify incorrect monthly health center vaccine orders (eg, insufficient ordering quantities based on last year’s monthly order) and common practitioner administration errors (eg, administering the third hepatitis B dose too soon). Collaboration with the Medicaid plans enhanced outreach. Our QI staff made 2 phone calls and mailed a postcard to families with children overdue for immunizations. If these attempts were unsuccessful, the health plans could sometimes supply new contact or health practitioner information and, if not, could perform home visits.

Provider Reminder/Recall and Assessment and Feedback for Providers

The DC DOH Immunization Information System (IIS) was used to generate reminders for our practitioners for each patient encounter. Individual patient IIS records listed past immunizations and those that were currently due and overdue. Since 2008, reporting to the IIS has been mandatory for all DC health care providers, licensed daycare providers, and public schools, making a very complete registry. According to the CDC IIS Annual Report, in 2008, the IIS captured 100% of DC’s childhood population between 4 months and 6 years old.¹⁷

The IIS was also used to produce immunization coverage summary reports. Monthly immunization compliance run charts (compliance rates graphed over time) were created for each health center individually and overall. Run charts for each center were shared with all practitioners as process performance feedback. All practitioners participated in at least 3 training sessions to learn how to interpret IIS records, strategize intervention implementation, and discuss run chart trends.

Expanding Access in Clinical Settings

To expand access to immunizations, practitioners at all health centers began immunizing at unscheduled “sick” visits in addition to well visits. Another appointment was scheduled for children receiving immunizations at sick visits who were also due for a well examination. Parents were only given health certificates (required for child care and school enrollment) once the child completed a well examination. Access to immunizations was also increased at health centers C and F with immunization-only clinics (held twice during influenza season at center C and monthly at center F).

Standing Orders

Because of the medical complexity of some of our patients, we chose to have a physician involved in all immunization-only visits. As a modified version of standing orders, nurses could initiate immunization orders for “shots-only visits,” but physicians approved all orders before immunizations were administered.

Client Reminder/Recall Systems

We sent reminder autodialer calls monthly to children due for immunizations. Because of our large patient population, we focused on children <24 months old. To recall children overdue for immunizations, we made least 2 live phone call attempts. When a parent answered, we attempted to schedule an appointment. If we were unable to reach a family via the telephone, we sent a postcard. Because outreach for overdue immunizations was labor intensive, we limited this activity to children <60 months old.

Educational Interventions

We created informational posters with pictorial examples of vaccine-preventable diseases, school immunization requirements, and immunization myths and facts. Because physical space varied among our health centers, centers A, B, and C placed posters in all examination and waiting rooms, whereas D had them in the procedure room, and E had them in the triage room. As an additional educational intervention, forms were created for practitioners to document for parents the immunizations their child received that day and the due date for future immunizations.

Vaccination Programs in WIC Settings

WIC is a federally subsidized nutritional program for pregnant women, infants, and children. For our intervention, we prescreened the IIS record for every WIC client <6 years old ahead of WIC appointments at the center colocated with health center C. We then asked WIC personnel to refer underimmunized children to our health center to receive same-day immunizations immediately after their WIC appointment.

Evaluation Methods

We asked immunization champions to assess the frequency with which activities were implemented at their site. Other process measures we tracked included the monthly number of calls for due immunizations that connected to a person or voicemail, and the number of phone calls made and postcards sent for overdue immunizations. We estimated the number of provider reminders generated by using billing data to determine the number of patient visits that occurred in the date range that IIS records were generated.

We measured the outcome of our QI program by using a before-and-after observational approach. Namely, the DC DOH evaluated the immunization coverage of our health centers (done routinely for all DC participants of the Vaccines for Children Program) by using the CDC’s Comprehensive Clinic Assessment Software Application (CoCASA) (earlier versions known as CASA). An immunization was considered valid only if it was administered at the recommended age/interval per the Advisory Committee on Immunization Practices (ACIP). Immunization rates for the health centers were compared at 3 time points: spring 2006 (preintervention), spring 2008 (after all QI activities were deployed), and spring 2009 (18 months postintervention initiation). Data were first imported from the DC IIS into CoCASA. For any child found to be overdue according to the IIS, the health center’s medical chart was reviewed. Any additional immunizations recorded in the chart were added into CoCASA (and the IIS).

Analysis

Our primary outcome was immunization coverage for 19- to 35-month-old children for the 4:3:1:3:3:1:3 series. Although ACIP recommended four doses of PCV7 at the time, we chose the 4:3:1:3:3:1:3 series because our health centers experienced frequent shortages of PCV7 during the national shortage in 2004.¹⁸ The 4:3:1:3:3:1:3 series was a key series assessed by the CDC in 2008 for children ages 19 to -35 months.¹⁹

Secondary outcomes were also assessed to determine the timeliness of immunization and sustainability of results. To examine the effect of the QI initiative on timeliness, we compared series completion rates among 19- to 24-month-old children in 2006 and 2008. This age range, which is 0 to 6 months beyond the upper age limit of when the 4:3:1:3:3:1:3 vaccines are due, is how the CDC’s Immunization Program Evaluation reporting requirements define “on time.” To determine sustainability, coverage for 2009 was compared with immunization coverage for 2008. All outcomes were considered as dichotomous variables (series complete or not). Comparisons were made by using the χ² test.

We performed analyses including all patients for whom 1 of our health centers was the provider of record in the IIS. We also performed analyses excluding all patients we deemed a MOGE (moved or gone elsewhere). We defined MOGE as any child with documentation of (1) a move out of the area, (2) a change to a health care provider not affiliated with one of the study health centers, or (3) no clinical encounter in >12 consecutive months for whom all outreach attempts were unsuccessful (wrong number/phone disconnected and postcard returned to sender/no response). The second set of analyses more accurately reflects immunization coverage for patients who used our health centers for primary care services versus urgent care only, because providers in the IIS are assigned solely according to which institution administered the child’s most recent immunization. Health center D, located at the main hospital, is regularly used as an urgent care clinic by children with other primary care providers. Incorrect IIS provider assignments to health center D became a more common occurrence when we began immunizing children at sick visits (regardless of whether they identified other primary care providers) as part of our QI program.

Results

Patient Population

There were 63 524 total patient visits overall in 2006 vs 71 935 in 2008, an increase of 12% (P < .0001). In 2006, 49 782 (78%) patients were publicly insured compared with 60 921 (85%) in 2008 (P < .0001).

Process Measures

In 2006, IIS records were not preprinted for any patient encounters. In 2008, an IIS record was preprinted for practitioners’ reference for virtually every scheduled patient encounter (41 685 records) at all centers, and ∼25% (7290 records) of all unscheduled “sick visit” encounters (records were not preprinted for all unscheduled encounters because of limited QI staff).

Before the study, in 2006, we did not send any reminder/recall phone messages or mailings. In 2008, we sent out 27 651 autodialer messages for due immunizations, and made 10 253 live phone calls and sent 523 postcards for overdue immunizations.

In 2006, personnel at WIC centers colocated with our health centers did not check their clients’ vaccine compliance, but did ask parents to bring immunization records to every visit that they then forwarded without review to the IIS. In 2008, during a 3-week trial, we checked the IIS record of 370 WIC clients. In this group, 21 children were overdue for immunizations. Of these, 1 child received same-day immunizations, 3 parents refused, and 17 did not present to their WIC appointments. Because of low yield, we did not continue this intervention.

Outcome Measures

In 2006, the health centers were the primary care providers for 1999 children ages 19 to 35 months compared with 1946 children in 2008 (ie, MOGEs were excluded both years). Immunization coverage for the 4:3:1:3:3:1:3 series improved from 71% to 87% (P < .0001) (Table 3). Coverage increased significantly at all six health centers. In addition, timely vaccination rates improved from 65% to 79% (P < .0001) and increased significantly at 4 of the 6 centers, including the largest and smallest (Table 4).

TABLE 3.

Vaccination Coverage for 4:3:1:3:3:1:3 Vaccination Series Among Children Aged 19 to 35 Months

Health Center	Series Complete in 2006 n (%)	Series Complete in 2008 n (%)	P	Change in Coverage From 2008 to 2009, %	P
A	61 (70)	105 (82)	<.05	+7	.11
B	159 (74)	151 (85)	<.01	+7	.02
C	866 (68)	939 (86)	<.0001	+1	.48
D	168 (79)	213 (86)	<.05	−1	.75
E	128 (82)	205 (94)	<.001	−3	.23
F	37 (59)	80 (89)	<.0001	+5	.12
Overall	1419 (71)	1693 (87)	<.0001	+1	.08

TABLE 4.

Vaccination Coverage for 4:3:1:3:3:1:3 Vaccination Series Among Children Aged 19 to 24 Months

Health Center	Series Complete in 2006 n (%)	Series Complete in 2008 n (%)	P
A	57 (66)	94 (73)	.21
B	146 (68)	143 (81)	<.01
C	791 (62)	855 (78)	<.0001
D	156 (74)	195 (79)	.18
E	120 (77)	193 (89)	<.01
F	26 (41)	72 (80)	<.0001
Overall	1333 (65)	1552 (79)	<.0001

We also performed data analysis with all patients for whom our centers were the immunizer of record in the IIS (including all MOGEs as not up to date, data not shown). This larger data set included 2232 children aged 19 to 35 months in 2006 and 2250 children in 2008. Immunization coverage overall for the 4:3:1:3:3:1:3 series improved from 64% to 75% (P < .0001). Immunization rates increased significantly at 5 of 6 health centers (P < .05) with the increase at center E reaching borderline significance (P = .07). In addition, timely immunization rates improved overall from 58% to 69% (P < .0001). Rates of timely immunization increased significantly at health centers B, C, D, and G (P < .05), and was borderline significant at center E (P = .07).

To determine whether gains in immunization coverage were sustained, we assessed compliance rates in 2009. In 2009, the health centers were the primary care providers for 2327 children aged 19 to 35 months (excluding MOGEs). Immunization coverage overall was 88%. This 1% improvement over 2008 coverage levels was borderline significant (P = .08) (Table 3). None of the individual health centers experienced a significant decrease in immunization rates, and, in fact, center B improved by 7% (P = .02). Reexamining the data with all patients for whom our centers were the immunizers on IIS record, in 2009, the health centers treated 2766 children aged 19 to 35 months. Overall, with this larger data set, we could not detect a statistically significant change in immunization coverage from 2008 to 2009 (75% vs 74%, respectively, P = .44).

Discussion

Our health centers implemented a comprehensive immunization best practices program based on an established theoretical framework. The cumulative effect was a 16% increase in immunization rates overall and a 14% increase in timely immunization. This occurred despite immunization rates in DC—whose overall childhood population is similar to our patient population, with the exception that ours includes more racial minorities and is more economically disadvantaged—remaining statistically unchanged (78% in 2006,²⁰ 77.6% in 2008,²¹ and 75.0% in 2009¹) over the same period.

All 6 health centers improved, including one with 3 practitioners and another with >25 practitioners. Gains were sustained beyond 18 months and were achieved by using the CoCASA software, which excluded any immunization administered before ACIP-recommended minimum ages/intervals. This is a more rigid method than that used in the annual CDC survey, which accepts all doses administered.^22,23 We achieved and sustained an 87% coverage rate despite national shortages of PCV7 for 8 months in 2004¹⁸ and H influenzae type b vaccine for 18 months from 2007 to 2009.²⁴

We implemented 6 Task Force recommendations and supplemented these recommendations with an additional activity: collaboration with community stakeholders. According to the CCM framework, interrelationships among health care institutions and the community are important for improving patient outcomes. However, because provider-community linkages require the cooperation of nonrelated groups, it is the least implemented of all CCM elements.²⁵ Collaborating with the DC DOH enhanced our feedback for practitioners and allowed us to create practitioner reminders. Collaborating with the Medicaid plans improved our outreach abilities.

During program implementation, all our health centers used paper records. Much time was spent reviewing charts to determine patients’ immunization status and printing IIS records for our large patient panel. Since the end of the study, we have transitioned to an electronic medical record system that has facilitated determination of a child’s immunizations status, because the information is all in 1 location. Ideally, an electronic record system would also facilitate searching for unimmunized children and sending automated phone messages to reduce the work burden of implementing client reminders, thereby rendering activities sustainable with less manpower.

We found that most WIC clients at our trial location were already immunization compliant. However, our findings may not be generalizable, because the WIC staff at the trial location, while not specifically checking a child for immunization compliance, requested that parents bring immunization records to every visit to forward to the IIS. Requesting immunization records at every appointment goes beyond what is nationally mandated, which is only to count four diphtheria-tetanus-acellular pertussis vaccinations by 24 months old and only at certification visits.²⁶ However, having WIC personnel request immunization records at every encounter may increase IIS completeness and remind parents of the importance of immunization.

Our improvement rates are consistent with those of similar projects. For instance, Melinkovich et al²⁷ increased immunization compliance 47% among young children over an 11-year span in Denver by using provider reminders and feedback, staff education, standing orders, and patient reminders. Some of the gains may have been due to secular trends, because immunization rates improved 30% in Colorado over the same time frame.^20,28 A recent Cochrane review found that patient reminder systems with and without other interventions increased immunization rates from 1% to 20% in randomized controlled trials.²⁹ These studies included various combinations of patient reminders, home visits, door-to-door campaigns, patient tracking, outreach, provider prompts, and client-held records.²⁹

Our study has limitations. We did not include a standard care comparison group. However, immunization rates in DC overall were similar to our baseline rates and remained constant during our study years.^1,20,21 Some may consider that we tested the effectiveness of a comprehensive QI program to be a limitation, because we cannot gauge the relative impact of each intervention. However, the efficacy of each intervention has already been well established and summarized by the Task Force.^10,11 In our estimation, and in concordance with the CCM, it is a strength of our QI program that interventions were implemented jointly, because they affect different aspects of the patient care spectrum and likely acted synergistically.

Conclusions

We found that our pediatric health centers were able to implement and sustain immunization best practices interventions over 18 months. By instituting these interventions as part of a comprehensive, theory-driven, and evidence-based approach to quality improvement, we increased immunization rates >10% with a predominantly minority and publicly insured patient population.

Glossary

ACIP

Advisory Committee on Immunization Practices

CCM

Chronic Care Model

CDC

Centers for Disease Control and Prevention

CoCASA

Comprehensive Clinic Assessment Software Application

IIS

immunization information system

MOGE

moved or gone elsewhere

PCV7

7-valent pneumococcal conjugate vaccine

QI

quality improvement

WIC

Women, Infants and Children