Improving Timely Childhood Immunizations through Pay for Performance in Medicaid-Managed Care

Abstract

Objective

To evaluate the impact of a “piece-rate” pay-for-performance (P4P) program aimed at rewarding up-to-date immunization delivery to 2-year-olds according to the recommended series.

Data Sources/Study Setting

Plan-level data from New York State's Quality Assurance Reporting Requirement and claims data from Hudson Health Plan for 2003–2007. In 2003 Hudson Health Plan, a not-for-profit Medicaid-focused managed care plan, introduced a U.S.$200 bonus payment for each fully immunized 2-year-old and provided administrative supports for identifying children who may need immunization. This represented a potential bonus of 15–25 percent above base reimbursement for eligible 2-year-olds.

Study Design

Case-comparison and interrupted times series.

Principal Findings

Immunization rates within Hudson Health Plan rose at a significantly, albeit modestly, higher rate than the robust secular trend noted among comparison health plans. Supplementary analyses suggest that there was no significant change in preexisting disparities during the study period, and that children with chronic conditions have significantly greater odds of being fully immunized during the entire study period.

Conclusions

This study suggests that a piece-rate P4P program with appropriate administrative supports can be effective at improving childhood immunization rates.

Despite widespread use of pay for performance (P4P) among commercial and government-sponsored health plans (Rosenthal et al. 2006; Kurhmerker and Hartman 2007;), there remains considerable uncertainty about how best to design and implement these programs (Dudley et al. 2004; Rosenthal et al. 2005; Petersen et al. 2006;). The Patient Protection and Affordable Care Act (PPACA) and Child Health Insurance Program Reauthorization Act (CHIPRA) contain legislation that will intensify the use of performance incentives (United States Congress 2009a, b). Experiments in payment policies will potentially build the evidence base for how to best use performance incentives to achieve higher quality, lower costs, and equity in the U.S. health care system.

One area of uncertainty is the formula through which performance is translated into additional revenue. Many programs use “tournaments” or fixed achievement thresholds to determine whether providers merit additional pay (Centers for Medicare and Medicaid Services 2005). A theoretical argument exists, and some empirical evidence supports the notion that this model rewards providers who are already performing well while providing little incentive for quality improvement (Rosenthal et al. 2005; Chien et al. 2007a; Hayward 2007; Karve et al. 2008; Friedberg et al. 2010;). Paying providers a “piece-rate” for each patient meeting a performance benchmark (e.g., each diabetic receiving HbA1c testing twice a year) is one way to reward continual improvement. This payment scheme theoretically rewards continuous improvement in performance regardless of the baseline levels. It may also minimize incentives for providers to avoid high-risk or nonadherent patients from their practices or panels and thereby reduce the risk of exacerbating disparities (Hofer et al. 1999).

Piece-rate performance incentives have been studied a handful of times particularly in conjunction with improving childhood immunization or well-child visit rates (Chien, Conti, and Pollack 2007b). Nearly 15 years ago, two randomized studies concluded that piece-rate P4P rewards were effective at improving immunization documentation in urban practices serving low-income patients (Fairbrother et al. 1999, 2001). These studies were performed at a time when the Centers of Disease Control (CDC) measured immunization rates using the 4 : 3 : 1 immunization series (four doses of vaccines against Diphtheria–Tetanus–Pertussis, three doses against Polio, and one dose of protection against Measles–Mumps–Rubella). Between 1999 and 2002, immunization recommendations evolved significantly to incorporate the development of new vaccines and guidelines for “catching up” children who experienced vaccination delays. By 2003, the CDC began measuring progress toward national childhood immunization goals using the 4 : 3 : 1 : 3 : 3 : 1 immunization series (the vaccines in the 4 : 3 : 1 series plus three doses against Haemophilus influenzae Type B, three against Hepatitis B, and one against Varicella Zoster Virus). This rate continues to be a core metric by which national, state, and local health agencies gauge their progress (Center for Disease Control and Prevention 2009). A recent study of piece-rate P4P evaluated the effectiveness of this strategy in raising attendance levels at the well-child visits in which the 4 : 3 : 1 : 3 : 3 : 1 immunization series is delivered. This study found that the two Medicaid-sponsored programs that raised rates more than the Medicaid national mean had combined their incentives with programmatic adjuncts like patient incentives and reminders (Felt-Lisk, Gimm, and Peterson 2007). “Piece-rate” payment strategies have a small but promising empirical base that warrants more rigorous examination.

This study takes advantage of a natural experiment that occurred within the Medicaid health plans in New York State. One health plan introduced a distinctive piece-rate P4P program aimed at improving the childhood immunization rates as measured by the 4 : 3 : 1 : 3 : 3 : 1 series, while others attempted more conventional tactics (e.g., reminders and outreach to providers and parents without incentives) (Jacobsen Van JC 2005). Patient-level data allow us to explore whether the P4P program is associated with changes in immunization rates for specific patient subgroups. Although previous studies have not been able to examine whether children of racial/ethnic minority backgrounds or with chronic health conditions are differentially impacted by P4P programs, it is known that there is a paradoxical racial/ethnic disparity in immunization rates among those who live below the federal poverty line—Hispanics have the highest immunization rates (followed by blacks then whites) (Center for Disease Control and Prevention 2009).

METHODS

Study Setting

Hudson Health Plan (Hudson) is a not-for-profit Medicaid and State Children's Health Insurance Program (SCHIP)-focused managed care plan contracting with approximately 115 eligible medical practices to serve about 81,000 members in six Hudson Valley counties (Westchester, Rockland, Orange, Ulster, Sullivan, and Dutchess). Half of these are children aged 0–18 years. Of the eligible practices, about 65 percent are solo or small practices and 35 percent are medium (4–9 full-time physicians) to large-sized (10 or more full-time physicians) practices. Smaller practices tended to be privately owned, whereas larger ones tended to be hospital-based or federally qualified health centers. On average, half of Hudson members are Hispanic, with the remainder divided relatively evenly between blacks, whites, and those of other race/ethnicity.

In the spring of 2003, Hudson introduced its piece-rate P4P program. This program used compliance with the 4 : 3 : 1 : 3 : 3 : 1 immunization series as its main performance metric. The performance measure was identical to New York State's Quality Assessment and Reporting Requirements (QARR) for childhood immunizations and the Health Plan Employer Data and Information Set's (HEDIS') 2-year-old immunization measure during the study period (New York State Department of Health [NYS DOH] 2005). Of note, Prevnar was excluded from performance measurement because of significant local and national vaccines shortages.

Hudson's program was remarkable in that earnings at risk represented a potential increase of 15–25 percent above base reimbursement in the care of 0–2-year-olds, well above the 10 percent estimated to be necessary to induce practice change (Baker and Carter 2005). It also provided practices with the level of administrative support thought to be important in raising well-child visit rates (Felt-Lisk, Gimm, and Peterson 2007). Financially, Hudson's program rewarded practices in two tiers: U.S.$100 for each 2-year-old who was fully immunized by the child's second birthday, and an additional U.S.$100 if the immunizations were administered in compliance with HEDIS 2003 specifications for timeliness. Administratively, Hudson provided practices with monthly lists of patients who had turned two in the prior month and quarterly reports summarizing each practice's immunization rates. In return, Hudson required practices to fax or mail copies of charts, immunization cards, and/or New York State immunization registry documents; they did not consider handwritten attestations sufficient. Hudson paid over U.S.$1.0 million of a potential U.S.$2.6 million in P4P bonuses across the study period 2003–2007.

Study Design, Data, and Analytic Approach

We use two quasi-experimental study designs and corresponding data sources to evaluate the effectiveness of Hudson's P4P program.

First, we used a case-comparison difference-in-difference study design to compare Hudson's immunization rates with those of other Medicaid-focused health plans within New York. For this aspect of the study, we used health plan-level QARR data, assembled by NYS DOH from all certified managed care health plans operating within the state. These data are subject to a full audit of administrative claims records and patient charts by a National Committee for Quality Assurance (NCQA)-certified auditor. Unfortunately, race/ethnicity data were not incorporated into the data made available through QARR.

To analyze the plan-level data from New York State's QARR program, we compared Hudson Health Plan's immunization rates with non-Hudson Medicaid plans outside of Hudson Valley (N = 16). We used the geographical boundaries of Hudson Health Plan to minimize the likelihood that our comparison group would be contaminated by spillover effects from the Hudson's P4P program. We also systematically examined the two largest publicly available repositories of health plan quality improvement efforts to determine whether the health plans in our comparison group reported using P4P strategies in a way that could impact 2-year-old immunization rates (NYS DOH 2007; Leapfrog Group 2009;). This investigation found that none of our comparison health plans tied P4P incentives to 2-year-old immunization rates. One plan did, however, tie P4P incentives to the well-baby visit rate for 15-month olds; another directed similar rewards to improving lead screening rates for 25-month olds. Because these efforts could indirectly raise immunization rates, we conducted sensitivity analyses around their inclusion or exclusion in our comparison group and found no difference in our results. Lastly, we examined NYS DOH data on health plan characteristics and found that Hudson was similar to the health plans in our comparison group with respect to the proportion of enrollees being children and annual fluctuations in enrollment.

Second, we conducted an interrupted time series study of 4 : 3 : 1 : 3 : 3 : 1 immunization rates using the claims that were the basis for payment. We analyzed the trends in immunization rates among children who were continuously enrolled in Hudson for at least 365 days with a maximum gap of 45 days. We focused on children whose insurance status was stable because this may confer a greater ability to identify and meet immunization and other medical needs. This approach was informed by the results from our practice survey (data not shown) which indicated that practices focused their immunization efforts on patients during clinic visits (e.g., educating them during regularly scheduled appointments), as opposed to performing more outreach (e.g., telephoning them or mailing reminders when immunizations were due). In this analysis, the key independent variables of interest concerning time were yearly and monthly time trends. Given the attenuation of immunization rates observed in plan-level data between 2005 and 2007, we tested two alternative time specifications, one with indicators for each year in the postintervention period and the other indicative of a quadratic trend (i.e., a monthly time trend and its square were included in the model).

As part of this analysis, we explored available patient and practice variables that could potentially impact immunization rates. In terms of patient factors that could influence immunization rates, we examined two factors—health status (diagnosed with a chronic health condition versus not) and race/ethnicity. We defined the presence of a chronic condition using a previously described method based on International Statistical Classification of Diseases and Related Health Problems (ICD-9) codes. Any child with an encounter carrying a diagnosis code consistent with conditions such as asthma, epilepsy, or cancer was considered to have a chronic health condition, while those without were considered “healthy” (Gallaher, Christakis, and Connell 2002). We defined patient race/ethnicity using available nonnested variables within the Hudson Health Plan's data, which reflect data obtained from members at enrollment. We expected children diagnosed with chronic conditions at an early age to have more opportunities to receive vaccinations because they generally use the health care system more often and because there may be a greater imperative among clinicians to ensure that these children receive their immunizations. We hypothesized that immunization rates could differ by race/ethnicity because this could reflect underlying attitudes toward or concerns about childhood vaccines. As part of this analysis, we examined whether expected preexisting paradoxical racial/ethnic disparities in immunization rates changed over the study period.

Regarding practice characteristics, we examined whether practices with greater numbers of Hudson enrollees or private practices may be more responsive to the P4P program. This is because greater numbers of members may represent greater potential earnings. Also, patients being cared for by private practices may have higher immunization rates because these practices may tend to serve patients with somewhat higher socioeconomic status because of their extremely limited ability to accept uninsured patients compared with federally qualified health centers who are mandated (and funded) to care for all patients irrespective of their ability to pay.

Lastly, we also evaluated two potential mechanisms by which immunization rates change among continuously enrolled children. Using ICD-9 and current procedural terminology (CPT) codes, we examined whether changes in immunization rates were associated with a greater number of visits per eligible child or a greater number of shots delivered per patient visit.

We use patient-level logistic regression models that account for clustering within provider and by geography as well as serial correlation in the data. Odds ratios are reported for the regression analyses. The statistical significance of differential effects by race/ethnicity and chronic health condition were tested through simulation and bootstrapping of standard errors. Interaction effects were examined for patient race/ethnicity, category of enrollment, chronic health condition, practice type, and practice rank in the top quartile for the percent of eligible Hudson patients.

RESULTS

Sample Characteristics

Hudson insured a population of 2-year-olds that was predominantly Hispanic (55 percent) but also black (16 percent), white (14 percent), and of other (11 percent) racial/ethnic backgrounds. Medicaid financing covered 81 percent of the target population, with the remainder enrolled in the SCHIP. Solo and small private practices cared for about 60 percent of the population of interest, while medium- to large-sized practices (federally qualified health centers and hospital-based clinics) cared for the remaining 40 percent. Approximately one-third (38 percent) of Hudson's 2-year-old population had at least one chronic condition (Table 1).

Table 1.

Sample Characteristics

Hudson Health Plan 2-Year-Old Children Eligible for Childhood Immunization and Screening Measure (N = 4,429)	N (%)
Year of birth
2001	451 (10)
2002	711 (16)
2003	1,016 (23)
2004	1,047 (24)
2005	1,204 (27)
Race/ethnicity
Hispanic	2,439 (55)
Black (non-Hispanic)	722 (16)
White (non-Hispanic)	638 (14)
Other	142 (11)
Category of enrollment
Medicaid	3,572 (81)
State Children's Health Insurance Program	842 (19)
Cared for by
Private practice	2,665 (60)
Federally qualified health center	1,594 (36)
Hospital clinic	177 (4)
Has chronic condition*	1,701 (38)
Immunization status
Full but delayed immunization	2,259 (54)
Full and timely immunization	2,037 (46)

^*As defined by ICD-9 codes per method of Gallaher.

Overall Impact of Hudson's P4P Program

Immunization rates for 2-year-olds rose over the study period from ∼60 percent in 2003 to ∼80 percent in 2007 for all New York Medicaid health plans. The increase for Hudson patients was 7 percent greater than that of the comparison health plans when comparing 2003–2005, and 11 percent greater when comparing 2003–2007. The difference-in-difference comparison for these changes in immunization rates between Hudson and non-Hudson Health Plans is not significant when comparing 2003–2005 (p <.30) but is significant when comparing 2003–2007 (p <.01) (Figure 1).

Figure 1.

Immunization Rates: Hudson versus Non-Hudson Medicaid Health Plans

Notes. Data based on plan-level claims records supplemented by audited chart review.

* Immunization trend 2003–2007 significantly greater for Hudson Health Plan p <.01.

Impact of Patient and Provider Characteristics on Immunization Rates within Hudson Health Plan

Timely and delayed immunization rates for continuously enrolled 2-year-olds did not significantly change over the first 3 years of the intervention (2003–2006), but it declined significantly in the fourth year (Table 2, 2007 trend OR=0.47–0.60, p <.001). Patients with chronic conditions, of Hispanic ethnicity, or being cared for by private practices and by practices with a high number of patients enrolled in Hudson Health Plan were more likely than their counterparts to be fully immunized in either a timely or delayed fashion. Patients of black race or insured by SCHIP were not more or less likely to be immunized than patients of white race or Medicaid insurance, respectively. As expected, immunization rates determined by Hudson's claims data augmented by voluntary reporting by the practices were systematically lower than those based on QARR data, which supplements claims data with patient chart review for all sampled patients.

Table 2.

Hudson Patient and Practice Predictors of Full Immunization

	Full but Delayed Immunization† Odds Ratio (SE)	Full and Timely Immunization† Odds Ratio (SE)
2004	0.99 (0.20)	1.09 (0.20)
2005	0.88 (0.18)	0.95 (0.17)
2006	0.87 (0.17)	0.96 (0.17)
2007 versus 2003	0.47 (0.10)***	0.60 (0.12)**
Insured by Medicaid versus State Children' Health Insurance Program	1.14 (0.11)	1.15 (0.11)
Cared for by private practice versus federally qualified health center	1.49 (0.20)**	1.52 (0.21)**
Number of Hudson patients	1.73 (0.23)***	1.65 (0.24)***
Race/ethnicity
Hispanics	1.52 (0.14)***	1.38 (0.15)**
Blacks (non-Hispanic) versus whites (non-Hispanic)	1.09 (0.12)	1.10 (0.13)
Has chronic illness versus not	1.24 (0.12)**	1.20 (0.08)**

^†Children continuously enrolled for 365 days with a gap of less than 45 days.

^*p <.05

^**p <.01

^***p <.001.

Children with chronic conditions or of Hispanic ethnicity had significantly greater odds of being immunized across the study period (Table 2, chronic OR=1.20–1.24, p <.008 and Hispanics OR=1.38–1.52, p <.002, respectively). Children being cared for by practices with the highest number of Hudson Health Plan members or by private practices were also significantly more likely to be immunized (Table 2, high number of Hudson enrollees OR=1.65–1.73, p <.001, and private practice OR=1.49–1.52, p = .003). These associations were the same irrespective of whether we were measuring delayed or timely immunization.

No Change in Disparities

Preexisting racial/ethnic differences among Hispanics, blacks, and whites fluctuate across the study period but remain essentially unchanged over the study period. Hispanics have and maintain the highest immunization rates by 4–5 percentage points in 2003, followed by blacks whose immunization rates are in turn 4–5 percentage points higher than whites. Disparities appear to narrow across the first 2 years such that rates are essentially identical in 2005, but then widen such that Hispanics have higher immunization rates than blacks or whites at the end of the period.

No Change in Visits per Child or Immunizations per Visit

Our attempt to better understand the two main mechanisms by which immunization rates could have changed was negative. For continuously enrolled children, there were no significant changes in the number of visits per eligible child or the number of shots delivered per patient visit (data not shown).

DISCUSSION

Immunization rates within a health plan that implemented a robust piece-rate P4P program rose at a significantly higher rate than among health plans that did not, despite robust secular trends. The piece-rate P4P incentives did not negatively impact children with chronic conditions or exacerbate disparities. This finding is significant in light of the concern that P4P may induce providers to avoid medically or socially complex patients—those with chronic health conditions or with minority racial/ethnic backgrounds. Of note, while the piece-rate P4P program did not exacerbate existing disparities within a Medicaid population, it did not mitigate them either.

This study allowed us to examine, but did not shed significant light on, the mechanisms by which improvements were achieved. Our exploratory analyses of patient-level claims data for continuous enrollees showed that patients actually experienced no significant change in visit number, number of shots per visit, immunization rates, or preexisting disparities. The lack of a significant finding may be because we could not assess visit rates change for children who are not continuously enrolled. It is possible that gains in immunization rates were concentrated among children who recently obtained Medicaid insurance. The increasing number of combination vaccines may also mask more substantial changes in immunization adherence (e.g., a single shot can bring a child with up to three delayed immunizations into compliance).

Our study demonstrates many strengths relative to the existing literature. We observed a robust piece-rate P4P intervention with substantial administrative supports (e.g., the provision of patient lists to practices) over a 4-year period, and we obtained within-state comparison health plan quality data, thus minimizing artifacts from state-to-state variations in Medicaid eligibility, coverage, and reimbursement.

There are, however, three key study limitations. We only have aggregate data for our comparison group. Although our concerns about immunization rates being affected by variations in state Medicaid laws or practices are mitigated, we are unable to pool the intervention and comparison group data and conduct a patient-level analysis to assess potential changes in the patients or practices that may affect immunization rates within New York over the study period. Secondly, our evaluation starts at the end of the same year that the intervention begins. We were not able to obtain health plan immunization rates before 2003 because plans were being benchmarked according to different immunization guidelines (e.g., the 4 : 3 : 1 : 3 or 4 : 3 : 1 : 3 : 3 series rather than the 4 : 3 : 1 : 3 : 3 : 1 series instituted in 2003). Data from 2003 are the best baseline possible given data constraints and because large-scale interventions like this take time to implement. We were not able to measure, however, a preintervention trend and look for a deviation from prior trends. Lastly, the data used for this study were not amenable to differentiating between better documentation and more complete immunization of the target population as has been noted in previous studies (Fairbrother et al. 1999, 2001). Differentiating between quality documentation and quality of care is an important distinction, but we have reason to believe that the overall quality of immunization documentation is more accurate and reliable than 15 years ago when previous studies noted that documentation and underlying care were dissociated. By 2003, proper immunization documentation was required and enforced to such a degree that under-documentation could be considered tantamount to poorer quality care. A short list of the substantial and sustained state and federal immunization efforts that have been implemented since the early P4P studies include the following: (a) Vaccines for Children (VFC) program; (b) the coordinated effort to update the recommended childhood immunization schedule by three main professional societies (i.e., Advisory Committee on Immunization Practices [ACIP], American Academy of Pediatrics [AAP], and American Academy of Family Physicians [AAFP]); (c) the passing of state laws requiring immunization documentation before school entry in all 50 states; and (d) advances in quality reporting within state departments of health (e.g., New York's QARR program). All of these programs require, verify, or enforce proper immunization documentation and most would agree that clinicians should base their treatment decisions on what is actually documented (i.e., if it was not documented, it was not done).

Aligning incentives with care quality is a key component of health reform (PPACA) generally, and child-oriented legislation (CHIPRA) specifically, and empirical evaluations may help identify effective payment strategies. This study adds to the existing literature that supports the effectiveness of piece-rate P4P programs but also underscores another pervasive theme: P4P alone is not a panacea due to the many patient and provider factors that affect performance. Ultimately, health plans must decide which interventions may be most cost-effective for their organization, the practices with which they contract, and their enrollees. Notably, given the modest but significant success of its P4P program, Hudson has chosen to continue in this vein and supplement it with patient-directed incentives.

SUPPORTING INFORMATION

Appendix SA1: Author Matrix.

Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.