A pediatric-focused review of the performance incentive literature

Abstract

Purpose of review

This article seeks to provide a pediatric-focused review of the performance incentive literature. The article will begin with an overview of the performance incentive literature within and outside of healthcare. The review will then detail the pediatric-specific literature, reflect on the breadth and focus of this literature compared with that for adult programs in medicine and school/teacher accountability efforts, and identify three concerning knowledge gaps.

Recent findings

The pediatric performance incentive literature is small, totaling five empirical studies. This literature indicates that performance incentives alone have not been effective at improving incentivized aspects of pediatric healthcare. Only one study evaluates whether pediatric performance incentives improve healthcare for children with significant health conditions. No studies investigate whether this strategy yields negative unintended consequences for children and adolescents or how risk adjustment can augment pediatric performance incentive efforts or attenuate unintended effects.

Summary

If approached to participate in or inform performance incentive efforts, pediatricians should appreciate the limitations of the existing empirical literature and ask proponents about their plans for monitoring and/or guarding against potentially negative unintended consequences. Numerous stakeholders are optimistic that this strategy will improve healthcare quality, but the empirical evidence suggests that skepticism is appropriate.

Introduction

Pediatric performance incentive programs are proliferating. Among the nearly 100 publicly listed performance incentive programs, 33 employ a performance measure applicable to children and/or adolescents [1]. Half of the 50 State Medicaid programs use performance incentives and 85% expect to do so in the next 5 years [2^•,3^•]. Inpatient efforts, especially in neonatal intensive care units, are also being contemplated [4^•].

Performance incentives – cash bonuses or penalties, and public reporting – explicitly link financial or reputation-based rewards and sanctions to specific healthcare processes or outcomes [5]. Public and private stakeholders are pursuing this strategy because they recognize that American healthcare finance provides limited incentives for quality [6,7], and believe that different tactics are needed to promote quality healthcare [8]. While a full discussion of the theoretical basis of performance incentives is beyond the scope of this article, basic features of performance incentive programs are detailed as part of Table 1.

Table 1.

Basic features of empirically evaluated pediatric performance incentive programs

			Incentive typea			Incentive triggersb			Type of care being incentivized			Other tactics incorporated into pay-for-performance interventionc
Program setting	Payor	Payee	Cash bonus	Cash penalty	Public reporting	Tournament	Achievement	Improvement	Prevention	Inpatient	Chronic disease	Registry	Collaborative learning	Decision support	Performance feedback	Other
1993–1995 [23]
Philadelphia, PA	Commercial  health plan	Medical  groups	•			•		•	•						•
1995–1996 [24]
New York City, NY	NA	Physicians	•				•	•	•						•	•
1995–1996 [25]
New York City, NY	NA	Physicians	•				•	•	•						•	•
2003–2005 [26••]
Local Initiative   Rewarding   Results, CA	Medicaid	Physicians  and medical  groups	•				•		•
2004–2005 [27••]
Cincinnati   region, OH	Commercial  health plan	Physician–  hospital  organization	•				•				•	•	•		•	•

^aPerformance incentives are generally categorized into two types: financial and reputation-based. Financial incentives usually comprise cash bonuses or penalties. These incentives change a payee’s earnings by providing a lump-sum bonus, imposing a penalty fee, or by manipulating per member per month reimbursement rates. The magnitude of payments or penalties is usually set by the payor, but it is generally agreed that changes must represent at least 10% of total revenue to induce the desired behavior. Reputation-based incentives involve publicly reporting information about the quality of healthcare that the payee is providing. Additional information about healthcare quality is thought to allow purchasers to make more informed decisions, thus promoting market demand for higher quality.

^bIncentive triggers determine when a cash bonus or penalty is warranted. In tournaments, only the top few performers earn the reward (e.g., 10 hospitals may score in the 95th percentile, but only the top two will earn the bonus). In achievement-based programs, anyone reaching a predetermined benchmark will earn the reward or avoid the sanction. In improvement-based programs, only those improving a certain increment (e.g., 10–15%) will earn the reward or avoid the sanction.

^cA variety of organizational tactics are used in conjunction with pay-for-performance programs. These include: disease registries (keeping lists of patients with certain conditions to better track their care and follow-up), collaborative learning (where members of entirely separate hospitals or practices come together to share similar problems and solutions regarding healthcare quality), decision support (paper or electronic tools that remind providers of care guidelines), performance feedback (where providers are given reports of how they are performing on targeted metrics relative to benchmarks or other providers), or other organization-level interventions (e.g., whether vaccine is supplied upfront rather than in arrears, or formal quality-improvement techniques). NA, not available.

By the end of the article, pediatricians will have a solid grasp of the existing literature so that they can decide how to participate in or inform performance incentive strategies targeting pediatric healthcare.

The empirical evidence remains equivocal regarding the ability of performance incentives to produce high-quality care

The current peer-reviewed literature on this subject comprises approximately 20 empirical studies (five of which are pediatric) [9,10^•,11^•,12–25,26^••,27^••]. Three groups of authors [28,29,30^•] have recently reviewed the bulk of this literature and arrived at similar conclusions. Approximately one-third of this literature finds performance incentives to have modestly significant effects [9,14–16], another one-third demonstrates weak or null effects [11^•,12,13,17,18], and the final one-third associates this strategy with negative unintended consequences [17,20–22,24,25]. These consequences include better documentation rather than improving underlying care [24,25], rewarding providers who are already doing well [17], and changing the willingness of healthcare organizations to care for socially and/or medically complex patients [20–22].

Evaluated programs have combined two main types of incentives, cash bonuses and report cards, with a range of incentive triggers. These triggers include whether payees achieve a specific benchmark (e.g., the percentage of patients receiving timely immunization), improve a specific amount on a specific quality measure, or outperform other providers on a specific dimension of care (also known as ‘tournaments’).

Each of these triggers has specific merits and drawbacks. Payment based on a specific absolute standard focuses most directly on the performance one seeks to achieve. Triggers based on program improvement that compensates payees may provide the strongest impetus for positive change among providers who may not currently comply with established benchmarks. Such approaches also limit the transfer of resources to providers who already comply with benchmarks and who fail to improve. Tournaments are valuable when it is difficult to anticipate the difficulties of improving program goals, but when multiple providers face similar challenges and patient populations.

It is difficult to discern whether the equivocal nature of research findings is due to performance incentives or contextual factors

Taken together, studied performance incentive programs have attempted to improve the quality of care for 23 different conditions and/or clinical situations [13 adult, 15 pediatric, and six overlapping – Primary and secondary prevention: immunizations for: diphtheria, tetanus, pertussis, Haemophilus spp., hepatitis, pneumococcus, measles, mumps, rubella, varicella, and influenza. Screening for: cancer (breast, colorectal, and cervical) and sexually transmitted diseases. Acute care: acute myocardial infarction, congestive heart failure, pneumonia, and coronary artery diseases. Chronic disease care: diabetes, tobacco addiction, substance abuse, and asthma]. These programs have also relied on metrics that range from care processes, presumably under a provider’s control (e.g., whether or not patients with persistent asthma are prescribed control medications), to care outcomes, not necessarily or fully under a provider’s control (e.g., whether or not patients with persistent asthma avoid emergency room visits and hospital admissions). The programs have combined two main types of incentives (i.e., cash bonuses and report cards) with a range of incentive triggers (e.g., whether a payee achieves a benchmark, improves a certain amount, and/or beats its competitors in a tournament), been sponsored by three very different purchasers of healthcare (Medicare, Medicaid, and commercial health plans), and been directed at a variety of healthcare providers: hospitals, medical groups, and individual physicians. It is not known how all these factors interact to promote or interfere with the effectiveness of this strategy.

Outside of healthcare, there is extensive literature on the use of performance incentives

One group of authors [31] has evaluated the effectiveness of performance incentives in industries analogous to healthcare – job training, executive compensation, and school/teacher performance. These industries were viewed as analogous to healthcare because payers cannot easily estimate a payee’s efforts, ascertain whether quality services were delivered, or determine whether desired outcomes were achieved. The authors [31] concluded that performance incentive strategies were not necessarily effective and could cause providers and educators to disregard those with more tailored needs (i.e., students whose academic performance was well above or well below that of the average student). In the incentive literature on job training, many contractors paid on the basis of students’ subsequent employment rates chose to serve candidates who were most likely to succeed and avoid candidates who were considered ‘difficult-to-place’ [32]. Researchers contributing to the analogous literature on school/teacher performance worry that performance incentives have the potential to widen race/ethnic disparities [33]. Recent research [34,35^•] suggests that current performance efforts may encourage schools to concentrate resources to children near the margin of rewarded benchmarks at the expense of children significantly ahead of or behind mandated performance levels.

Some authors have begun to provide pediatric-focused viewpoints on performance incentives

Some authors [4^•,36,37^•] have highlighted inherent difficulties related to designing and implementing any pediatric performance incentive program such as small sample sizes and a paucity of evidence-based process and outcome measures. Other authors [38] have called attention to how quality measurement and improvement activities may provide the basis for implementing performance incentives (e.g., quality improvement activity is a mandatory part of pediatric residency training and will be necessary for maintaining pediatric board certification).

The pediatric-specific performance incentive literature totals five empirical studies

Three of five studies of pediatric-specific performance incentive literature [23–25] evaluate programs that were implemented from 1993 –1996; a further two studies [26^••,27^••] examine programs implemented in the past 4 years (2003–2005).

Table 1 summarizes the basic features of these pediatric performance incentive programs. Commercial health plans offered cash bonuses to medical practice(s) in two of the studies [23,27^••]; Medicaid health plans provided similar incentives to a variety of payees (solo, small, and large group practices) in another study [26^••]. It is unclear who provided these incentives in the two remaining studies [24,25]. All programs used cash bonuses as part of their performance incentive package; none used cash penalties or public reporting. Whether the bonus amounts reached the size believed to be necessary for motivating change was unclear. In four studies [24,25,26^••,27^••], payees had to achieve a benchmark level of performance to receive a bonus; in the remaining study [23], payees earned the bonus in a tournament fashion (only top performers won the reward). Four programs [23–25,26^••] targeted preventive care: recommended number of well child visits, immunization rates, and routine screening for growth, abnormal blood pressure, proper vision and hearing, anemia, tuberculosis, bacteriuria, elevated lead levels, and/or dentistry referrals. One program [26^••] targeted evidence-based components of care for asthma (e.g., written self-management plan, children with persistent disease while taking control medications, vaccinations for influenza). Overall, these programs incentivized outcome measures more than process measures. Programs almost uniformly employed additional organizational tactics (e.g., performance feedback and how supplies such as vaccinations were distributed to healthcare providers). One program [26^••] involved an extensive quality improvement effort (disease registry, collaborative learning, formal quality improvement techniques) to more than 50 primary care sites for the year prior to the introduction of performance incentives.

Performance incentives have had an equivocal effect in pediatrics

The three older studies [23–25] had randomized study designs, while the two more recent studies [26^••,27^••] used quasi-experimental methods. In one of the randomized studies [23], performance incentives had a null effect. In the two remaining randomized studies [24,25], these incentives were associated with significant increases in targeted metrics, but these increases may have reflected more complete documentation by nurses rather than improved underlying care (i.e., taking advantage of urgent care visits to provide ‘catch-up’ vaccinations). In one of the quasi-experimental studies of performance incentive efforts undertaken by Medicaid health plans [26^••], only one of the five permutations produced changes that were significantly different from state and national trends. In the fifth and final study [27^••], practice-level performance incentives were associated with large improvements in the quality of asthma care, but the authors attributed the bulk of this change to a host of other quality improvement tactics rather than performance incentives themselves. Table 2 provides basic information about these study designs and their authors’ conclusions.

Table 2.

Study designs and conclusions of evaluated pediatric pay-for-performance programs

Study design	Study timing/duration	Evaluation endpoint(s)	Effect*	Author conclusions
Practices randomly assigned   to three groups [23]	18 months during  intervention	Percentage of children with:		Incentives may have been  ineffective because of  competing insurers,  providers unaware of  program, effect may have  been dwarfed by strong  secular trends
Control		Immunizations up-to-date  at 2–15 months	NS
Feedback only		Recommended number of  well visits at 2–6 years	NS
Feedback plus bonus		Appropriate well visit  content at 2–6 years	NS
Physicians randomly assigned   to three groups [24]	12 months during  intervention period	Percentage of children with:		Incentive improves  documentation not  underlying care
Control		Immunizations up-to-date  at 3–35 months	SS
Feedback plus bonus		Missed opportunities to vaccinate	NS
Feedback plus enhanced   fee-for-service		Vaccinations received  outside the practice	SS
Physicians randomly assigned   to three groups [25]	12 months following the  original intervention  period	Percentage of children with:
Control		Immunizations up-to-date  at 3–35 months	SS	Incentive improves  documentation not  underlying care
Feedback plus bonus		Missed opportunities to vaccinate	PS
Feedback plus enhanced   fee-for-service		Vaccinations received outside the practice	SS
Quasi-experimental [26••]		Percentage of:		Incentive may have contributed  to incremental improvements,  but only one program was  associated with a large impact
Control: plan, state and   national-level   administrative data	~30 months during  the intervention	Babies with ≥6 visits in  first 15 months of life	PS
Cases : variety of seven different   State Medicaid performance   incentive programs		Babies completing set  of 3 well child visits	PS
Quasi-experimental [27••]
Alludes to interrupted time series   analysis; presents some pre   and postintervention data.	18 months during  the intervention	Percentage of patients  with asthma:		Incentive is aligned with  practice- and network-level  quality improvement,  efforts may be effective
		With written self-management plan With persistent disease while  taking control medications Vaccinated for influenza	PS PS PS

^*NS, not significant; PS, partly significant; SS, statistically significant.

There are three major gaps in the pediatric performance incentive literature

A single study [27^••] explores how pediatric performance incentives impact healthcare for children with significant health conditions. The pediatric literature has primarily examined the effect of incentives directed at immunization delivery in the primary care setting. This is quite different from the adult performance incentive literature, which is split nearly evenly between studies of programs that target preventive care [12 –14,16–18] and those that aim to improve chronic disease or acute inpatient care [9,10^•,11^•,15,17,20].

Exclusive focus on the average healthy child may distract attention from those with significant health needs. Irrespective of whether performance incentives yield their intended effect, they can alter how pediatricians allocate their personal time and resources across different healthcare tasks. This issue is especially important among heterogeneous populations, and probably depends on how programs are operationalized. For example, if a performance incentive program directed at immunization compliance encourages providers to discover and remedy aspects of their practice that create barriers to up-to-date immunizations for everyone, then immunization rates could improve because everyone’s care is improved. Setting benchmarks could, however, cause providers to target those most easy to change – those only one or two immunizations away from being up-to-date or those most responsive to basic outreach efforts (e.g., reminder cards, phone calls). Under such incentives, aggregate immunization rates could improve, but there is little incentive for additional attention to children who have additional needs (e.g., catch-up series or translation services).

No studies evaluate pediatric performance incentive programs for unintended consequences. The nonpediatric and health-related literature indicate that performance incentives can reward those already doing well [17], widen race/ethnic disparities in care [22,33], or change providers’ willingness to care for socially and/or medically complicated patients [21,22,34,35^•]. These findings may be pertinent because, although adult-based performance incentive efforts have different payors, payees, patients, and health conditions, both pediatric and adult efforts take place within the medical context and share the common goal of patient health. Additionally, even though school/teacher accountability lies outside the realm of medicine and/or healthcare, their findings may be pertinent for understanding how organizations approach populations of generally healthy children and are one of the health outcomes that pediatricians care about, school achievement.

Discussions of risk adjustment are also absent from the existing pediatric performance incentive literature. Risk adjustment refers to the attenuation of healthcare metrics to more accurately reflect the actual health status or recent medical experience of patients; this statistical technique may help mitigate the selection effects created by performance incentive programs [39]. Adult-based efforts have paid close attention to whether their performance metrics need risk adjustment and closely adhered to process-of-care measures to obviate the need for this technique; pediatric efforts have not.

Although the issue of risk adjustment is widely acknowledged, we know of no empirical analyses of pay-for-performance that specifically implement risk-adjustment systems in general pediatric populations. Risk adjustment may be especially important within populations of children with special health needs. For example, analysis of emergency department use (a plausible performance measure) among children receiving title V services found that variation in qualifying diagnosis was the dominant source of variation in emergency department use [40].

Immunization status is a prime example of how patient heterogeneity poses challenges in pediatric pay-for-performance approaches. Immunization status has been traditionally labeled a process measure, but pediatricians cannot ensure ‘up-to-date’ status if families do not present their children for vaccination. Further research is needed to determine if and when risk-adjustment techniques are appropriate for pediatric metrics.

In summary, this article shows that only five studies [23–25,26^••,27^••] specifically evaluated pediatric pay-for-performance programs. The article shows that these studies [23–25,26^••,27^••] demonstrate that the incentives mainly have an equivocal effect, even though the medical literature as a whole may be modestly significant. The article also highlights three important gaps in the pediatric performance incentive literature. First, we do not understand whether this approach can improve healthcare quality for children with significant health conditions. Second, there is no data to support or refute a concern for negative unintended consequences, even though evaluations of performance incentives in adult medical and nonmedical settings suggest that this is a distinct risk. Third, there is no discussion of the role of risk-adjustment in mitigating or contributing to intended and/or unintended effects.

Conclusion

As pediatric performance incentive programs proliferate, it is likely that pediatricians will be approached to participate in and inform these efforts. Pediatricians may want to share the results and limitations of the existing empirical literature and ask proponents hard questions about tracking or guarding against negative unintended consequences. The political climate around this strategy may be optimistic, but the empirical evidence indicates that some skepticism is appropriate.