Abstract
Objective
To analyze the effect of setting higher targets, in a primary care pay-for-performance scheme, on rates of influenza immunization and exception reporting.
Study Setting
The U.K. Quality and Outcomes Framework links financial rewards for family practices to four separate influenza immunization rates for patients with coronary heart disease (CHD), chronic obstructive pulmonary disease, diabetes, and stroke. There is no additional payment for immunization rates above an upper threshold. Patients for whom immunization would be inappropriate can be excepted from the practice for the calculation of the practice immunization rate.
Data
Practice-level information on immunizations and exceptions extracted from electronic records of all practices in England 2004/05 to 2009/10 (n = 8,212–8,403).
Study Design
Longitudinal random effect multilevel linear regressions comparing changes in practice immunization and exception rates for the four chronic conditions before and after the increase in the upper threshold immunization rate for CHD patients in 2006/07.
Principal Findings
The 5 percent increase in the upper payment threshold for CHD was associated with increases in the proportion of immunized CHD patients (0.41 percent, CI: 0.25–0.56 percent), and exception was reported (0.26 percent, CI: 0.12–0.40 percent).
Conclusions
Making quality targets more demanding can not only lead to improvement in quality of care but can also have other consequences.
Keywords: Quality and Outcomes Framework (QOF), influenza immunization, pay-for-performance, upper threshold
Background
The Quality and Outcomes Framework (QOF) was introduced in 2004 to reward U.K. family practices for providing high-quality care as measured by a range of clinical, organizational, and patient experience indicators (Roland 2006). Sixty-five clinical quality indicators measured the percentage of relevant patients with 11 chronic conditions for whom certain interventions had been achieved. For example, one indicator measured the proportion of patients with diabetes who received influenza immunization in the previous influenza season (September to March). Payments to practices for clinical indicators were proportional to achievement between a lower threshold (initially set at 25 percent for all indicators) and an upper threshold (which varied across indicators from 50 to 90 percent). Practices received no additional reward for achievement above the upper threshold for an indicator.
Previous studies have examined the effect of introducing pay-for-performance schemes in health care. They suggest, in general (Petersen et al. 2011; Rosenthal and Frank 2011), and for the QOF, in particular (Mangin and Toop 2006; Toop and Mangin 2006; Serumaga et al. 2004), that incentive schemes have at best modest effects on clinical performance and a small detrimental effect on nonincentivized aspects of care (Doran et al. 2003). Performance improvements from the QOF scheme have been limited, mainly due to generally high levels of achievement at the time the scheme was implemented (Doran et al. 2009), which in turn can be attributed to improving trends in the pre-QOF era (Campbell et al. 2007, 2009). In general, variation in performance of activities incentivized under the QOF scheme has reduced with poorer performing practices improving toward the level of better performing practices (Doran et al. 2006). Differences in influenza immunization rates between practices narrowed after the introduction of the QOF, although inequalities in uptake attributed to socioeconomic status remained (Norbury, Fawkes, and Guthrie 2007).
The focus of this article is not the overall effect of pay-for-performance schemes, but the impact of program design, specifically the level at which targets are set. The criterion for upper thresholds in the QOF was that they should reflect the “maximum practically achievable level to deliver clinical effectiveness” (Department of Health 2008). While the choice of indicators to be included in the QOF was guided by clinical evidence on the effects of the interventions (Department of Health 2004), there was no evidence base for the setting of thresholds, and they were initially set arbitrarily. Thresholds were adjusted for a minority of indicators in the third year of the scheme, but again these adjustments were arbitrary rather than evidence based.
A key question in setting the parameters of pay-for-performance schemes, including upper thresholds, is their effect on clinical achievement. In this article, we provide evidence on the impact of increasing upper thresholds by comparing the changes in achievement for the same intervention (influenza immunization) for four chronic conditions covered by the QOF (chronic obstructive pulmonary disease [COPD], coronary heart disease [CHD], diabetes, and stroke) before and after 2006/07, when the upper threshold for immunization of CHD patients was increased from 85 to 90 percent, whereas the upper thresholds for the other conditions remained at 85 percent (Table 1).
Table 1.
Characteristics of the Influenza Immunization Indicators in the QOF*
| Financial Years | ||||
|---|---|---|---|---|
| Condition | Indicator Definition | Attribute | 2004/05–2005/06 | 2006/07–2009/10 |
| Coronary heart disease | The percentage of patients with coronary heart disease who have a record of influenza immunization during 1 September to 31 March | Lower threshold | 25% | 40% |
| Upper threshold | 85% | 90% | ||
| Maximum points | 7 | 7 | ||
| Chronic obstructive pulmonary disease | The percentage of patients with COPD who have had influenza immunization during 1 September to 31 March | Lower threshold | 25% | 40% |
| Upper threshold | 85% | 85% | ||
| Maximum points | 6 | 6 | ||
| Diabetes mellitus | The percentage of patients with diabetes who have had influenza immunization during 1 September to 31 March | Lower threshold | 25% | 40% |
| Upper threshold | 85% | 85% | ||
| Maximum points | 3 | 3 | ||
| Stroke | The percentage of patients with transient ischemic attack or stroke who have had influenza immunization during 1 September to 31 March | Lower threshold | 25% | 40% |
| Upper threshold | 85% | 85% | ||
| Maximum points | 2 | 2 | ||
For each indicator, points are determined by reported achievement (RA = number immunized/number with condition who have not been exception reported for the indicator). For RA below the lower threshold, no points are earned. For RA between the upper threshold (UT) and lower threshold (LT), the number of points earned is calculated as maximum points × [(RA − LT)/(UT − LT)], and for RA above the upper threshold, the maximum points are earned. In 2004/05, each point was worth £75 ($120) for an average practice, and for 2005/06 to 2009/10, each point was worth £125 ($200).
Increasing upper payment thresholds raises the bar for maximum remuneration and seems to be an obvious way to improve performance. We use the experience of the world's most elaborate and comprehensive primary care pay-for-performance scheme to examine whether raising thresholds does improve performance and whether it has any other consequences.
Methods
Data
Data for the influenza immunization indicators for patients with CHD, COPD, diabetes, and stroke were taken from the Quality Management and Analysis System (QMAS), the national information system supporting the QOF, for the financial years 2004/05 to 2009/10. QMAS automatically extracts data from practice clinical record systems for over 99 percent of the English language practices. For each practice g and indicator i, QMAS reports (1) the number of eligible patients not exception reported by the practice Dgi; (2) the number of patients for which the indicator was met Ngi; and (3) the number of exception reported patients Egi (available only from 2005/06 onward).
For each practice and indicator, we calculated three outcomes:
Reported achievement—the proportion of eligible and nonexception reported patients who received an influenza immunization: RAgi = Ngi/Dgi;
Population achievement—the proportion of all eligible patients, including those exception reported by the practice, for whom the target achieved: PAgi = Ngi/(Dgi + Egi);
Exception reporting rate—the proportion of eligible patients exception reported by the practice: ERgi = Egi/(Dgi + Egi).
Practices are permitted to exception report patients for a variety of reasons, including informed dissent, recent registration with the practice, or a contraindication for the specified intervention. Payments to practices for each indicator are based on reported achievement RAgi = Ngi/Dgi and so depend in part on exception reporting. There is evidence that some practices game their exception reporting to increase their reported achievement (Doran et al. 2009; Gravelle, Sutton, and Ma 2010). We therefore use population achievement PAgi = Ngi/(Dgi + Egi) as a measure of the health gain from immunization for the practice population.
Statistical Analyses
Our aim was to estimate the effect of the increase in the upper payment threshold for the CHD indicator in 2006/07 on reported achievement, population achievement, and exception reporting for CHD patients. The lower threshold was also increased from 25 to 40 percent for all four indicators in 2006/07, but because the increase was the same across all indicators, it is impossible to disentangle its effect from other unobserved factors that may have changed at the same time. Very few practices had reported achievement under 40 percent in 2005/06 (on average 10 across each indicator).
We investigated the effect of the increase in the CHD upper threshold using a type of controlled before-and-after design with nonequivalent controls (Shadish, Cook, and Campbell 2011). We estimated a multiple regression multilevel model to compare, for each practice, the change in each CHD outcome variable (reported achievement, population achievement, exception reporting) before (2005/06) and after (2006/07–2009/10) the increase in the CHD upper threshold with the change in the same outcome variable for COPD, diabetes, and stroke patients between 2005/06 and 2006/07–2009/10.
This design has a number of advantages. First, using a multilevel model enables us to control for unobserved practice-level factors that lead the outcome variables to be correlated over the years and to differ across practices. Second, using indicators for different disease groups, we control for unobserved time invariant factors that vary across disease groups and may affect the outcome variable. For example, patients with some conditions may be more willing to be vaccinated than those with other conditions. Third, by comparing the changes over time across disease groups, we control for unobserved time varying factors that affect the outcome for all disease groups in similar ways. For example, the patient benefitting from immunization may vary across years in relation to the match between the influenza vaccine and circulating virus strains. A detailed description of the statistical methods is provided in the online Supporting Information Appendix SA2.
For each of the three outcomes, we estimated three models to measure the association of the increase in the CHD upper threshold with a change in the CHD outcome. In Model 1, it is assumed that the upper threshold increase during 2006/07 has the same effect in all 4 years, 2006/07 to 2009/10, following the change. In Model 2, we allow the effect to differ for each of the 4 years following the change to see whether the effect was immediate or temporary.
In Model 3, we again assume that the outcome change is the same in all four postchange years, but we examine whether the effect was focused only on those that were financially affected by the change. We do so by allowing for the possibility that the upper threshold increase had different effects on practices with different reported achievement in 2005/06. For example, did practices change their CHD outcomes more from 2006/07 onward, if in 2005/06, their reported achievement was further below the new CHD upper threshold? We tested for this by categorizing practices into three groups, according to their reported achievement in 2005/06: (1) above the new upper threshold (RA ≥ 90 percent), (2) above the old upper threshold but below the new upper threshold (85 percent ≤ RA < 90 percent), and (3) below the old upper threshold (RA < 85 percent).
As we do not have data on exceptions for 2004/05, all the regression models used 5 years of data, from 2005/06 to 2009/10.
Results
Under the QOF, the maximum total available remuneration for the four influenza indicators for an average practice was £2,250 ($3,670). In 2009/10, the average practice reported immunizing 183 CHD, 84 COPD, 218 diabetes, and 85 stroke patients and received totals of £856 ($1,395), £744 ($1,213), £367 ($598), and £244 ($398) for each condition, respectively. The implied average QOF payment per patient vaccinated is £3.88 ($6.32). This is an under-estimate of the payment per patient vaccinated for two reasons. First, the counts of vaccinated patients for each disease include patients with more than one of the four diseases, and the QMAS dataset does not enable us to correct for multimorbidity. Second, as described in the online Appendix SA2, practices receive additional payments under the Directed Enhanced Service (DES) for certain patient groups. For example, for a 65-year-old patient with CHD and diabetes, a practice would be remunerated five times (twice under the QOF for each condition, twice under the DES for each condition, and once under the DES since age ≥ 65) for providing a single influenza immunization.
Figure 1 shows the distributions of reported achievement in 2005/06 and 2006/07 for each of the four indicators. In all cases, the distributions are left skewed with most practices achieving above the upper thresholds. Very few practices had reported achievement under 40 percent (between 6 and 10 per year summing over all four indicators). For COPD, diabetes, and stroke, the distributions show little change over time, whereas the distribution for CHD patients shifts to the right in 2006/07. In Figure 2, in contrast, practice population achievement scores are more normally distributed and show little change over time for any of the conditions. The exception rate distributions for 2005/06 and 2006/07 in Figure 3 are right skewed with most practices exception reporting 0–30 percent of the patients. There appears to be a small increase in exception reporting between 2005/06 and 2006/07, which is more pronounced for CHD.
Figure 1.
Distribution of Practice Reported Achievement Rates before (2005/06) and after (2006/07) the Upper Threshold Increase in the Influenza Immunization Indicator for CHD
Figure 2.
Distribution of Practice Population Achievement Rates before (2005/06) and after (2006/07) the Upper Threshold Increase in the Influenza Immunization Indicator for CHD
Figure 3.
Distribution of Practice Exception Reporting Rates before (2005/06) and after (2006/07) the Upper Threshold Increase in the Influenza Immunization Indicator for CHD
Table 2 shows that mean reported achievement increased between 2005/06 and 2006/07 for all the influenza immunization indicators but changed little between 2007/08 and 2009/10, except for COPD where reported achievement increased in 2009/10. For the three indicators with constant upper thresholds (COPD, diabetes, and stroke), from 2005/06 onward, over 80 percent of all practices were receiving maximum remuneration (i.e., reported achievement exceeded 85 percent), except for stroke in 2005/06. By 2009/10, the percentage of practices with maximum remuneration for these indicators ranged from 85 percent for stroke to 95 percent for COPD.
Table 2.
Practice Performance on the QOF Influenza Immunization Indicators
| Financial Year | |||||||
|---|---|---|---|---|---|---|---|
| Indicator for | Measure | 2004/05 | 2005/06 | 2006/07 | 2007/08 | 2008/09 | 2009/10 |
| CHD | Mean reported achievement % (SD) | 86.7 (10.0) | 90.5 (7.0) | 92.5 (6.3) | 92.4 (5.7) | 92.4 (5.6) | 92.3 (6.0) |
| Mean population achievement % (SD) | NA | 80.7 (7.4) | 81.0 (7.3) | 80.6 (7.2) | 81.1 (6.9) | 80.7 (8.1) | |
| Mean exception reporting % (SD) | NA | 10.7 (5.9) | 12.3 (6.2) | 12.8 (6.5) | 12.1 (6.2) | 12.5 (7.6) | |
| % of practices with reported achievement ≥ 90% | 43.4 | 62.3 | 81.1 | 82.2 | 82.2 | 82.2 | |
| ≥85% | 67.8 | 82.9 | 90.6 | 92.0 | 92.8 | 92.2 | |
| COPD | Mean reported achievement % (SD) | 88.3 (11.1) | 91.7 (7.4) | 92.7 (6.4) | 92.4 (6.2) | 92.3 (6.0) | 93.4 (5.9) |
| Mean population achievement % (SD) | NA | 81.8 (8.8) | 81.5 (8.5) | 80.5 (8.6) | 80.9 (8.1) | 81.1 (8.8) | |
| Mean exception reporting % (SD) | NA | 11.0 (7.1) | 12.1 (7.3) | 12.9 (7.6) | 12.3 (7.1) | 13.1 (8.0) | |
| % of practices with reported achievement ≥ 90% | 54.1 | 70.1 | 74.6 | 72.7 | 72.6 | 80.1 | |
| ≥85% | 72.4 | 88.5 | 92.1 | 93.1 | 93.4 | 95.1 | |
| Diabetes mellitus | Mean reported achievement % (SD) | 85.2 (11.2) | 89.7 (7.5) | 90.9 (6.9) | 90.8 (6.2) | 90.6 (6.0) | 90.9 (6.0) |
| Mean population achievement % (SD) | NA | 77.8 (7.8) | 77.6 (7.8) | 77.0 (7.5) | 77.7 (6.9) | 77.7 (8.0) | |
| Mean exception reporting % (SD) | NA | 13.2 (6.5) | 14.5 (6.7) | 15.1 (6.8) | 14.2 (6.2) | 14.5 (7.5) | |
| % of practices with reported achievement ≥ 90% | 39.3 | 57.6 | 65.1 | 62.3 | 59.2 | 62.1 | |
| ≥85% | 62.2 | 82.0 | 87.6 | 88.6 | 88.9 | 89.7 | |
| Stroke | Mean reported achievement % (SD) | 83.8 (12.9) | 88.3 (9.4) | 89.8 (8.3) | 90.0 (7.8) | 89.8 (7.4) | 89.7 (7.7) |
| Mean population achievement % (SD) | NA | 76.4 (9.7) | 76.5 (9.3) | 76.4 (9.2) | 76.8 (8.8) | 76.2 (9.8) | |
| Mean exception reporting % (SD) | NA | 13.4 (8.0) | 14.7 (8.2) | 15.0 (8.2) | 14.4 (8.1) | 14.9 (9.1) | |
| % of practices with reported achievement ≥ 90% | 35.4 | 49.5 | 57.1 | 55.8 | 54.0 | 53.0 | |
| ≥85% | 56.8 | 76.9 | 83.3 | 85.3 | 85.1 | 85.2 | |
Note. Reported achievement = N/D, population achievement = N/(D + E), exception reporting = E/(D + E), where N is number immunized, D is number with condition and not exception reported, and E is number exception reported.
For the CHD indicator, there was an 18.8 percent increase in the proportion of practices with reported achievement over 90 percent in 2006/07, when the upper threshold was raised from 85 to 90 percent. There was no similar increase in 2006/07 for the other three indicators. For all indicators, population achievement was relatively stable over time. Exception reporting increased between 2005/06 and 2006/07, with the largest increase for the CHD indicator.
Table 3 reports the results from the regression models. The coefficients on the year indicators in the top four rows of the results for Model 1 show the average changes across all four indicators in each year relative to 2005/06. Averaged across all four indicators, reported achievement and exception reporting was higher from 2006/07 onward than in 2005/06, whereas population achievement was lower. The coefficients on the CHD, diabetes, and stroke indicators show the average difference over the 5 years between each of these indicators and the COPD indicator. Across the whole time period, the COPD indicator had the highest levels of reported achievement and population achievement, with the greatest contrast observed against the stroke indicator. Over the whole period, CHD had the lowest exception rates and stroke had the highest.
Table 3.
Regression Models for Reported Achievement (RA), Population Achievement (PA), and Exception Reporting (ER) for Influenza Immunization Indicators
| Reported Achievement | Population Achievement | Exception Reporting | ||||
|---|---|---|---|---|---|---|
| Dependent Variable | Coefficient % (95% CI) | p-Value | Coefficient % (95% CI) | p-Value | Coefficient % (95% CI) | p-Value |
| Model 1 | ||||||
| 2006/07* | 1.17 (1.09, 1.26) | <.001 | −0.11 (−0.21, −0.02) | .020 | 1.27 (1.19, 1.36) | <.001 |
| 2007/08* | 1.06 (0.97, 1.16) | <.001 | −0.70 (−0.80, −0.60) | <.001 | 1.84 (1.74, 1.93) | <.001 |
| 2008/09* | 0.94 (0.83, 1.05) | <.001 | −0.20 (−0.31, −0.09) | <.001 | 1.16 (1.05, 1.26) | <.001 |
| 2009/10* | 1.19 (1.06, 1.31) | <.001 | −0.16 (−0.28, −0.04) | .010 | 1.36 (1.24, 1.48) | <.001 |
| CHD indicator | −0.46 (−0.59, −0.32) | <.001 | −0.31 (−0.47, −0.15) | <.001 | −0.49 (−0.63, −0.34) | <.001 |
| DM indicator | −1.25 (−1.34, −1.16) | <.001 | −3. 17 (−3.28, −3.07) | <.001 | 1.89 (1.80, 1.98) | <.001 |
| Stroke indicator | −3.00 (−3.06, −2.93) | <.001 | −4.67 (−4.74, −4.59) | <.001 | 2.18 (2.11, 2.25) | <.001 |
| Practice disease register (100s)† | −0.53 (−0.57, −0.49) | <.001 | −0.24 (−0.28, −0.20) | <.001 | 0.06 (0.02, 0.10) | .003 |
| Effect of change in CHD upper threshold on CHD outcome | 0.69 (0.55, 0.82) | <.001 | 0.41 (0.25, 0.56) | <.001 | 0.26 (0.12, 0.40) | <.001 |
| Model 2‡ | ||||||
| Effect of CHD upper threshold change on CHD outcome: | 0.76 (0.59, 0.94) | <.001 | 0.41 (0.22, 0.61) | <.001 | 0.29 (0.11, 0.47) | .001 |
| In 2007/08 versus 2006/07 | 0.04 (−0.13, 0.21) | .652 | 0.15 (−0.05, 0.34) | .139 | −0.14 (−0.31, 0.04) | .135 |
| In 2008/09 versus 2006/07 | 0.11 (−0.07, 0.28) | .231 | 0.19 (−0.002, 0.39) | .052 | −0.11 (−0.29, 0.07) | .216 |
| In 2009/10 versus 2006/07 | −0.45 (−0.62, −0.28) | <.001 | −0.11 (−0.31, 0.08) | .250 | −0.31 (−0.49, −0.13) | .001 |
| Model 3§ | ||||||
| Effect of CHD upper threshold change on CHD outcome for practices with 2005/06 CHD RA≥90% | 0.49 (0.35, 0.63) | <.001 | 0.37 (0.21, 0.53) | <.001 | 0.13 (−0.02, 0.28) | .080 |
| 2005/06 CHD RA in (85, 90)% range | 0.98 (0.80, 1.16) | <.001 | 0.60 (0.40, 0.80) | <.001 | 0.29 (0.10, 0.47) | .003 |
| 2005/06 CHD RA below 85% | 1.47 (1.27, 1.68) | <.001 | 0.85 (0.62, 1.08) | <.001 | 0.50 (0.29, 0.72) | <.001 |
Compared with 2005/06.
D for the RA model, D + E for the PA and ER models, where D is number with condition and not exception reported and E is number exception reported.
Model 2 also contains indicator dummies, year dummies, and disease register. See Appendix SA2 for details.
Model 3 also contains indicator dummies, year dummies, disease register, and dummies for RA ≥ 90% and RA ∈ (85, 90). See Appendix SA2 for details.
CHD Reported Achievement and the Raising of the Upper Threshold
The coefficient in the last row of Model 1 shows that CHD reported achievement increased significantly, relative to the other indicators, when the CHD upper threshold increased. Model 1 assumes that the increase is the same in all postincrease years, 2006/07 to 2009/10. Model 2 is similar to Model 1 but allows the association between the CHD upper threshold increase and CHD outcomes to vary across the years following the increase in the upper threshold. We report only this association in the row for Model 2. The increase in reported achievement associated with the change in the upper threshold in the first year of the change (2006/07) was 0.76 percent, and the increase, relative to 2005/06, was similar in 2007/08 or 2008/09. However, the increase in 2009/10 relative to 2005/06 was significantly smaller compared with the increase in 2006/07 relative to 2005/06.
Model 3 allows for the possibility that the association between the CHD outcome and the increase in the CHD upper threshold depends on reported achievement in 2005/06 (the year preceding the threshold increase). The results reported in the last three rows in Table 3 show that, compared with their reported achievement for the other three indicators, practices had higher CHD reported achievement rates after the raising of the CHD upper threshold increase, and that effect was observed across all three levels of 2005/06 reported achievement. The increase was largest (1.47 percent, CI: 1.27–1.68 percent) for practices achieving below the old CHD upper threshold (85 percent) in 2005/06.
CHD Population Achievement and the Raising of the Upper Threshold
The Model 1 association between the CHD upper threshold increase and CHD population achievement (0.41 percent, CI: 0.25–0.56 percent) was smaller than the association with CHD reported achievement (0.69 percent, CI: 0.55–0.82 percent). Allowing the association to differ across years, we observe in Model 2, a similar increase in population achievement in 2006/07 (0.41 percent, CI: 0.22–0.61 percent) and no significant change in this increase in the next 3 years. The results for Model 3 show that the increase in CHD population achievement after the raising of the upper threshold was greater than for the other indicators, across all levels of reported achievement in 2005/06. Practices with reported achievement below 85 percent in 2005/06 showed the largest increase in population achievement of 0.85 percent (CI: 0.62–1.08 percent).
CHD Exception Reporting and the Raising of the Upper Threshold
Relative to the changes in the exception rates for other indicators, there was a small but statistically significant 0.26 percent (CI: 0.12–0.40 percent) increase in CHD exception reporting after the raising of the CHD upper threshold (last row of Model 1). Allowing the change in CHD exception reporting to vary across years (Model 2) produces a 0.29 percent (CI: 0.11–0.47 percent) increase in exception reporting for 2006/07. Although there was no significant change in 2007/08 or 2008/09 relative to 2006/07, the effect of the upper threshold increase on exception reporting was significantly smaller in 2009/10. The results for Model 3 show the increase in CHD exception reporting relative to the change for the other indicators after the raising of the upper threshold was larger for practices whose reported achievement was lower in 2005/06. Practices with reported achievement below 85 percent in 2005/06 increased exception reporting by 0.50 percent (CI: 0.29–0.72 percent) after the raising of the upper threshold. For practices achieving at the 90 percent level or above in 2005/06, the relative increase in exception reporting was not statistically significant.
Discussion
Limitations
Certain characteristics of the influenza indicators limit the generalizability of our results to other QOF indicators. First, most QOF indicators relate to nonseasonal secondary preventative activities, and practice performance is assessed over a 15-month period, whereas influenza immunization is a primary preventative activity conducted over the shorter influenza season. Second, immunization rates are likely to vary somewhat by year, because the timely availability of vaccine varies by year, and fluctuations in influenza activity are likely to be associated with variation in patients’ attitudes toward immunization. However, the statistical analysis allows for this greater potential unexplained variation in calculating significance levels by comparing across four influenza immunizations that are likely to be subject to the same kind of temporal shocks arising from changes in the efficacy of the vaccine. Third, although patterns of achievement on the influenza indicators are typical of QOF indicators in general, rates of exception reporting are relatively high at over 10 percent, compared with an average of 5–6 percent for all QOF indicators (Doran et al. 2009).
Characteristics of QOF indicators also limit generalizability to other types of pay-for-performance schemes. For example, the financial incentive to exceed the upper threshold is weaker than incentive schemes based on all-or-nothing thresholds. In 2006/07, an average practice attaining the upper threshold of 90 percent reported achievement for the CHD indicator would have earned £875 ($1,400), but a practice with 85 percent reported achievement would have earned only £87.50 ($140) less.
Discussion
Practice performance on influenza immunization indicators of the QOF varied slightly across conditions and years. Across all conditions, compared with 2005/06, reported achievement was higher, whereas population was lower or at the same levels in 2009/10. This discrepancy can be attributed to an overall increase in exception reporting rates over time, which drove up the measure of achievement on which practices are rewarded (reported achievement). The variability in achievement and exception reporting rates is to be expected as CHD, COPD, diabetes, and stroke patients are likely to differ in terms of disease severity, age, and co-morbidities. However, it is not clear why exception reporting increased over time.
The negative effect of practice disease register size on both population and reported achievement counters the general trend for larger practices to slightly outperform smaller ones under the QOF scheme (Doran et al. 2006). One possible explanation is that the work burden is more difficult to manage when the number of patients in need of influenza immunization is large, perhaps due to the narrow time window in which the immunizations are provided.
The raising of the CHD upper payment threshold by 5 percent in 2006/07 led to small increases in both reported and population achievement for CHD. As levels of population achievement and, especially, reported achievement were already very high by 2006/07, there was not much room for improvement, and the small magnitude of the effects was unsurprising. In addition, it appears that practices improved their reported achievement rates by increasing exception reporting as well as immunizing more patients. This finding is not necessarily an indication of practice “gaming” because practices may be improving their record keeping and reporting of valid exceptions: the raised bar may be driving practices to become more thorough in documenting exceptions.
The effect of the upper threshold increase on achievement and exception reporting was immediate, with the greatest increases occurring in 2006/07—the first year of the change. There was no evidence that the effect was focused on those practices that were achieving performance above the old upper threshold, but below the new upper threshold. The largest increases in reported achievement, population achievement, and exception reporting were observed for practices that had the lowest initial levels of reported achievement.
We estimated that population achievement for CHD immunization increased by 0.41 percent (CI: 0.25–0.56 percent) after the increase in the CHD upper threshold, relative to the changes in immunization for COPD, diabetes, and stroke patients, where the upper threshold did not increase. This is a relatively small change, but across England, it equates to an extra 7,730 immunizations per year for CHD patients. Using results from the FLUCAD study (Ciszewski et al. 2007), which enlisted patients with stable CHD, these additional immunizations would be expected to prevent 371 influenza cases per year. The reduction in influenza cases is expected to have reduced cardiac events, but the magnitude of this effect is uncertain. Although reviews on the effectiveness of influenza immunization in preventing cardiovascular events have been inconclusive (Keller et al. 2010; Warren-Gash, Smeeth, and Hayward 2007), recent studies suggest a protective effect (Natarajan and Cannon 2008; Phrommintikul et al. 2011).
Total QOF payments for immunization of CHD patients decreased from an estimated £7,224,000 ($11,775,120) in 2005/06 to £7,122,000 ($11,608,860) in 2006/07, whereas payments for patients with the other three conditions did not change. However, these figures do not capture the upper threshold change effect in isolation, nor were our calculations adjusted for practice prevalence (Guthrie, McLean, and Sutton 2011). In addition, a full costing of the effects of the upper threshold increase would require information on the opportunity costs of the changes in resource use inside practices.
Conclusion
Rates of influenza immunization for specified vulnerable groups in England have remained generally high under a pay-for-performance program that includes financial incentives for immunizing CHD patients. A small (5 percent) rise in the upper payment threshold for patients with CHD was associated with a small but immediate increase in immunization rates for this group of patients, including those in practices that were already achieving above the new threshold. There was no evidence that low-performing practices reduced their immunization efforts, because they could not reach the new upper threshold. However, rates of exception reporting of patients also increased.
The risks associated with raising thresholds are that patients will be coerced or treated inappropriately, or that practices will be financially compromised by caring for dissenting or difficult-to-manage patients. The provision to exception report under the QOF provides protection against these risks. The upper thresholds for the QOF influenza indicators have currently exceeded by over 80 percent of practices, and these thresholds therefore do not serve their intended purpose of representing the maximum level that is practically achievable. Raising the upper thresholds could stimulate increases in immunization rates and thereby reduce morbidity and mortality in vulnerable groups.
Acknowledgments
Joint Acknowledgment/Disclosure Statement: The research on which this paper is based was funded in part by the Department of Health via its core funding to the National Primary Care Research and Development Centre, which ceased to exist in December 2010. The views expressed are those of the authors and not necessarily those of the DoH. We are grateful to the National Health Service Information Centre for providing access to the QMAS data. We thank the editor and two referees for helpful comments and suggestions that materially improved this manuscript.
Disclosures: None.
Disclaimers: None.
SUPPORTING INFORMATION
Additional supporting information may be found in the online version of this article:
Appendix SA1: Author Matrix.
Appendix SA2: U.K. Incentives for Influenza Immunization Indicators and Statistical Models Details.
Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.
References
- Campbell S, Reeves D, Kontopantelis E, Middleton E, Sibbald B, Roland M. Quality of Primary Care in England with the Introduction of Pay for Performance. New England Journal of Medicine. 357(2):181–90. doi: 10.1056/NEJMsr065990. [DOI] [PubMed] [Google Scholar]
- Campbell SM, Reeves D, Kontopantelis E, Sibbald B, Roland M. Effects of Pay for Performance on the Quality of Primary Care in England. New England Journal of Medicine. 361(4):368–78. doi: 10.1056/NEJMsa0807651. [DOI] [PubMed] [Google Scholar]
- Ciszewski A, Bilinska ZT, Brydak LB, Kepka C, Kruk M, Romanowska M, Ksiezycka E, Przyluski J, Piotrowski W, Maczynska R, Ruzyllo W. Influenza Vaccination in Secondary Prevention from Coronary Ischaemic Events in Coronary Artery Disease: FLUCAD Study. European Heart Journal. 2007;29(11):1350–8. doi: 10.1093/eurheartj/ehm581. [DOI] [PubMed] [Google Scholar]
- Department of Health. Investing in General Practice: The New General Medical Services Contract. London: NHS Confederation and the British Medical Association; 2008. T. D. o. Health. [Google Scholar]
- Department of Health. GMS Statement of Financial Entitlements for 2004/5. London: Department of Health; 2004. p. 273. [Google Scholar]
- Doran T, Fullwood C, Gravelle H, Reeves D, Kontopantelis E, Hiroeh U, Roland M. Pay-for-Performance Programs in Family Practices in the United Kingdom. New England Journal of Medicine. 2009;355(4):375–84. doi: 10.1056/NEJMsa055505. [DOI] [PubMed] [Google Scholar]
- Doran T, Campbell S, Fullwood C, Kontopantelis E, Roland M. Performance of Small General Practices under the UK's Quality and Outcomes Framework. British Journal of General Practice. 2006;60(578):335–44. doi: 10.3399/bjgp10X515340. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Doran T, Kontopantelis E, Valderas JM, Campbell S, Roland M, Salisbury C, Reeves D. Effect of Financial Incentives on Incentivised and non-Incentivised Clinical Activities: Longitudinal Analysis of Data from the UK Quality and Outcomes Framework. British Medical Journal. 2003;342:d3590. doi: 10.1136/bmj.d3590. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Gravelle H, Sutton M, Ma A. Doctor Behaviour under a Pay for Performance Contract: Treating, Cheating and Case Finding? Economic Journal. 2010;120(542):F129–56. [Google Scholar]
- Guthrie B, McLean G, Sutton M. Workload and Reward in the Quality and Outcomes Framework of the 2004 General Practice Contract. British Journal of General Practice. 2011;56(532):836–41. [PMC free article] [PubMed] [Google Scholar]
- Keller T, Weeda VB, van Dongen CJ, Levi M. Influenza Vaccines for Preventing Coronary Heart Disease. Cochrane Database of Systematic Reviews. 2010;0(3):CD005050. doi: 10.1002/14651858.CD005050.pub2. [DOI] [PubMed] [Google Scholar]
- Mangin D, Toop L. The Quality and Outcomes Framework: What Have You Done to Yourselves? British Journal of General Practice. 2006;57(539):435–7. [PMC free article] [PubMed] [Google Scholar]
- Natarajan P, Cannon CP. Myocardial Infarction Vaccine? Evidence Supporting the Influenza Vaccine for Secondary Prevention. European Heart Journal. 2008;32(14):1701–3. doi: 10.1093/eurheartj/ehr053. [DOI] [PubMed] [Google Scholar]
- Norbury M, Fawkes N, Guthrie B. Impact of the GP Contract on Inequalities Associated with Influenza Immunisation: A Retrospective Population-Database Analysis. British Journal of General Practice. 2007;61(588):e379–85. doi: 10.3399/bjgp11X583146. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Petersen LA, Woodard LD, Urech T, Daw C, Sookanan S. Does Pay-for-Performance Improve the Quality of Health Care? Annals of Internal Medicine. 2011;145(4):265–72. doi: 10.7326/0003-4819-145-4-200608150-00006. [DOI] [PubMed] [Google Scholar]
- Phrommintikul A, Kuanprasert S, Wongcharoen W, Kanjanavanit R, Chaiwarith R, Sukonthasarn A. Influenza Vaccination Reduces Cardiovascular Events in Patients with Acute Coronary Syndrome. European Heart Journal. 2011;32(14):1730–5. doi: 10.1093/eurheartj/ehr004. [DOI] [PubMed] [Google Scholar]
- Roland M. Linking Physicians’ Pay to the Quality of Care–A Major Experiment in the United Kingdom. New England Journal of Medicine. 2006;351(14):1448–54. doi: 10.1056/NEJMhpr041294. [DOI] [PubMed] [Google Scholar]
- Rosenthal MB, Frank RG. What Is the Empirical Basis for Paying for Quality in Health Care? Medical Care Research and Review. 2011;63(2):135–57. doi: 10.1177/1077558705285291. [DOI] [PubMed] [Google Scholar]
- Serumaga B, Ross-Degnan D, Avery AJ, Elliott RA, Majumdar SR, Zhang F, Soumerai SB. Effect of Pay for Performance on the Management and Outcomes of Hypertension in the United Kingdom: Interrupted Time Series Study. British Medical Journal. 2004;342:d108. doi: 10.1136/bmj.d108. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Shadish WR, Cook TD, Campbell DT. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Belmont, CA: Houghton Mifflin; 2011. [Google Scholar]
- Toop LJ, Mangin D. Quality and Outcomes Framework. Evidence of Net Benefit Is Lacking. British Medical Journal. 2006;335(7631):1170. doi: 10.1136/bmj.39416.414688.BE. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Warren-Gash C, Smeeth L, Hayward AC. Influenza as a Trigger for Acute Myocardial Infarction or Death from Cardiovascular Disease: A Systematic Review. Lancet Infectious Diseases. 2009;9(10):601–10. doi: 10.1016/S1473-3099(09)70233-6. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.