Abstract
Objectives To test the impact of provider performance pay for anaemia reduction in rural China.
Design A cluster randomised trial of information, subsidies, and incentives for school principals to reduce anaemia among their students. Enumerators and study participants were not informed of study arm assignment.
Setting 72 randomly selected rural primary schools across northwest China.
Participants 3553 fourth and fifth grade students aged 9-11 years. All fourth and fifth grade students in sample schools participated in the study.
Interventions Sample schools were randomly assigned to a control group, with no intervention, or one of three treatment arms: (a) an information arm, in which principals received information about anaemia; (b) a subsidy arm, in which principals received information and unconditional subsidies; and (c) an incentive arm, in which principals received information, subsidies, and financial incentives for reducing anaemia among students. Twenty seven schools were assigned to the control arm (1816 students at baseline, 1623 at end point), 15 were assigned to the information arm (659 students at baseline, 596 at end point), 15 to the subsidy arm (726 students at baseline, 667 at end point), and 15 to the incentive arm (743 students at baseline, 667 at end point).
Main outcome measures Student haemoglobin concentrations.
Results Mean student haemoglobin concentration rose by 1.5 g/L (95% CI –1.1 to 4.1) in information schools, 0.8 g/L (–1.8 to 3.3) in subsidy schools, and 2.4 g/L (0 to 4.9) in incentive schools compared with the control group. This increase in haemoglobin corresponded to a reduction in prevalence of anaemia (Hb <115 g/L) of 24% in incentive schools. Interactions with pre-existing incentives for principals to achieve good academic performance led to substantially larger gains in the information and incentive arms: when combined with incentives for good academic performance, associated effects on student haemoglobin concentration were 9.8 g/L (4.1 to 15.5) larger in information schools and 8.6 g/L (2.1 to 15.1) larger in incentive schools.
Conclusions Financial incentives for health improvement were modestly effective. Understanding interactions with other motives and pre-existing incentives is critical.
Trial registration number ISRCTN76158086.
Introduction
Inexpensive, efficacious technologies and services exist for improving human health in developing countries, but implementation and coverage are often low. Clear examples include point-of-use water disinfectants, insecticide treated bed nets, oral rehydration therapy, fortified food staples, condoms, improved cooking stoves, and basic primary healthcare services. Why have efforts to disseminate these technologies and services not produced greater population health gains? Given tremendous efforts by donors and international organisations in recent years, the answer cannot simply be that they are unavailable or unaffordable.
Misalignment between supplier incentives and the ultimate objective of improving health may be an important part of this puzzle. Building on the logic of performance pay in human resource management, a straightforward solution may be rewarding providers directly for producing socially desirable outcomes. Beyond well known applications in wealthy countries (such as the British National Health Service’s quality and outcomes framework), a growing number of aid organisations are also experimenting with pay-for-performance incentives under the umbrella of “results based financing,” rewarding measurable increases in the use of traditionally underused health inputs such as the WHO’s Expanded Programme on Immunisation bundle of childhood vaccinations.1 2 3 4 5 6 7 8
The full promise of pay-for-performance incentives extends far beyond current applications to increase the use of specified technologies and services, however.1 9 In particular, rewarding the ultimate objective—health improvement—without specifying how it should be achieved can strengthen incentives for creativity and innovation in service delivery. Given broad decision making authority, local providers are better able to use their superior knowledge of what is likely to work (and not work) in local settings. To the best of our knowledge, performance pay for health improvement in a developing country has not been tried or evaluated.
This paper presents first evidence on the effectiveness of rewarding providers for better health outcomes. Despite China’s rapid economic development, the prevalence of anaemia among children in rural China ranges from 20% to 60%, implying more than 10 million affected children.10 11 12 In addition to causing debilitating fatigue and retarding growth, childhood anaemia may also impair cognitive development and inhibit human capital accumulation—lowering socioeconomic status throughout the life course.13 14 15 16 The high prevalence of childhood anaemia in China and many other developing countries is remarkable given that it can (in principle) be confronted through simple, low cost nutritional interventions.17 18 We therefore conducted a cluster randomised field experiment in rural Chinese primary schools (a natural contact point with children19) to study the impact of information, subsidies, and financial incentives for school principals to reduce anaemia among their students.
Methods
Setting and participants
Through a canvass survey, we first created a sampling universe of all primary schools in 10 nationally designated poor counties spread across two provinces with high anaemia rates—Ningxia and Qinghai. We then identified all schools having six grades (that is, “complete” primary schools, or wanxiao) and boarding facilities. These criteria were used because China’s government is currently consolidating existing rural schools into new ones with these characteristics. A total of 85 schools met these criteria, and we randomly selected 72 for inclusion in our study. Finally, we randomly selected half of fourth and fifth grade students in study schools (sampling 3944 students in total). Fourth and fifth grade students were chosen because they are old enough for test scores to be relevant but also young enough not to have reached puberty (at which point nutritional requirements differ more markedly from childhood and vary by sex).
Of the 3944 students enrolled in the study, we were able to obtain haemoglobin concentration measurements for 3553 (90.1%) in the follow-up survey. Attrition was almost entirely due to students not being present at the time of the endpoint survey; the refusal rate was less than 1%. This refusal rate is consistent with previous studies conducted by the authors in China.20 The probability of attrition was not statistically different across the study arms.
The study received ethical approval from the Stanford University Institutional Review Board (IRB) on 21 July 2009 (protocol number 17071). All necessary permissions were obtained from the Chinese government as well. All participating children gave their assent to participate in the project, and their legal guardians provided consent. All study participants were aware of the (minimal) risks involved and understood that their participation was purely voluntary.
Study outcomes and data collection
Primary outcomes
Our study includes two primary outcomes. The first is change in altitude adjusted haemoglobin concentration. Using estimates from our previous studies,20 we calculated that we would require 30 schools (15 per trial arm) and 55 students per school for each inter-arm comparison to detect a standardised effect size of 0.4 for haemoglobin concentration with 80% power at the 5% significance level. We assumed an intra-cluster correlation of 0.25, a pre-intervention and post-intervention correlation of 0.5, and 10% loss to follow-up. The actual unadjusted intra-cluster correlation in our sample was lower than expected (0.14 for student level changes in haemoglobin concentration, our primary outcome; 0.23 at baseline survey; and 0.17 at follow-up in the control group). Our second primary outcome variable is anaemia status after the intervention. Anaemia is defined as haemoglobin concentration below 115 g/L at the time of the final survey.
Primary school surveys
We conducted baseline and follow-up surveys of principals and students in all study schools. These surveys collected detailed information about (a) nutritional characteristics of school meals (meat, soybean, fruit, and vegetable composition); (b) use of anaemia related nutritional supplements (such as iron or multivitamin or mineral supplements); and (c) school characteristics (class size, infrastructure, school budgets, and school expenditure records). Because the effectiveness of any incentive scheme depends critically on local context,21 22 we also collected administrative information about explicit financial incentives that local education bureaus offer for good test scores. During our follow-up school surveys, we gathered information from principals about strategies they pursued to reduce anaemia among students as well.
Performance in response to any set of incentives must be verifiable, so, with the assistance of nursing teams from Xi’an Jiaotong Medical School, we also measured student haemoglobin concentration directly using finger prick blood samples (with HemoCue 201+). We randomly retested 10% of all sampled students as well; if the second measure differed from the original one by more than 3.0 g/L among three or more students, we retested all sample students in that school. Short surveys were also administered to students to collect basic information about their ages, socioeconomic characteristics, and eating patterns. Specifically, students were asked about their consumption of a variety of different foods over the past week.23
Household surveys
We conducted baseline and follow-up household surveys for each child in our school based sample. Specifically, we interviewed each child’s household head (usually a parent), collecting information about household socioeconomic characteristics, individual health behaviours, and nutritional characteristics of household meals (consumption of meat, beans, fruits, and vegetables).
Figure 1 depicts the flow of participants through each stage of the study, and table 1 shows student characteristics by trial arm at the time of our baseline survey.
Fig 1 Flow of participants through study
Table 1.
Baseline characteristics of 3553 primary school students in rural China included in trial of information, subsidies, and incentives for reducing anaemia. Values are numbers (percentages (95% CI)) of students in trial arm unless specified otherwise
| Characteristic | Trial arm* | |||
|---|---|---|---|---|
| Control (n=1623 students, 27 schools) | Information (n=596 students, 15 schools) | Subsidy (n=667 students, 15 schools) | Incentive (n=667 students, 15 schools) | |
| Mean (95% CI) haemoglobin concentration (altitude adjusted g/L) | 126 (123 to 128) | 125 (120 to 129) | 124 (119 to 129) | 124 (119 to 129) |
| Anaemic (haemoglobin <115 g/L) | 338 (20.8 (14.4 to 27.3)) |
130 (21.8 (9.9 to 33.7)) |
160 (24.0 (12.4 to 35.5)) |
161 (24.1 (10.3 to 38.0)) |
| Female | 778 (47.9 (46.2 to 49.7) |
281 (47.2 (45.0 to 49.3) |
317 (47.5 (43.2 to 51.9) |
319 (47.8 (43.0 to 52.7) |
| Mean (95% CI) age (months) | 123.3 (120.6 to 125.9) | 124.7 (121.8 to 127.6) | 123.4 (118.4 to 128.3) | 120.5 (116.4 to 124.7) |
| Boarding student | 554 (34.1 (25.5 to 42.8)) |
248 (41.6 (29.3 to 53.9)) |
236 (35.4 (24.1 to 46.7)) |
244 (36.6 (30.5 to 42.7)) |
| Migrant mother | 386 (23.8 (15.3 to 32.3)) |
156 (26.2 (16.7 to 35.6)) |
158 (23.7 (8.3 to 39.0)) |
165 (24.7 (14.5 to 35.0)) |
| Mother has primary school education or less | 1403 (86.4 (80.8 to 92.1)) |
545 (91.4 (86.8 to 96.1)) |
572 (85.8 (77.9 to 93.6)) |
569 (85.3(78.8 to 91.8)) |
*Control arm (no intervention); information arm (school principals received information about anaemia); subsidy arm (principals received information and unconditional subsidies for reducing anaemia); incentive arm (principals received information, subsidies, and financial incentives for reducing anaemia).
Randomisation and interventions
After the baseline survey, our research team randomly assigned study schools to a control arm or one of three experimental arms (described below). To account for our three treatment arms while maximising power given project resources, we allocated 27 schools to the control group (following Dunnet,24 25 who shows that the required number of control units is approximately N√t, where N is the number of treatment arm units and t is the number of treatments), 15 schools to the “information” arm, 15 to the “subsidy” arm, and 15 to the “incentive” arm. The allocation of schools to treatment arms was done by one of the authors in private using STATA 10. The randomisation procedure was repeated until achieving balance in haemoglobin concentration across the four arms (with 95% confidence). Participants and enumerators were not explicitly informed of study arm allocation, but we cannot rule out that any of them became aware of it.
Schools in all four groups were located an average of 30 km apart, and because there was only one study school per township, there was no communication among study schools through local school districts. It is therefore unlikely that principals or other study participants would have had an opportunity to communicate about the study across study clusters.
The four trial arms were:
Control—Schools in the control group did not receive any intervention. Principals were told only that they would be participating in a study on how better nutrition might affect iron deficiency.
Information arm—We provided three types of information to principals: (a) the share of enrolled students who were anaemic, (b) descriptions of effective methods for reducing iron deficiency anaemia (including vitamin supplementation, adding more meat at lunch, and other dietary changes, as well as the potential importance of educating parents about anaemia), and (c) details about anaemia’s relation with school attendance, educational performance, and cognitive development (as reported in peer reviewed academic studies).14 15 16 19 Principals did not receive financial assistance of any kind.
Subsidy arm (information + subsidy)—Principals were given the same information as in the information arm, but, because purchasing inputs to reduce anaemia might be difficult with current school operating budgets (which are small and allow for little discretion), schools in this arm also received earmarked operating budget subsidies. The subsidy schools were given ¥1.5 (about $0.22, £0.14, €0.18) per student per day, an amount sufficient to buy two to three ounces (55-85 g) of red meat. These subsidies were in principle only to be used for nutrition related expenses (we studied reallocation of school funds as well). Principals did not receive financial incentives of any kind.
Incentive arm (information + subsidy + incentives)—To test the effectiveness of direct rewards for health improvement, school principals in this group received performance payments for anaemia reductions among their student bodies (in addition to the information and subsidies described above). Under the governance structure of Chinese primary schools, school principals make executive decisions about school operations.26 Incremental incentive payments were made as salary supplements to principals on a per student basis (¥150 per student who changed from anaemic to non-anaemic over the course of the intervention). This amount was chosen to equal roughly two months of salary (¥3000) for successfully reducing the total number of students with anaemia by 50%—a feasible reduction according to our early pilot experience.
Statistical analysis
We estimated student level changes in haemoglobin concentration in the intervention arms relative to the control arm using multilevel mixed effect linear models with hierarchical error structures allowing for random effects at the county and school levels. Per our trial protocol, baseline adjusted models controlled for student haemoglobin concentrations before our interventions. Our multivariate adjusted models also controlled for student sex, student age, whether the student boards at school, mother’s migration status, mother’s education level, and whether the principal was eligible for a bonus based on student exam scores. For probability of anaemia at study end, we report average marginal effects from baseline adjusted and multivariate adjusted mixed effect logistic regressions with the same error structure.
Additionally, we investigated how the impact of treatment arms varied by pre-existing principal incentives for good test scores. Roughly 20% of schools in our sample had explicit test score incentives through the cadre evaluation system (ganbu kaohe zhidu).27 These were balanced across experiment arms and often gave principals up to one month of additional salary for mean test scores ranking in their district or county’s top decile. To examine differences in intervention effects between schools with and without pre-existing principal incentives for good test scores, we included interactions between treatment arms and a dummy variable indicating local government use of test score incentives in the multivariate adjusted models described above. We calculated the interaction effect for each intervention by taking the difference between the estimated marginal effect of each intervention among principals with test score incentives and the marginal effect estimated for principals without test score incentives. All analyses were performed using STATA 10.
In the linked data supplement, we also analyse behavioural responses to each intervention arm, testing for differences (and differential changes) in (a) principals’ strategies to reduce anaemia, (b) principals’ efforts to educate parents about good nutrition and anaemia, (c) the composition of students’ diets (at home and at school), and (d) types of school expenditures.
Our trial was registered in January 2011 after the completion of project fieldwork in May 2010. In conducting the trial, we originally intended to write for social scientists, among whom trial registration is a relatively new phenomenon.28 Through the project, we more clearly recognised the role of trial registration (including documenting studies whose findings are not ultimately published and preventing deviation from originally planned analyses). Our paper and the results we report are consistent with our original focus. We conducted one set of analyses that was not included in the original protocol: our analyses of heterogeneous effects by test score incentives. However, these analyses provide insight into the contextual factors and cultural context that mediate the impact of performance incentives. Throughout the rest of the paper we distinguish between planned and unplanned analyses.
Results
Primary outcomes: changes in haemoglobin concentrations and anaemia prevalence
Figure 2 shows the distribution of student haemoglobin concentration (g/L) both at baseline and follow-up in each study arm (in separate panels). Each distribution, including the control group, shifted right between the baseline and follow-up surveys. The increase in haemoglobin concentration in the control group over the six month study period presumably reflects a well documented seasonal effect.29 Using non-parametric Kolmogorov-Smirnov tests that account for our cluster randomised design to compare the distribution of changes (baseline to follow-up) across trial arms, we are able to distinguish each intervention arm from the control group (information arm v control, P=0.012; subsidy arm v control, P<0.001; incentive arm v control, P=0.003), but there are no statistically meaningful differences among intervention arms.
Fig 2 Distribution of primary school students’ haemoglobin concentrations at baseline and follow-up analysis by trial arm. (Control arm = no intervention; information arm = school principals received information about anaemia; subsidy arm = principals received information and unconditional subsidies for reducing anaemia; incentive arm = principals received information, subsidies, and financial incentives for reducing anaemia)
Table 2 presents baseline adjusted and multivariable adjusted estimates for both mean change in haemoglobin concentration and for percentage point change in anaemia status. Among the three intervention arms, only the incentive group was associated with a statistically significant increase in haemoglobin concentration compared with the control group (baseline adjusted estimate 2.1 g/L (95% confidence interval –0.4 to 4.7); multivariate adjusted estimate 2.4 g/L (0 to 4.9)). The accompanying reductions in anaemia at study end associated with the incentive intervention were 4 percentage points (95% confidence interval –0.08 to 0) for baseline adjusted analysis and 5 percentage points (–0.09 to –0.01) for multivariate adjusted analysis. These declines imply reductions of 18% (36% to 0%) and 23% (41% to 5%), respectively, relative to the baseline prevalence of 22%. We find no statistically significant changes in haemoglobin status between the intervention arms (rows 3 to 6 in the body of table 2). The school level intra-class correlations for change in haemoglobin concentration in the baseline adjusted and multivariate adjusted models were 0.086 and 0.082, respectively.
Table 2.
Effects of different combinations of information, subsidies, and incentives for reducing anaemia on haemoglobin (Hb) concentration and anaemia prevalence among primary school students in rural China
| Trial arms compared* | Baseline adjusted | Multivariable adjusted | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Change in Hb concentration (g/L)† | Anaemic (Hb <115 g/L)‡ | Change in Hb concentration (g/L)§ | Anaemic (Hb <115 g/L)¶ | ||||||||
| Difference (95% CI) | P value | Marginal effects (95% CI) | P value | Difference (95% CI) | P value | Marginal effects (95% CI) | P value | ||||
| Information v control | 1.5 (–1.2 to 4.1) | 0.275 | –0.02 (–0.06 to 0.03) | 0.430 | 1.5 (–1.1 to 4.1) | 0.245 | –0.02 (–0.06 to 0.02) | 0.364 | |||
| Subsidy v control | 1.0 (–1.6 to 3.6) | 0.471 | –0.03 (–0.07 to 0.01) | 0.188 | 0.8 (–1.8 to 3.3) | 0.548 | –0.03 (–0.07 to 0.02) | 0.220 | |||
| Incentive v control | 2.1 (–0.4 to 4.7) | 0.099 | –0.04 (–0.08 to –0.00) | 0.042 | 2.4 (–0.0 to 4.9) | 0.054 | –0.05 (–0.09 to –0.01) | 0.019 | |||
| Subsidy v information | –0.5 (–3.6 to 2.5) | 0.74 | –0.01 (–0.06 to 0.04) | 0.67 | –0.8 (–3.7 to 2.2) | 0.62 | –0.01 (–0.05 to 0.04) | 0.80 | |||
| Incentive v information | 0.7 (–2.3 to 3.6) | 0.66 | –0.02 (–0.07 to 0.02) | 0.33 | 0.9 (–2.0 to 3.8) | 0.53 | –0.03 (–0.07 to 0.02) | 0.25 | |||
| Incentive v subsidy | 1.2 (–1.8 to 4.1) | 0.44 | –0.01 (0.67 to 0.59) | 0.59 | 1.7 (–1.2 to 4.6) | 0.26 | –0.02(–0.06 to 0.02) | 0.37 | |||
| Adjusted school level intra-class correlations | 0.08 | 0.09 | 0.08 | 0.09 | |||||||
*Control arm (no intervention); information arm (school principals received information about anaemia); subsidy arm (principals received information and unconditional subsidies for reducing anaemia); incentive arm (principals received information, subsidies, and financial incentives for reducing anaemia).
†Estimated with multilevel mixed effect linear regression adjusting for baseline Hb status and clustering at the school and county level.
‡Estimated with multilevel mixed effect logistic regression adjusting for baseline Hb status and clustering at the school and county level.
§Estimated with multilevel mixed effect linear regression adjusting for baseline Hb status, whether the principal was eligible for a bonus based on student exam scores, student sex, student age, whether the student boards at school, mother’s migration status, and mother’s education level, and clustering at the school and county level.
¶Estimated with multilevel mixed effect logistic regression adjusting for baseline Hb status, whether the principal was eligible for a bonus based on student exam scores, student sex, student age, whether the student boards at school, mother’s migration status, and mother’s education level and clustering at the school and county level.
Subgroup analysis: test score incentives
Although this was not originally planned, we next analysed how the impact of each intervention varied with pre-existing incentives for principals to achieve good academic performance among students. Table 3 shows marginal effects separately for schools with and without principal incentives for good academic results, and the last two columns report differences associated with these incentives. In the presence of the incentives, the information and incentive interventions were substantially more effective, raising mean haemoglobin concentration by 9.8 g/L (4.1 to 15.5) and 8.6 g/L (2.1 to 15.1) more than schools without them. Corresponding reductions in the probability of anaemia at study end were 11 percentage points (–0.19 to –0.03) in the information arm and 6 percentage points (–0.12 to 0.01) in the incentive arm (declines of 50% and 27% relative to the baseline prevalence of 22%). The effect of the subsidy arm did not vary significantly with academic performance incentives.
Table 3.
Interaction effects of allocation to trial interventions (information, subsidies, and incentives for reducing anaemia) with pre-existing incentives for school principals to achieve good academic performance among students on haemoglobin (Hb) concentration and anaemia prevalence
| Trial arms compared* | Without incentive for good academic results (n=2871 students, 58 schools)† | With incentive for good academic results (n=682 students, 14 schools)‡ | Interaction effect§ | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Change in Hb concentration (g/L)¶ | Anaemic at follow-up (Hb <115 g/L)** | Change in Hb concentration (g/L) | Anaemic at follow-up (Hb <115 g/L) | Change in Hb concentration (g/L) | Anaemic at follow-up (Hb <115 g/L) | ||||||||||||
| Marginal effect (95% CI) | P value | Marginal effect (95% CI) | P value | Marginal effect (95% CI) | P value | Marginal effect (95% CI) | P value | Difference (95% CI) | P value | Difference (95% CI) | P value | ||||||
| Information v control | –0.6 (–3.2 to 2.0) | 0.66 | 0.02 (–0.03 to 0.07) | 0.46 | 9.2 (4.0 to 14.4) | <0.01 | –0.09 (–0.14 to –0.03) | <0.01 | 9.8 (4.1 to 15.5) | <0.01 | –0.11 (–0.19 to –0.03) | 0.01 | |||||
| Subsidy v control | –0.2 (–2.8 to 2.5) | 0.91 | –0.01 (–0.06 to 0.04) | 0.65 | 2.6 (–2.5 to 7.7) | 0.32 | –0.04 (–0.10 to 0.02) | 0.24 | 2.7 (–3.1 to 8.6) | 0.35 | –0.03 (–0.10 to 0.05) | 0.51 | |||||
| Incentive v control | 1.1 (–1.4 to 3.6) | 0.38 | –0.03 (–0.07 to 0.01) | 0.12 | 9.7 (3.7 to 15.6) | <0.01 | –0.09 (–0.15 to –0.03) | <0.01 | 8.6 (2.1 to 15.1) | 0.01 | –0.06 (–0.12 to 0.01) | 0.07 | |||||
| Subsidy v information | 0.4 (–2.6 to 3.4) | 0.78 | –0.03 (–0.09 to 0.03) | 0.30 | –6.6 (–12.6 to –0.6) | 0.03 | 0.05 (–0.01 to 0.11) | 0.09 | –7.0 (–13.7 to –0.4) | 0.04 | 0.08 (–0.00 to 0.17) | 0.06 | |||||
| Incentive v information | 1.7 (–1.2 to 4.5) | 0.25 | –0.05 (–0.11 to 0.00) | 0.07 | 0.5 (–6.3 to 7.3) | 0.89 | –0.00 (–0.03 to 0.03) | 0.94 | –1.2 (–8.6 to 6.1) | 0.75 | 0.05 (–0.01 to 0.11) | 0.11 | |||||
| Incentive v subsidy | 1.3 (–1.6 to 4.1) | 0.39 | –0.02 (–0.07 to 0.03) | 0.42 | 7.1 (0.3 to 13.9) | 0.04 | –0.05 (–0.12 to 0.01) | 0.09 | 5.8 (–1.5 to 13.2) | 0.12 | –0.03 (0.04 to 0.04) | 0.40 | |||||
*Control arm (no intervention); information arm (school principals received information about anaemia); subsidy arm (principals received information and unconditional subsidies for reducing anaemia); incentive arm (principals received information, subsidies, and financial incentives for reducing anaemia).
†Marginal treatment effects evaluated for schools where principal did not have bonus incentives based on student academic performance.
‡Marginal treatment effects evaluated for schools where principal had bonus incentives based on student academic performance.
§Interaction effect is the difference in marginal treatments effects between schools in which principals had bonus incentives for academic performance and those in which they did not.
¶Results for change in Hb concentration between baseline and follow-up are estimated with multilevel mixed effect linear regression adjusting for baseline Hb status, whether the principal is eligible for a bonus based on academic performance, student sex, student age, whether the student boards at school, mother’s migration status, and mother’s education level and clustering at the school and county level.
**Results for anaemia status at follow-up are estimated with multilevel mixed effect logistic regression adjusting for baseline Hb status, whether the principal is eligible for a bonus based on academic performance, student sex, student age, whether the student boards at school, mother’s migration status, and mother’s education level and clustering at the school and county level.
Supplemental analysis: behavioural responses to interventions
The appendix and its figure in the data supplement present analyses of how the school principals and parents responded to the study’s interventions—both to reduce anaemia and in potentially unintended ways. We summarise its main findings here.
Principals’ strategies to reduce anaemia
The appendix presents three stylised facts about principals’ major strategies for reducing anaemia.
(1) Compared with those in the subsidy group, school principals in the incentive group were more likely to use their subsidies to pursue iron supplementation strategies that increased only students’ multimicronutrient intake and not their energy intake (for example, with vitamins or minerals and iron fortified wheat) and were less likely to pursue broad feeding strategies that increased both multimicronutrient intake and energy intake (for example, with meat and other food added to school lunches). This response is consistent with these principals’ narrow incentive to raise iron levels.
(2) Consistent with research linking broader nutritional gains to school performance,30 principals with incentives to achieve good academic performance focused relatively more on feeding.
(3) Compared with those in the control group, principals in all the intervention groups provided nutritional information to parents to reduce anaemia. However, principals in the information arm with academic performance incentives seemed to have done so most intensively. This may be because these principals understood the link between anaemia and school performance (an explicit part of the information they received) and also had incentives to improve academic performance but lacked the resources to intervene through schools.
Unintended behavioural responses by parents and principals
We also investigated the possibility of unintended (perverse) behavioural responses to the study’s interventions. First, we examined if there were offsetting reductions in dietary quality at home in response to improvements in school meals.31 32 Second, we estimated if school spending on functions other than nutrition (on administration or teaching, for example) was diverted to reduce anaemia. We found little evidence of either.
Discussion
This paper reports first evidence on the effectiveness of direct provider incentives for better health outcomes. On balance, we found that these incentives were moderately effective in reducing anaemia rates among school children in rural China. Importantly, we also found that analyses failing to recognise their interactions with other incentives could miss important population health gains associated with them. Primary school principals with incentives for good academic performance seem to have made more effective use of subsidies to reduce anaemia—and performance pay for anaemia reduction may have nearly doubled their impact. Moreover, although the provision of information about anaemia to school principals was ineffective on average, when combined with principal incentives for good academic performance, information was associated with significant health gains on a par with the full bundle of information, subsidies, and anaemia reduction incentives. School principals in the information arm with incentives for good academic performance had strong incentives to reduce anaemia, but, because they lacked resources to do so through schools, they pursued innovative strategies to improve students’ diets at home that ultimately proved effective. Considerably more research is needed, but our findings broadly suggest that policymakers may be able to exploit synergies between financial incentives and other motives or pre-existing incentives (often embedded in local institutions and cultural context) to produce substantial population health gains.
Several limitations are important to note. First, as in other studies,33 34 there is presumably error in our haemoglobin concentration measurement despite our explicit protocol to minimise it. However, this error should be uncorrelated with assignment to experimental arms. Second, we selected study schools according to criteria set by China’s government in its policy for consolidating rural schools. However, our results may not generalise to other Chinese schools or to other cultural and institutional settings—particularly where performance incentives are less common. Third, our specific randomisation procedure was re-randomisation until achieving balance in haemoglobin concentration across the four study arms (with 95% confidence). This approach is valid, but other procedures offer additional benefits.35 Fourth, our study was not powered ex ante to detect meaningful differences between intervention arms. Finally, because incentives for academic performance were not randomly assigned, interactions with trial arms should be interpreted cautiously.
Further research is needed to understand several important dimensions of performance pay for health improvement. These include (a) understanding how to structure performance pay in multitasking environments—in particular, for providers with broader clinical responsibility36; (b) identifying optimal combinations of fixed (unconditional) and variable (performance based) compensation (research on executive compensation suggests that smaller unconditional components may be appropriate—but this varies with the degree to which agents can influence the rewarded outcome)37 38 39; (c) examining the relative merits of performance pay at different staff levels within organisations; and (d) documenting the comparative effectiveness and cost effectiveness of financial and non-financial incentives (given that financial incentives can be costly).
Overall, our study suggests that performance incentives—and direct rewards for health improvement in particular—may hold promise for promoting a broad class of seemingly under-used health technologies more effectively. However, it also offers a cautionary tale about introducing incentives naively without understanding the local institutions within which they are applied. Failing to do so may hamper their effectiveness, while harmonising them with other motives and incentives can amplify their impact.
What is already known on this topic
Inexpensive, highly effective technologies and services exist for improving human health in developing countries, but implementation and coverage are often low
Rewarding the ultimate objective—health improvement—without specifying how it should be achieved can strengthen incentives for creativity and innovation in service delivery
However, performance pay based directly on health outcomes in a developing country has not been tried or evaluated
What this study adds
Direct provider incentives for better health outcomes were moderately effective in reducing anaemia among primary school children in rural China
Importantly, analyses failing to recognise interactions with pre-existing motives and incentives may miss important sub-population health gains
We thank the Xi’an Jiaotong University School of Medicine for assistance with the surveys; the Chinese Ministry of Education for its cooperation; and the school principals, teachers, and students for their participation. We are grateful to Ershiyi Jinweita for their assistance procuring the vitamins used in this study. We are indebted to the generous support of Eric Hemel and Barbara Morgen. Finally, Eran Bendavid (MD, MS, Stanford University) and Marcos Vera-Hernández (PhD, University College London) provided helpful feedback on preliminary versions of the paper.
Contributors: GM, RM, LZ, and SR formulated the concept and designed the study. RL generated the randomisation sequence and enrolled schools. RL, LZ, YS, and QZ managed all the interventions, data collection, and other field activities. RL, SS, and PF managed the data and conducted the statistical analysis with guidance from GM, SR, LZ and RM. QZ, YS, and AM assisted in statistical analysis and interpretation. GM and SR wrote the manuscript, and all other authors participated in critical revisions. GM and LZ were essential in obtaining project funding. SY, LZ and AM provided administrative and technical project support. All authors were able to comment on the draft report and saw and approved the final version. GM, SS, SR and RL had full access to all of the data used in the study and accept responsibility for the integrity of the data and the accuracy of the data analysis.
Funding: The project was funded by the International Initiative for Impact Evaluation (3IE), the Leibniz Institute of Agricultural Development in Central and Eastern Europe (IAMO), and Stanford University (the China Fund at the Freeman Spogli Institute for International Studies, the Stanford Center for International Development, and the Presidential Fund for Innovation in International Studies). Supplemental funding to the Rural Education Action Project (REAP) was provided by Pfizer.
None of the funders or material contributors to this project played any role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Data sharing: The technical appendix, statistical code, and data set are available from the corresponding author at ngmiller@stanford.edu.
Cite this as: BMJ 2012;345:e4809
Web Extra. Extra material supplied by the author
Appendix: Extra statistical analyses. (1) Behavioural responses among school principals to the study’s interventions to reduce anaemia; (2) the possibility of unintended consequences attributable to these interventions; (3) factors correlated with local governments’ use of incentives for principals to achieve good academic performance among students
Appendix figure: Difference in the distributions of haemoglobin concentrations
References
- 1.Loevinsohn B, Harding A. Buying results? Contracting for health service delivery in developing countries. Lancet 2005;366:676-81. [DOI] [PubMed] [Google Scholar]
- 2.Doran T, Foolwood C, Gravelle H, Reeves D, Kontopantelis E, Hiroeh U, et al. Pay-for-performance programs in family practices in the United Kingdom. N Engl J Med 2006;355:375-84. [DOI] [PubMed] [Google Scholar]
- 3.Soeters R, Habineza C, Peerenboom PB. Performance-based financing and changing the district health system: experience from Rwanda. Bull World Health Organ 2006;84:884-9. [PMC free article] [PubMed] [Google Scholar]
- 4.Rosenthal M, Dudley RA. Pay-for-performance: will the latest payment trend improve care? JAMA 2007;297:740-4. [DOI] [PubMed] [Google Scholar]
- 5.Guterman S, Davis K, Schoenbaum S, Shih A. Using Medicare payment policy to transform the health system: a framework for improving performance. Health Aff 2009;28:W238-50. [DOI] [PubMed] [Google Scholar]
- 6.Maynard A, Bloor K. Will financial incentives and penalties improve hospital care? BMJ 2010;340:c88. [DOI] [PubMed] [Google Scholar]
- 7.Montagu D, Yamey G. Pay-for-performance and the Millennium Development Goals. Lancet 2011;377:1383-5. [DOI] [PubMed] [Google Scholar]
- 8.Basinga P, Gertler PJ, Binagwaho A, Soucat ALB, Sturdy J, Vermeersch CMJ. Effect on maternal and child health services in Rwanda of payment to primary health-care providers for performance: an impact evaluation. Lancet 2011;377:1421-8. [DOI] [PubMed] [Google Scholar]
- 9.Bloom E, King E, Bhushan I, Kremer M, Glingingsmith D, Loevinsohn B, et al. Contracting for health: evidence from Cambodia. Brookings Institution, 2006.
- 10.Wang L. The general survey report of nutrition and health in China in 2002. People’s Medical Publishing House, 2005.
- 11.Chen J, Zhao X, Zhang X, Yin S, Piao J, Huo J, et al. Studies on the effectiveness of NaFeEDTA-fortified soy sauce in controlling iron deficiency: a population-based intervention trial. Food Nutr Bull 2005;26:177-208. [DOI] [PubMed] [Google Scholar]
- 12.Wang W. Nutrition status quo of school-aged children in rural Shaanxi. School of Public Health, Xi’an Jiaotong University, 2007.
- 13.Halterman JS, Kaczorowski JM, Aligne CA, Auinger P, Szilagyi G. Iron deficiency and cognitive achievement among school-aged children and adolescents in the United States. Pediatrics 2001;107:1381-6. [DOI] [PubMed] [Google Scholar]
- 14.Stoltzfus RJ. Iron-deficiency anemia: reexamining the nature and magnitude of the public health problem. J Nutr 2001;131:697-700S. [DOI] [PubMed] [Google Scholar]
- 15.Yip, R. Iron. In: Bowman BA, Russell RM, eds. Present knowledge in nutrition. 8th ed. International Life Sciences Institute Press, 2001:311-28.
- 16.Bobonis GJ, Miguel E, Puri-Sharma C. Anemia and school participation. J Hum Resour 2006;41:692-721. [Google Scholar]
- 17.Stoltzfus RJ. 2011. Iron interventions for women and children in low-income countries. J Nutr 2011:141:756-62S. [DOI] [PubMed]
- 18.World Health Organization. Iron deficiency anaemia: assessment, prevention and control: a guide for programme managers. WHO, 2011.
- 19.Bundy DAP, Shaeffer S, Jukes M, Beegle K, Gillespie A, Drake L, et al. School based health and nutrition programs. In: Jamison D, Claeson M, Breman J, Meacham A, eds. Disease control priorities for developing countries. Oxford University Press, 2006:20-75.
- 20.Luo R, Shi Y, Zhang L, Liu C, Rozelle S, Sharbono B, et al. Nutrition and educational performance in rural China’s elementary schools: results of a randomized controlled trial in Shaanxi Province. Econ Develop Cultural Change (forthcoming).
- 21.Gibbons R, Murphy K. Optimal incentive contracts in the presence of career concerns: theory and evidence. J Polit Econ 1992;100:468-505. [Google Scholar]
- 22.Leonard K, Masatu M, Vialou A. Getting doctors to do their best: the roles of ability and motivation in health care quality. J Hum Resour 2007;42:682-700. [Google Scholar]
- 23.Kroke A, Klipstein-Grobusch K, Voss S, Moseneder J, Thielecke F, Noack R, et al. Validation of a self-administered food-frequency questionnaire administered in the European Prospective Investigation into Cancer and Nutrition (EPIC) Study: comparison of energy, protein, and macronutrient intakes estimated with the doubly labeled water, urinary nitrogen, and repeated 24-h dietary recall methods. Am J Clin Nutr 1999;70:439-47. [DOI] [PubMed] [Google Scholar]
- 24.Dunnett CW. A multiple comparison procedure for comparing several treatments with a control. J Am Stat Assoc 1955;50:1096-121. [Google Scholar]
- 25.Dunnett CW. New tables for multiple comparisons with a control. Biometrics 1964;20:482-91. [Google Scholar]
- 26.Education law of the People’s Republic of China (adopted at the third meeting of the eighth National People’s Congress). In: Education law in China. Law Press China, 1995.
- 27.Whiting S. The cadre evaluation system at the grass roots: the paradox of party rule. In: Naughton B, Yang D, eds. Holding China together: diversity and national integration in the post-Deng era. Cambridge University Press, 2004:101-19.
- 28.Rasmussen OD, Malchow-Møller N, Andersen TB. Walking the talk: the need for a trial registry for development interventions. J Develop Effect 2011;3:502-19. [Google Scholar]
- 29.Luo R, Kleiman-Weiner M, Rozelle S, Zhang L, Liu C, Sharbono B, et al. Anemia in rural China’s elementary schools: prevalence and correlations in Shaanxi province’s poor counties. Ecol Food Nutr 2010;49:357-72. [DOI] [PubMed] [Google Scholar]
- 30.Alderman H, Behrman J, Lavy V, Menon R. Child health and school enrollment: a longitudinal analysis. J Hum Resour 2001;36:185-205. [Google Scholar]
- 31.Babu S, Hallam A. Socioeconomic impacts of school feeding programmes: empirical evidence from a south Indian village. Food Policy 1989;14:58-66. [DOI] [PubMed] [Google Scholar]
- 32.Jacoby H. Is there an intrahousehold “flypaper effect”? Evidence from a school feeding programme. Econ J 2002;112:196-221. [Google Scholar]
- 33.Morris S, Ruel M, Cohen R, Dewey K, de la Brière B. Precision, accuracy, and reliability of hemoglobin assessment with use of capillary blood. Am J Clin Nutr 1999;69:1243-8. [DOI] [PubMed] [Google Scholar]
- 34.Boulton F, Nightingale M, Reynolds W. Improved strategy for screening prospective blood donors for anaemia. Transfus Med 1997;4:221-5. [DOI] [PubMed] [Google Scholar]
- 35.Bruhn M, McKenzie D. In pursuit of balance: randomization in practice in development field experiments. Am Econ J Appl Econ 2009;1:200-32. [Google Scholar]
- 36.Holmstrom B, Milgrom P. Multitask principal-agent analyses: incentive contracts, asset ownership, and job design. J Law Econ Organ 1991;7:24-52. [Google Scholar]
- 37.Jensen M, Murphy K. CEO incentives—it’s not how much you pay, but how. Harv Bus Rev 1990;68:138-53. [PubMed] [Google Scholar]
- 38.Hall B, Liebman J. Are CEOs really paid like bureaucrats? Q J Econ 1998;113:653-91. [Google Scholar]
- 39.Raith M. Specific knowledge and performance measurement. RAND J Econ 2008;39:1059-79. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Appendix: Extra statistical analyses. (1) Behavioural responses among school principals to the study’s interventions to reduce anaemia; (2) the possibility of unintended consequences attributable to these interventions; (3) factors correlated with local governments’ use of incentives for principals to achieve good academic performance among students
Appendix figure: Difference in the distributions of haemoglobin concentrations