Perils of scaling up: Effects of expanding a nutrition programme in Madagascar

Abstract

Scaling up integrated nutrition programmes from small, targeted interventions or pilot studies to large‐scale government‐run programmes can be challenging, with risks of changing the nature and quality of the interventions such that effectiveness is not sustained. In 1999, the Government of Madagascar introduced a nationwide, community‐based, growth‐monitoring and nutrition education programme, which was gradually scaled up throughout the country until 2011. Data from three nationally representative surveys, administered pre‐ and post‐programme implementation, in participating and non‐participating communities, were used to evaluate the effectiveness of the programme to reduce malnutrition in children under 5 after two phases of expansion (1999–2004 and 2004–2011). In our analyses, we compared “original” communities, who had initiated the programme during the first phase, and “new” communities, who initiated the programme during the second phase. “Original” communities demonstrated a significant effect on mean weight‐for‐age and on the prevalence of underweight by 2004; this effect was sustained at a reduced level through 2011. In contrast, “new” communities showed no benefits for any childhood nutritional outcomes. An explanation for these findings may be that community health workers in the “new” communities reported lower motivation and less use of key messages and materials than those in the “original” communities. Frontline workers reported increased workload and irregular pay across the board during the second phase of programme expansion. Our findings underscore the risk of losing effectiveness if programme quality is not maintained during scale‐up. Key factors, such as training and motivation of frontline workers, are important to address when bringing a programme to scale.

Key messages.

• Madagascar's long‐standing commitment to eradicating childhood malnutrition resulted in the expansion of its National Nutrition Programme to reach coverage at scale.

• The National Nutrition Programme was effective in reducing malnutrition during early programme scale‐up and a period of economic growth. This benefit was sustained, but at a diminished level, during a phase of political instability and severe economic decline. In contrast, there was no sustained impact in communities that joined during the second phase of programme expansion.

• Programme scale‐up is necessary for achieving reductions in malnutrition at the population level. Our results demonstrate how programme expansion without an attention to quality can undermine programmatic effectiveness.

1. INTRODUCTION

Nutrition programmes must be brought to scale in low‐ and middle‐income countries in order to achieve the global commitment to ending all forms of malnutrition (Richter et al., 2017; Ruel & Alderman, 2013). In spite of an existing knowledge base about what nutrition interventions are most effective to improve health outcomes (Bhutta et al., 2014; Black et al., 2008), there is an urgent need for cost‐effective, scalable approaches that can address the entire population of children who are at risk of malnutrition and poor development. In addition to the challenge of bringing programmes and policies to scale (Bhutta, Lassi, Pariyo, & Huicho, 2010; Nabarro, 2010), there is the additional challenge of integrating new programmes and approaches into existing policies and systems (de Jongh, Gurol‐Urganci, Allen, Jiayue Zhu, & Atun, 2016; Mounier‐Jack, Mayhew, & Mays, 2017). In this paper, we present a case study from Madagascar, one of the few countries in sub‐Saharan Africa with a long‐standing commitment to addressing malnutrition through a large‐scale programme integrated within an existing health delivery infrastructure (Marek, Diallo, Ndiaye, & Rakotosalama, 1999).

There is strong evidence demonstrating that interventions can be effective in preventing maternal and infant death from malnutrition (Bhutta et al., 2014), as well as chronic underweight and wasting (Black et al., 2008). There is also a consensus on the need for several levels of action to tackle nutritional issues, including nutrition‐specific actions (e.g., micronutrient supplementation or fortification and dietary diversification), nutrition‐sensitive actions (e.g., promotion of food security and social safety nets), and enabling policy/political environments (Black et al., 2013; Ruel & Alderman, 2013). Although theoretically it should be possible to provide the same level of goods or services to small or large communities, there are logistical constraints that can get in the way of expanding the reach of existing services to larger or more remote communities (Moffitt, 2006). In general, efforts to scale up nutrition in the context of low‐income countries are facilitated by core investments in human, organizational, and financial resources (Pelletier et al., 2013). In particularly vulnerable countries and regions, such as in fragile or conflict‐affected states, good governance has been identified as the key consideration determining whether scaling‐up nutrition will be successful (Taylor, Perez‐Ferrer, Griffiths, & Brunner, 2015). Integration into existing health systems can also make efforts to promote nutrition more effective, but such integration can also be challenging (Atun, de Jongh, Secci, Ohiri, & Adeyi, 2010; Mounier‐Jack et al., 2017).

The overall aim of this study was to examine the effectiveness of Madagascar's government‐run programme at reducing childhood malnutrition during a 14‐year programme expansion, using three nationally representative surveys that tracked communities over time. The first objective of our analysis was to exploit the gradual phase‐in of the programme—first during a period of stable economic growth and later following a severe political and economic crisis—to estimate whether the programme had any effect on the prevalence of childhood malnutrition and document how the impact varied over time. The second objective of the analysis was to explore possible reasons for changes in programme impact over time. For this second objective, we focused our analysis on differences in the socio‐demographic characteristics of the community frontline workers, their skills and work motivation, as well as their training, which was variable across different phases of expansion of the programme.

2. METHODS

2.1. Programme and context

Madagascar is one of the poorest countries in the world, with 78% of people living below the international poverty line of $1.90 a day and nearly 50% of children under 5 years suffering from chronic malnutrition (UNICEF Madagascar, 2017). The nutrition programme in Madagascar was introduced in 1999 as a government‐run project funded by a World Bank loan and gradually rolled out during a first phase of expansion to reach approximately 3,600 project sites in about half of the districts nationwide by 2004 (Table 1). After 2004, the programme became the National Community Nutrition Programme under the National Office of Nutrition and was expanded to reach all districts across the country, reaching over 5,500 sites by 2011.

Table 1.

Programme components, implementation arrangements, and survey waves and sites

Note. ADE: Enquête Anthropométrique et Développement de l'Enfant; USAID: United States Agency for International Development.

Madagascar's programme is centred on community‐based monthly growth‐monitoring sessions as an entry point to provide nutrition education and promote behavioural change in feeding and hygiene practices for pregnant/lactating women and children under 5 in the community (Table 1). The programme relies on local frontline community health workers (CHWs), elected by the community for service delivery. Children who lose weight or are identified as moderately or severely malnourished (by standard World Health Organization [WHO] reference definitions) are referred to health centres for treatment. During the monthly meetings, the CHW provides nutrition education and facilitates cooking demonstrations, where basic recipes using locally available food are prepared and shared with participating mothers and children. To ensure wide geographical coverage, the programme is implemented by contracting local non‐governmental organizations to perform monitoring and supervision of the CHWs. The CHWs receive a monthly salary payment of ~$16 and reimbursement of expenses of an additional ~$7 for purchases of materials for cooking demonstrations (UNICEF Madagascar, 2017).

An evaluation after the first phase of scale‐up in 2004 found significant effects of the programme on child weight‐for‐age and on the prevalence of underweight (low weight‐for‐age)—the target indicators for the intervention (Galasso & Umapathi, 2009). The success and ownership of the community‐based programme spurred the creation of the National Nutrition Office and the second phase of scale‐up after 2005. However, a period of economic growth from 2001 to 2008 (on average 5% per year) was interrupted by a coup d'état in 2009, which resulted in a severe economic downturn that lasted through 2012. Despite this political and economic instability, and a temporary suspension of external funding, the government maintained its commitment to nutrition using limited domestic finances to cover the salary of government employees, when possible. The programme remained operational, due in large part to the motivation of the frontline workers.

2.2. Data sources

The evaluation of the government‐run programme drew from three waves of nationally representative anthropometric surveys of the population of households with pregnant women and children aged 0 to 59 months (Enquête Anthropométrique et Développement de l'Enfant or ADE hereafter) in Madagascar (Table 1). The surveys were modelled after the Demographic and Health Surveys but differed in that the ADE tracked the same communities longitudinally. At each survey wave, a new cross‐section of households was sampled in these communities. The three survey rounds, administered in 1997/1998, 2004, and 2011 by Madagascar's National Institute of Statistics, were aligned with the timing of the expansion of the community nutrition programme. The 1997/1998 survey fielded just before programme launch was administered with the objective of estimating the prevalence of childhood malnutrition at the district level in order to target the intervention to the districts with underweight prevalence above the national average (43%). The 2004 survey was administered during the same season as the pre‐programme survey and after the first wave of programme expansion. The 2011 survey was administered during the same season as the previous waves and after the second wave of scale‐up throughout the country. The three surveys allowed us to capture the potential impact as communities came into and out of the programme. Survey data for a group of communities that never participated allowed us to control for underlying trends in outcomes in the absence of the programme, both during periods of economic growth and during periods of economic and political crisis.

2.3. Survey sample

The sample in the 1997/1998 survey included 420 communities. Communities were categorized into four groups depending on their treatment status at the time of each survey: communities that never participated (“never”), those that entered during the first phase of the programme between 1999 and 2004 and continued through to 2011 (“original”), those that participated between 2004 and 2011 (“new”), and those that entered before 2004 but closed the programme prior to 2011 (“closed”; Table 1). Because the programme targeted rural and semi‐urban communities, clusters sampled in the capital and all provincial capitals (n = 26) and communities that were not visited in 2004 and 2011 (n = 12) were excluded from the analysis. The final analytic sample included 408 communities: 124 “never” participated, 142 “original” or continuously participating, 126 “new” (participating in 2011 but not in 2004), and 16 communities that “closed” the programme after 2004. Additional information on the ADE surveys and sample is available in the Supporting Information.

2.4. Measures

2.4.1. Childhood nutrition indicators

The main nutritional outcomes of interest were weight‐for‐age z‐score (WAZ) and height‐for‐age z‐score (HAZ), based on the WHO 2006 growth standards (WHO, 2011). Binary indicators were calculated for moderate underweight and stunting, which reflect WAZ and HAZ, respectively, of two standard deviations below the median of the reference population for the WHO 2006 growth standards.

2.4.2. CHW characteristics, training, and working conditions

In participating communities, the surveys included a questionnaire for the CHWs, with questions about their socio‐economic background, training, programme activities, and involvement in non‐programme work. In 2011, an explicit effort was made to track and interview up to three community workers per site even if they had stopped working, to document ruptures or interruptions of service delivery in the midst of the ongoing political crisis. Also in 2011, we included a motivation at work module adapted from Gagné and Deci (2005) based on organizational behaviour and psychological theories of self‐determination that describe external motivation (stemming from external pressure and economic incentives) and intrinsic motivation (stemming from personal values, beliefs, and pro‐social behaviour) as important drivers for performance.

In a subset of 150 clusters, we added a direct assessment of the CHWs' receptive vocabulary knowledge using a Malagasy adaptation of the Peabody Picture Vocabulary Test–third edition (Dunn & Dunn, 1997). CHW motivation and vocabulary knowledge were collected to account for possible differences in the selection of frontline workers as well as differences in their provision of service over time. In the analysis, we present results for both the raw total score for vocabulary knowledge as well as a residual score, conditional on CHW age, education, and dialect spoken.

2.4.3. Programme participation and knowledge gain

In both 2004 and 2011, the surveys included a special survey module administered to females in programme villages about their participation and perceived knowledge gained about nutrition information (on a 4‐point Likert scale) from the programme.

2.4.4. Demographic and socio‐economic characteristics

Data were collected on basic demographic and socio‐economic information at the village and household levels, including main economic activities (at the village level) and recent history of agro‐climatic shocks. Household‐level information included the age, sex, and education of all household members.

2.5. Statistical analysis

Our analysis was performed in three stages. First, we described differences in community and programme site characteristics for the groups of communities that experienced varying programme status over time (“original,” “new,” “closed,” and “never” participating communities). Second, we exploited the gradual programme phase‐in to estimate programme impact on children's nutritional outcomes and document the evolution of impact over time. Finally, we compared the characteristics, training, and working conditions of the CHWs, as well as participants' perception of knowledge gained, to examine possible reasons for changes in programme impact over time.

2.5.1. Programme impact

To estimate the impact of the intervention on children's nutritional status at different points in time, we used the method of difference in differences (DD hereafter; Ashenfelter & Card, 1985; Imbens & Rubin, 2015). DD is commonly used to evaluate the effect of a programme in a quasi‐experimental pre–post study design to address potential bias from selection into the programme and overall time trends. The method of DD entails comparing the averages (or means) for “original” and “new” sites to “never” sites without the programme (first difference), before and after the programme started (i.e., in 2004 and 2011; second difference).

We tested three regression model specifications: (a) ordinary least squares (OLS) with a DD specification, (b) DD with community fixed effects to account for any additive observable or unobservable differences across communities, and (c) DD with community fixed effects (FE) reweighted by the product of two sets of weights. The first set of weights was based on a propensity score of programme assignment during the first expansion in 2004 as a function of village level socio‐economic characteristics and recent history of agro‐climatic shocks. The second set of weights modelled subsequent entry in 2011. We combined DD with reweighting to balance the villages on observed characteristics and account for potential differences in selection into the programme and into different durations of the programme by initial socio‐economic conditions (Hirano, Imbens, & Ridder, 2003). The maintained assumption was that any difference in unobservable characteristics that led communities to select into the programme or different programme duration was the same across sites.

We also performed a sensitivity analysis by trimming the sample to areas of common support (i.e., excluding areas at the extremes of the propensity score, below 0.1 and above 0.9; Crump, Hotz, Imbens, & Mitnik, 2009; Imbens, 2015). Standard errors of estimates were adjusted for clustering at the community level and significant differences shown at the 90%, 95%, and 99% levels. Model covariates included child age, birth order indicator and gender, and education level of the mother (and for the first specification, rural and province indicators). We excluded villages with closed sites from the weighted regression due to the small sample size. Additional information on the model specifications and weight calculations is available in the Supporting Information.

2.5.2. CHW characteristics, training, work conditions, and participant perceptions

To provide suggestive evidence of what might account for the differences in impact between “original” and “new” sites, we compared characteristics of the CHWs by site type, both without and with reweighting (using the same weights as those used in the impact estimation) to account for potential differences in selection by initial conditions. Because there was some turnover of CHWs over time, CHW characteristics and training were used for the most recently employed CHWs in each site. We also compared programme participation and women's perceptions of knowledge gained about nutrition (without and with reweighting).

2.6. Ethical considerations

The study was reviewed and approved by a local Malagasy human subjects committee, and participants were not included in the study unless they provided verbal consent to participate.

3. RESULTS

3.1. Descriptive statistics

The programme was implemented in rural or peri‐urban communities throughout the country (Table 2, panel A). More than three fourths of the clusters were rural, with agriculture and livestock‐rearing as main economic activities. Among programme communities, sites that started in the first phase of the programme (“original” and “closed”) were generally worse off nutritionally at baseline than the “never” communities, reflecting the programme's objective of targeting districts with a higher malnutrition rate at the onset of the programme (Figure 1). For example, the difference in the prevalence of underweight at baseline (1997/1998) between the “original” group and the “never” group was about 8 percentage points. Similarly, the baseline prevalence of stunting in the “original” group was 7% higher when compared with the “never” group. “Original” communities were worse off along other socio‐economic characteristics: They had lower educated mothers (with higher shares of women with primary school education in both 1997/1998 and 2004 than in “new” and “never” sites) and were less well‐connected (with higher access to a secondary road rather than to a paved road). The “new” sites were more likely to raise livestock for income generation but less likely to engage in commerce than the other site types. Finally, “closed” and “never” communities were more likely to be peri‐urban, compared with the “original” or “new” sites, which were more likely to be rural. Weighted standardized differences for these village characteristics by programme type (“original” vs. “never,” “new” vs. “never,” and “original” vs. “new”) are substantially lower than the non‐weighted standardized differences, which suggests that the reweighting performed well in achieving balance in observable characteristics across groups (Table S1).

Table 2.

Descriptive statistics for the communities and sites by treatment status

(A) Community characteristics	Closed (N = 16)	Original (N = 142)	New (N = 126)	Never (N = 124)
	M (SD)	M (SD)	M (SD)	M (SD)
Population size (village)	1,092 (709)	1,187 (793.1)	1,411 (1235)	1,256 (916.3)
Number of households	230 (176)	245 (172)	286 (224)	268 (225)
Proportion women with primary schooling in 1997	0.49 (0.19)	0.54 (0.19)	0.48 (0.19)	0.5 (0.2)
Proportion women with primary schooling in 2004	0.59 (0.21)	0.58 (0.17)	0.54 (0.17)	0.55 (0.19)
	Proportion (n)	Proportion (n)	Proportion (n)	Proportion (n)
Agriculture is a main activity	0.88 (14)	0.97 (137)	0.94 (119)	0.94 (116)
Livestock is a main activity	0.75 (12)	0.75 (107)	0.85 (107)	0.73 (91)
Commerce is a main activity	0.25 (4)	0.16 (23)	0.10 (13)	0.26 (32)
Rural location	0.63 (10)	0.87 (124)	0.84 (106)	0.77 (95)
Has health facility	0.81 (13)	0.82 (116)	0.79 (99)	0.82 (101)
Has access to safe water in the dry season	0.38 (6)	0.37 (53)	0.37 (46)	0.37 (46)
Has electricity	0.44 (7)	0.18 (26)	0.23 (29)	0.27 (33)
Experienced lean seasons	0.94 (15)	0.93 (132)	0.94 (115)	0.94 (116)
Has access to paved road	0.31 (5)	0.23 (32)	0.24 (30)	0.27 (34)
Has access to secondary road	0.31 (5)	0.56 (80)	0.37 (47)	0.34 (42)
Has water course access	0.19 (3)	0.13 (18)	0.26 (33)	0.25 (31)

(B) Site characteristicsa	Closed (N = 8–9)	Original (N = 140)	New (N = 109–110)
	M (SD)	M (SD)	M (SD)
Number of CHW beforehand	0.67 (0.87)	0.31 (0.56)	0.49 (0.69)
Distance site to commune (min)	45.8 (40.1)	40.1 (36.5)	35.8 (34.5)
Max distance of households to site (min)	71.7 (23.4)	62.0 (29.1)	57.8 (27.1)
Number villages per site	2.89 (1.54)	2.24 (1.32)	1.72 (1.13)
	Proportion (n)	Proportion (n)	Proportion (n)
Site is permanent	0.44 (4)	0.61 (86)	0.42 (46)
Most families live near site	0.56 (5)	0.64 (90)	0.73 (80)
Site has a latrine	0.38 (3)	0.31 (44)	0.21 (23)
Village has CSB health centre	0.67 (6)	0.41 (57)	0.46 (50)

Note. CHW: community health worker; CSB (Centres Santé de Base): basic health center.

^a Site characteristics were reported by the CHWs and are missing for some sites (N differs between village and site characteristics).

Figure 1.

Line graphs for the proportion of (a) underweight and (b) stunting among children under 5 years by survey year for each of the four site types (“original,” “new,” “never,” and “closed”). Included under each line graph is a table of the raw mean underweight or stunting proportion with standard deviation (SD), total number of children by site type (n), and number of sites by site type.

Among programme communities (Table 2, panel B), “original” sites have a larger and more dispersed catchment area, covering more villages per site, than “new” sites (2.24 vs. 1.72 villages per site). In addition, the physical site provided by the community to carry out the programme activities was nearly 1.5 times more likely to be permanent, which is suggestive of the longer term engagement of the community.

3.2. Programme impact

Table 3 presents the main results of the impact of the programme on nutritional outcomes by type of community over time. Panel A shows the impact on the continuous z‐scores for weight‐for‐age and height‐for‐age, and panel B shows the impact on the binary indicators of moderate underweight and stunting. Estimates obtained with the OLS and reweighted fixed effects specifications are shown. P values for post‐estimation Wald tests are given to indicate whether programme effect estimates were statistically equal to each other (bottom of panels). Unless otherwise specified, quoted estimates in the text are from the OLS specification. An extended version of the table is available in Table S2, including estimates from the unweighted fixed effects specification.

Table 3.

Programme impact on primary nutritional outcomesi

(A) Continuous outcomes	WAZ		HAZ
	OLS	FE, weighted	OLS	FE, weighted
	Coefficient (β)i [95% CI]		Coefficient (β) [95% CI]
Impact “original” sites in 2004	0.277***	0.281***	0.154**	0.273**
	[0.161, 0.394]	[0.143, 0.419]	[0.024, 0.285]	[0.079, 0.467]
Impact “original” sites in 2011	0.178**	0.230**	0.08	0.211**
	[0.051, 0.305]	[0.062, 0.398]	[−0.058, 0.219]	[0.006, 0.416]
Impact “new” sites in 2011	−0.022	−0.034	0.012	0.002
	[−0.143, 0.098]	[−0.161, 0.093]	[−0.129, 0.153]	[−0.177, 0.181]
Test if β “original” in 2004 = β “original” in 2011 (P value)	0.02	0.5	0.25	0.45
Test if β “original” in 2011 = β “new” in 2011 (P value)	<0.001	<0.001	0.3	0.03

(B) Binary indicators	Underweight (WAZ < −2 SD)		Chronic malnutrition or stunting (HAZ < −2 SD)
	OLS	FE, weighted	OLS	FE, weighted
	Coefficient (β)i [95% CI]		Coefficient (β) [95% CI]
Impact “original” sites in 2004	−0.085***	−0.088***	−0.034*	−0.046*
	[−0.126, −0.044]	[−0.139, −0.037]	[−0.072, 0.003]	[−0.097, 0.005]
Impact “original” sites in 2011	−0.060**	−0.069**	−0.012	−0.05
	[−0.107, −0.013]	[−0.137, −0.001]	[−0.055, 0.031]	[−0.118, 0.018]
Impact “new” sites in 2011	0.015	0.050*	−0.014	−0.014
	[−0.030, 0.060]	[−0.009, 0.109]	[−0.060, 0.031]	[−0.083, 0.056]
Test if β “original” in 2004 = β “original” in 2011 (P value)	0.02	0.52	0.41	0.9
Test if β “original” in 2011 = β “new” in 2011 (P value)	<0.001	<0.001	0.93	0.23

Note. CI: confidence interval; FE: fixed effects; HAZ: height‐for‐age z‐score; OLS: ordinary least squares; WAZ: weight‐for‐age z‐score.

Programme impact was estimated using (a) OLS with a difference‐in‐differences specification and (b) difference in differences with community FE reweighted by the product of two set of weights to account for potential differences in selection into the programme and into different durations of the programme. Subsumed regressors are age and birth order dummies, gender, and education level of the mother (and for OLS, rural and province indicators). Weighted regression excludes villages with closed sites.

ⁱStandard error of estimated coefficients are adjusted for clustering at the community level.

^* P < 0.10.

^** P < 0.05.

^*** P < 0.001.

3.2.1. Continuous measures

“Original” sites that were rolled out during the first phase of the programme were significantly worse off nutritionally than those that never took up the programme (−0.289 SD and −0.201 SD for WAZ and HAZ, respectively; see Table S2), indicating the effort of targeting the neediest areas during the programme launch. By 2004, “original” sites caught up to “never” participating communities in WAZ (estimated gain, β = 0.277 SD). The gain in WAZ was sustained in 2011 (β = 0.178 SD), although the drop in magnitude of effect between 2004 and 2011 was statistically significant (P value = 0.02) under the OLS specification. The estimated gains in HAZ by 2004 were more modest (β = 0.154 SD) and were not significantly different from zero in 2011 in either of the OLS or fixed effects model specifications. The reweighted specification demonstrated a lack of stability in the estimated impact on HAZ for the “original” sites in 2011, with larger sustained effects over time compared with the other two specifications (β = 0.211 SD).

In contrast to the “original” sites, the “new” sites that were added after 2004 in the second phase of programme expansion showed no programme impact on either continuous nutritional outcome by 2011. The “new” sites did not exhibit a difference in outcomes compared with the “never” sites before 2004, confirming the validity of the empirical methodology used to assess impact.

Communities that closed before 2011 provide an interesting insight in the trajectory of programme impact: They were as worse off at baseline as the “original” communities, experienced a comparable gain in their WAZ during programme participation by 2004 (β = 0.206 SD), but had no impact in 2011 after the community sites had discontinued operation (Table S2).

3.2.2. Binary indicators

Table 3, panel B, and Figure 1a show that “original” communities experienced a significant reduction in the prevalence of underweight in 2004 nearly bridging the disparity with “never” communities in 1997/1998 (estimated reduction, β = −0.85). The reduction in underweight for “original” communities was sustained in 2011, although the estimated magnitude of the effect in 2011 (β = −0.060) was lower (accounting for a downward trend in “never” communities over time; see Figure 1a). As with the continuous measure, the drop in magnitude of effect between 2004 and 2011 in “original” sites was statistically significant (P value = 0.02) but only under the OLS specification.

The impact of participating on the prevalence of stunting was very small in 2004 for the “original” communities (<5%) and faded out in 2011 such that we cannot reject the hypothesis that the two estimates in 2004 and 2011 are statistically different from each other. Consistent with the continuous measures, “new” sites added after 2004 showed no impact (at the 95% confidence level) on either underweight or stunting prevalence by 2011 (Figure 1b).

With the exception of the estimated impact on HAZ in the “original” sites, the estimates across model specifications (OLS vs. village fixed effects vs. village fixed effects combined with weighting) were qualitatively comparable. The impact of the programme on WAZ for the “original” communities was significantly different from the impact for the “new” communities across all specifications. The results were also robust to trimming to areas of common support (data not shown).

3.3. Work conditions, CHW characteristics, and participant perceptions

What can explain the difference in impact across the two waves of programme expansion? All programme communities experienced increased population pressure over time (i.e., more children and mothers in the site census), resulting in an increasing workload of frontline workers. The political and economic crisis occurring between 2009 and 2012 added another layer of strain to the delivery of the intervention: the salary (and the cost reimbursement for materials and cooking demonstrations) of the CHWs was not consistently paid, and community workers increasingly relied on other sources of income for survival (Figure 2). These broad underlying changes applied to all programme community types.

Figure 2.

Working conditions of community health workers (CHWs) in 2004 and 2011. The figure shows that all programme communities (“original” and “new” combined) experienced increased workload of CHWs between 2004 and 2011, including increases in population pressure (e.g., more children and mothers in the site census), inconsistently paid salary, reliance on other sources of income, and involvement in commune and community health programmes.

In order to understand what could explain the difference in impact between “original” and “new” communities, we present summary statistics of the characteristics of the current CHWs across these two groups in 2011 (Table 4). The summary statistics are presented both as actual means, as well as reweighted means, using the same weights used in the impact estimates. In Table 4, panel A, the unweighted mean socio‐economic characteristics show that CHWs in “original” communities had comparable age, education level, and work motivation as those in the “new” communities. However, after reweighting, “original” CHWs had fewer years of completed education than those in the “new” sites (weighted average of 19% of workers with primary education or less as opposed to 9%), were about 5 years younger on average, and more likely to be heads of households. Controlling for differences in age and education, we cannot detect any difference in skills (as measured by residual vocabulary score) across the two groups, but we detected higher motivation (0.3 SD higher score) among CHWs in “original” communities compared with “new.”

Table 4.

Description for “original” and “new” sites of CHWa characteristics, training, and work load, as well as women'sb participation and perceived nutrition knowledge gain

	Unweighted		Reweighted c
	Original	New	Original	New	Difference
(A) CHW socio‐economic and demographic characteristics
	(N = 133)d	(N = 98)d	(N = 271)d	(N = 268)d
	M (SD)	M (SD)	M (SD)	M (SD)	P valuee
Age (years)	38.14 (8.93)	40.48 (9.24)	35.84 (9.3)	40.85 (8.92)	<0.001
Number of children	3.23 (2.6)	4.25 (2.87)	2.77 (2.47)	4.05 (2.93)	<0.001
Number years as CHW	4.53 (2.28)	3.69 (2.33)	4.04 (2.25)	4.3 (3.13)	0.24
Distance to site (min)	7.74 (15.11)	11.25 (20.14)	8.66 (14.61)	11.71 (21.36)	0.04
Work motivation scale	44.24 (8.68)	43.51 (8.77)	44.76 (8.66)	41.88 (9.26)	<0.001
Vocabulary raw score	67.86 (9.88)	67.94 (8.09)	66.72 (9.33)	70.38 (8.79)	0.01
Vocabulary residualf	−0.12 (9.1)	−1.28 (6.12)	−0.76 (8.51)	−0.74 (5.84)	0.98
	Proportion (n)	Proportion (n)	Proportion (n)	Proportion (n)	P value
Is head of household	0.42 (59)	0.3 (33)	0.4 (120)	0.27 (74)	<0.001
Original resident in the same village	0.6 (84)	0.56 (62)	0.54 (163)	0.58 (161)	0.30
Education level
Primary	0.13 (18)	0.12 (13)	0.19 (57)	0.09 (25)	<0.001
Some secondary	0.56 (79)	0.6 (66)	0.57 (172)	0.64 (177)	0.08
Secondary	0.29 (41)	0.27 (30)	0.23 (71)	0.26 (73)	0.42
Has other revenue	0.64 (90)	0.66 (73)	0.51 (156)	0.65 (181)	<0.001
Speaks official Malagasy	0.37 (50)	0.44 (47)	0.33 (89)	0.47 (127)	<0.001
(B) CHW training, use of materials, and nutrition messages
Training different from before?	0.81 (96)	0.76 (74)	0.83 (201)	0.76 (189)	0.07
More theoretical (than practical)	0.1 (12)	0.05 (5)	0.09 (21)	0.06 (16)	0.36
Use of materials
Health card	0.83 (111)	0.73 (77)	0.83 (227)	0.83 (225)	0.82
Counselling card	0.81 (108)	0.65 (69)	0.82 (223)	0.68 (184)	0.00
Posters	0.71 (95)	0.61 (65)	0.7 (190)	0.59 (160)	0.01
Radio	0.02 (3)	0.05 (5)	0.02 (6)	0.04 (12)	0.15
Messages given
Exclusive breastfeeding	0.56 (75)	0.39 (41)	0.61 (167)	0.43 (117)	<0.001
Hygiene	0.68 (91)	0.69 (73)	0.58 (157)	0.67 (182)	0.03
Infant feeding	0.63 (84)	0.69 (73)	0.68 (184)	0.67 (183)	0.93
Vaccination	0.63 (84)	0.54 (57)	0.57 (154)	0.6 (163)	0.43
Diarrhoea	0.4 (53)	0.28 (30)	0.32 (88)	0.32 (87)	0.93
Complementary foods	0.55 (74)	0.53 (56)	0.54 (146)	0.5 (137)	0.44
Child sickness	0.64 (86)	0.5 (53)	0.59 (161)	0.49 (133)	0.02
Vitamin A	0.49 (65)	0.42 (45)	0.43 (118)	0.45 (122)	0.73
Minimum weight gain	0.37 (49)	0.41 (43)	0.4 (109)	0.51 (140)	0.01
Other	0.27 (36)	0.16 (17)	0.24 (66)	0.11 (30)	<0.001
Difficulty conveying key messages
Exclusive breastfeeding	0.1 (13)	0.07 (7)	0.12 (33)	0.11 (29)	0.59
Hygiene	0.1 (13)	0.14 (15)	0.12 (32)	0.12 (33)	0.90
Infant feeding	0.05 (7)	0.17 (18)	0.07 (20)	0.22 (59)	0.00
Vaccination	0.04 (6)	0.07 (7)	0.04 (10)	0.05 (13)	0.52
Diarrhoea	0.06 (8)	0.02 (2)	0.04 (11)	0.01 (2)	0.01
Complementary foods	0.05 (7)	0.2 (21)	0.13 (34)	0.24 (66)	<0.001
Child sickness	0.13 (18)	0.16 (17)	0.2 (54)	0.23 (62)	0.40
Vitamin A	0.03 (4)	0.03 (3)	0.05 (14)	0.07 (19)	0.37
Minimum weight gain	0.07 (10)	0.09 (10)	0.15 (40)	0.11 (30)	0.20
Other	0.27 (36)	0.16 (17)	0.21 (58)	0.19 (53)	0.60
(C) Women's participation in programme sites and perceived knowledge gained
	(N = 2766)	(N = 2126)	(N = 6006)g	(N = 5379)g
	Proportion (n)	Proportion (n)	Proportion (n)	Proportion (n)	P value
Participated last 3 months	0.35 (960)	0.33 (701)	0.34 (2048)	0.35 (1856)	0.65
Member of household ever participated	0.7 (1929)	0.61 (1305)	0.68 (4081)	0.61 (3299)	<0.001
Ever participated	0.57 (1583)	0.5 (1059)	0.55 (3322)	0.51 (2756)	<0.001
Conditional on ever participation
Attended culinary demonstration	0.59 (329)	0.52 (106)	0.58 (779)	0.57 (246)	0.84
Health knowledge improved
Yes a lot	0.55 (539)	0.46 (151)	0.56 (1270)	0.53 (371)	0.12
Yes a little	0.31 (303)	0.31 (103)	0.28 (626)	0.28 (198)	0.80
Not really	0.13 (122)	0.21 (68)	0.15 (330)	0.16 (115)	0.25
Not at all	0.01 (12)	0.02 (5)	0.02 (39)	0.03 (20)	0.06
Participated in other nutritional programme	0.02 (59)	0.02 (42)	0.02 (104)	0.02 (115)	0.12

Note. CHW: community health worker.

^a Up to three former and current community health workers (CHWs) were interviewed in programme sites. Only data for the most recently employed CHWs (i.e., only one CHW per site) were included when computing the descriptive statistics.

^b A special questionnaire was administered in 2011 to women with children under 5 years of age in current or former participating communities.

^c The mean values for all factors were reweighted to balance the comparison groups, accounting for potential differences in selection of sites into the programme and into different durations of the programme by initial conditions. For reweighting, we used the product of two sets of estimated weights: The first set was based on a propensity score from a model of programme assignment during the first expansion in 2004 as a function of village level socio‐economic characteristics as well as a recent history of agro‐climatic shocks; the second set was modelled on subsequent entry in 2011.

^d The total number of CHWs with non‐missing data varied by characteristic. The minimum number of CHWs included across all characteristics by programme participation type is shown, with the exception of receptive vocabulary knowledge. Vocabulary was assessed in a subset of sites, and information is included for the most recently employed CHWs in 71 “original” and 31 “new” sites. For the reweighted means, the counts for CHWs reflect the expanded dataset after reweighting.

^e P values are from t tests for comparisons of the reweighted means between “original” and “new” sites.

^f A residual score for receptive vocabulary knowledge was estimated post regression after conditioning on CHW age, education, and dialect spoken.

^g The counts for women in the reweighted means reflect the expanded dataset after reweighting.

Table 4, panel B, presents descriptive information about the CHW training and ability to retain and convey key messages included in the intervention. The CHWs who joined post‐2004 reported making less use of support and educational materials (e.g., 0.68 used counselling cards in “new” vs. 0.82 in the “original” communities), although it is unclear whether this was because they did not receive the material or they did not use it. CHWs in “new” communities also reported conveying different messages in their normal activities, with fewer messages in terms of exclusive breastfeeding (0.43 vs. 0.61 in “original” sites) and how to treat child sickness. They also reported more difficulties in conveying messages about feeding infants and the introduction of complementary feeding.

Table 4, panel C, presents summary statistics of participation and women's perception of improved nutrition knowledge for households living in the participating communities across the two groups. CHWs in “new” sites were less able to mobilize participation among eligible women within participating communities (unweighted: 0.61 ever participated in “new” vs. 0.7 in “original”). Conditional on participation, women in “original” communities reported larger improvements in nutrition knowledge from having participated in the programme (unweighted proportions: 0.46 reported that the knowledge improved a lot in “new” sites compared with 0.55 in “original” sites).

4. DISCUSSION

In this unique study, we assessed the long‐term impact of the National Nutrition Programme in Madagascar, which began operation in 1999 and expanded gradually to reach children in over 5,500 communities across the country by 2011. The programme was originally designed to reduce the prevalence of underweight in young children, and our findings suggest that the programme was effective in achieving this goal in the early years of the programme. Programme participation in the early years was also associated with small improvements in linear growth and a reduced prevalence of stunting in 2004, although these outcomes were not the focus of the intervention. After programme scale‐up, the estimated effects of the programme on childhood malnutrition were reduced in the “original” sites. Sites that had closed between 2004 and 2011 demonstrated comparable benefits to those found in the “original” sites in 2004 but had no evidence of impact in 2011. There was no measured impact in the “new” sites after programme scale‐up. Taken together, these findings suggest that programme effects evident in the early phases of the programme faded out after programme expansion. The lack of impact in the second phase of expansion cannot be explained by Madagascar's economic and political crisis, which uniformly hit both “original” and “new” communities, but instead is likely driven by a decrease in programme quality.

The key strengths of this programme evaluation were the large sample size, national representativeness of the samples, and longitudinal nature of the data. Given that we have 14 years of data measured over time at multiple levels, we were able to control for a large set of time varying (and time‐invariant) factors that might affect both programme assignment as well as programme sequential entry over the period of data collection. Although we had insufficient exogenous variation to draw inferences about the effect of the quality of the programme over time, the findings from the descriptive comparison of “original” and “new” sites align with what is known about the changing local context and the importance of the training of frontline community workers. Furthermore, we have complementary qualitative data from interviews with CHWs that we used to assist in interpretation and contextualization of the data. However, there were some limitations. First, the validity of the findings in this study may be limited by the reliance on quasi‐experimental data, which allows for drawing causal inferences only under certain assumptions. Second, the programme was differentially assigned to areas with a higher prevalence of underweight, which requires further assumptions to draw unbiased conclusions.

Despite these limitations, there are many ways that this study adds to the limited evidence on scale‐up of health interventions by CHWs in low‐ and middle‐income countries. One explanation for the decline in programme effectiveness over time relates to CHW working conditions and training. Our examination of the data suggests that the fade out of programme impact was a function of an increased workload (e.g., more children to weigh and greater likelihood of taking additional paid work), as well as differential training of the CHWs. Training for CHWs during the initial programme roll‐out was very “hands‐on” with many refreshers, including opportunities for practice. In the second phase of programme expansion, however, the government used a “training of the trainers” approach due to the larger workforce and reduced funding following the political and economic crisis, possibly limiting the effectiveness of the community teaching and referrals. In addition to these differences in workload and training, qualitative focus group discussions with CHWs and non‐governmental organizations in March–April 2011 found that the selection process of the community workers in the “original” participating communities was much more rigorous. Specifically, the selection process was tightly overseen, with the presence of province and central technical representatives. The more controlled selection and higher quality training of the “original” communities were reflected in the observed higher motivation of the CHWs in the early phase of expansion.

These findings are consistent with the existing literature documenting failed scale‐up efforts within nutrition programmes, which are often due to an insufficiently trained workforce of CHWs (Fanzo et al., 2015; Shrimpton et al., 2016). Employing local workers is a cost‐effective approach to delivering services to improve child health (Nkonki, Tugendhaft, & Hofman, 2017; Sunguya et al., 2013), and programmes that have done so have been effective (e.g., Aboud, Moore, & Akhter, 2008; Bhandari et al., 2004). CHWs cannot be effective, however, if they have not received sufficient training and are not provided with coaching and support. This issue has been identified as a key hindrance to scaling‐up nutrition programmes in the African context in particular (Ellahi, Annan, Sarkar, Amuna, & Jackson, 2015).

In addition to the above, Madagascar's programme had intrinsic weaknesses, which may have contributed to its diminishing effectiveness over time. The programme was designed in the late 1990s based on the best evidence available at the time and was centred on community‐based, monthly growth‐monitoring sessions and nutrition education, with the objective of reducing weight faltering but not length. The evidence on the effectiveness of these approaches in improving child nutrition is mixed. For example, evidence from other studies has demonstrated that the best outcomes for linear growth have involved intensive counselling and strongly emphasized dietary diversity and the promotion and consumption of animal source foods (Dewey & Adu‐Afarwuah, 2008; Imdad, Yakoob, & Bhutta, 2011). Individualized counselling was minimal in the Madagascar programme given the CHW workload in a high population growth and low‐resource environment. In addition, in times of hardship, caregivers may not have been able to act on the advice of community workers to provide higher quality, micronutrient rich foods to their children.

Key elements contributing to the unsuccessful scale‐up of Madagascar's National Nutrition Scale‐up were re‐enforced in a recent paper identifying critical factors for the success of scale‐up (Gillespie, Menon, & Kennedy, 2015). Ideally, there must be a clear vision for what needs to change, an intervention or programme that has been shown to affect the target outcome, and an enabling contextual environment (e.g., household, community, and political) to support change. Furthermore, for successful scale‐up, key drivers should be in place at multiple levels, along with government ownership and incentives embedded in the system. Finally, scale‐up success is achieved when there is a clear strategy for how to expand, along with multilevel capacity (i.e., individual, organizational, and systemic) that can strategically and operationally support scale‐up. Other factors key to scale‐up success include having good governance in place, adequate financial resources, and frameworks for monitoring, evaluation, learning, and accountability.

The National Nutrition Programme in Madagascar has many strengths to build on going forward. People at all levels of the government are intensely dedicated to the success of the programme, as evidenced by their commitment to the programme through periods of instability in governance and funding. Additionally, the established large‐scale infrastructure upon which the programme grew throughout the country is a key foundational strength. Furthermore, the programme relies on a dedicated team of frontline CHWs who are elected by the community and paid, although wages are small. Connections that these frontline workers have with health centres allows for the referral of the most vulnerable children to treatment services as needed. Across the board, the resilience, ownership, and commitment of the community workers were demonstrated by the fact that only a small fraction of communities closed over time, despite a severe period of political and economic crisis.

On the basis of the lessons learned from this study of the long‐term effectiveness of the programme, the Madagascar government has modified their status quo programme, narrowing the age range of the intervention to the first 1,000 days, as well as reducing the workload per CHW. The government subsequently supported a pilot study of programme variants to improve linear growth and child development to assess the cost‐effectiveness of increasing the intensity and the scope of the community‐based nutrition intervention (Fernald et al., 2016). The experimentation has brought formative research and collaboration with domestic and global partners to improve the training and messaging for the community workers. More recently, the government has worked to integrate the health and nutrition sectors, including implementing a 10‐year multiphase programmatic approach (The World Bank, 2017) that begins with the health and nutrition of adolescents and pregnant women, includes nutrition and nurturing care in the first 1,000 days, and will subsequently layer nutrition‐sensitive interventions to address stunting and promote child development at scale (Kerber et al., 2007; Richter et al., 2017).

In conclusion, using longitudinal data from a series of three nationally representative surveys, we show that Madagascar's National Nutrition Programme was effective in reducing malnutrition during early programme scale‐up and that this benefit was sustained at a diminished level during a phase of political instability and severe economic decline. However, we found no evidence of impact in communities that joined during the second phase of programme expansion. Our evidence suggests that the fade out of programme impact was a function of an increased workload for CHWs over time, as well as differential selection and training of the CHWs by programme expansion phase. Our results reinforce the importance of attending to the nature and quality of the intervention when increasing coverage. Although programme scale‐up is necessary to achieve reductions in malnutrition at the population level, programme expansion without an attention to quality can undermine programme effectiveness.

CONFLICTS OF INTEREST

The authors have no conflicts of interest to declare.

CONTRIBUTIONS

AMW, EG and LCHF contributed to the conceptualization, writing, revision, and critical review of the manuscript. AMW and EG carried out the analyses.

Supporting information

Table S1: Absolute value of standardized differences (Bayoumi, 2016) between paired groups of sites, unweighted and reweighted with inverse probability of treatment weights

Table 2. Program impact on primary nutritional outcomes (Panel A for continuous and Panel B for binary indicators)

Figure S1: Flow diagram of ADE surveys and communities by survey year and program participation

Weber AM, Galasso E, Fernald LCH. Perils of scaling up: Effects of expanding a nutrition programme in Madagascar. Matern Child Nutr. 2019;15(S1):e12715 10.1111/mcn.12715