Inclusion and exclusion errors in the targeted supplementary feeding programme of Ethiopia

Abstract

Ethiopia's targeted supplementary feeding (TSF) programme aims to rehabilitate moderately malnourished children and pregnant and lactating women in selected chronically food‐insecure districts. Screening for malnutrition is made by health extension workers through the quarterly community health days (CHD) events based on mid‐upper arm circumference (MUAC) thresholds. This validation study examined the extent of targeting errors of inclusion (providing aid to the nonneedy) and exclusion (failure to reach the needy) in the TSF programme, among preschool children in 6 TSF districts. The study was conducted within 7 days after the completion of the CHD event. Multistage cluster sampling was employed to recruit 1,104 children. Data were collected using interviewer‐administered questionnaire and by reviewing CHD registers. A paired t test was used to compare the MUAC measurements taken during the CHD and during the survey. The study found a global acute malnutrition prevalence of 13.0%. During the CHD, only 54.8% of the children were screened for malnutrition. The overall inclusion and exclusion errors of the TSF were 16.5% and 40.3%, respectively. The reasons for the exclusion errors were low coverage of the screening programme (67.2%) and MUAC measurement errors (32.8%). The mean including standard deviation (M ± SD) of the MUAC measured by health extension workers (11.8 ± 0.9 cm) was significantly lower than the measurements made by fieldworkers in the survey (12.1 ± 1.0 cm; p < .001). The study concluded that high targeting errors are committed in the TSF programme of Ethiopia. Targeting can be enhanced through accurate measurement of MUAC and maximization of the coverage of the screening programme.

Key messages.

• Considerable inclusion and exclusion errors are committed in the targeted supplementary feeding programme of Ethiopia.

• Low coverage of the screening programme is the primary factor for the exclusion errors.

• HEWs commit systematic MUAC measurement errors skewed to overestimation of malnutrition, and thus increase of inclusion errors.

• Targeting can be enhanced through accurate measurement of MUAC and maximization of the community coverage of the screening programme.

1. BACKGROUND

Over the past 15 years, Ethiopia has made remarkable progress in reducing childhood malnutrition (Central Statistical Agency [CSA] and Demographic and Health Surveys [DHS] Program, 2017). Between 2000 and 2016, the magnitude of stunting was substantially reduced from 58% to 38% and prevalence of underweight dropped from 41% to 24% (CSA and DHS Program, 2017; CSA and ORC Macro, 2001). The success is driven by the steady economic growth and the expansion of primary health care attained since the beginning of the new millennium. Between 2000 and 2014, the national gross domestic product increased by an average of 9% per annum, and the potential primary health care coverage was increased from 54% to 94% (National Planning Commission, 2015).

Nonetheless, malnutrition still remains a major problem in Ethiopia. According to a national survey conducted in 2014, 40% and 25% of children were stunted and underweight, respectively, and 9% were wasted (CSA, 2014). Food insecurity, which frequently evolves into serious food crises, continues to threaten the lives of millions of Ethiopians. Erratic rainfall, poverty, land degradation, increasing population pressure, and inefficient farming system remain the key drivers of food insufficiency in the country (Siyoum, Hilhorst, & van Uffelen, 2012). Ethiopia remains one of the largest recipients of food aid in the world, and food assistance has been a vital mechanism by which millions of Ethiopians survive (Mowlds, Nicol, & O'Cleirigh, 2010; Siyoum et al., 2012).

The Ethiopian government and its development partners have been intervening to mitigate the root causes and consequences of food insecurity. The Productive Safety Net Programme (PSNP) aimed at enabling chronically food‐insecure households to resist shocks, build assets, and become food sufficient was launched in 2005, and currently, it has covered up to eight million Ethiopians (Ministry of Agriculture, 2014). Community‐based management of acute malnutrition, which promotes timely detection and treatment of acute malnutrition at the community level, has been rapidly scaled up countrywide since 2008 (United Nations Children's Fund [UNICEF], 2012a). Further more than a million school‐age children are benefiting from the national school feeding programme.

Targeted supplementary feeding (TSF) is a type of selective food assistance that provides nutritious ration to malnourished members of most at‐risk groups of the population, usually including preschool children and pregnant and lactating women (PLW; United Nations High Commissioner for Refugees, 1999). Ethiopia's TSF programme aims to rehabilitate moderately malnourished children and PLW in selected chronically food‐insecure districts (Skau, Belachew, Girma, & Woodruff, 2009). The programme has been implemented through the semiannual/quarterly enhanced outreach service (EOS)/community health days (CHD) events since 2004. The screening for malnutrition takes place at designated EOS/CHD sites based on mid‐upper arm circumference (MUAC) thresholds and identification of nutritional oedema. Acute malnourished individuals are issued fortified blended food and vegetable oil. Further, the EOS/CHD implements vitamin A supplementation (VAS) and deworming of children (Skau et al., 2009).

Food aid interventions frequently commit two errors of targeting—inclusion and exclusion errors—that have serious programmatic implications (Barrett & Maxwell, 2005; Taylor & Seaman, 2004; World Food Programme [WFP], 2016). Inclusion errors (also called leakage) connote provision of aid to the nonneedy, whereas exclusion errors (also called undercoverage) imply failure to reach the needy (Sabates‐Wheeler, Hurrell, & Devereux, 2015). Inclusion errors waste resources and make the programme inefficient, whereas the latter preclude the neediest and make the intervention ineffective (Coady, Grosh, & Hoddinott, 2004). An ideal targeting scheme would enrol all the needy individuals and only the needy ones (Barrett & Maxwell, 2005; Taylor & Seaman, 2004; WFP, 2016). Yet targeting can never be faultless, and a trade‐off between coverage and leakage is common. Therefore, the main interest of food aid programmes should be increasing coverage, by the same token minimizing exclusion errors, with the least amount of inclusion errors (García‐Jaramillo & Miranti, 2015).

Ethiopia's TSF programme is one of the largest supplementary feeding programmes in the world (Skau et al., 2009). In 2016, with the cost of 90 million U.S. dollars, 750,000 beneficiaries had been covered by the scheme (WFP, 2016). Yet the targeting efficiency of the programme has not been explored, and the available limited grey literature signified targeting concerns (Skau et al., 2009; UNICEF, 2012b). Accordingly, this study sought to measure the extent and identify the causes of targeting errors in six TSF‐implementing districts of Ethiopia.

2. METHODS AND MATERIALS

2.1. Study setting and description of the programme

The study was conducted in January 2013 in six chronically food‐insecure districts implementing the TSF programme through the CHD events. The districts were Alamata and Ahferom from the Tigray region, Antsokia Gemza from the Amhara region, Dolo Mena and Gemechis from the Oromia region, and Wonago from the Southern Nations, Nationalities, and Peoples Region. Projections based on the 2007 national census indicated that in 2013, the districts had a combined population exceeding 850,000, of whom 91% were rural residents.

The EOS/CHD is a prominent child survival strategy in Ethiopia. The EOS was launched in 2004 with the support of UNICEF as an interim approach for reducing child mortality, while the government was rolling out the Health Extension Programme (HEP). Semiannually, VAS and deworming of children, as well as screening of children and PLW for malnutrition, are carried out via centrally organized EOS campaigns. Starting from 2008, in the four regions, the EOS has gradually transited to the CHD. The CHD is a less campaign‐intensive approach whereby health extension workers (HEWs) deployed at the village level organize semiannual VAS and deworming of children and quarterly screening for malnutrition at health posts and outreach sites. Starting from 2011, the government is also undergoing a gradual and complete integration of EOS/CHD activities into the routine HEP. However, at the time of the survey, all the study districts were implementing the child survival activities via CHD modality.

In selected chronically food‐insecure districts, malnourished children and PLW screened during the EOS/CHD are referred to the TSF programme. The TSF is managed by the disaster risk management sector of the government, and it is supported by the WFP. Individual children identified with malnutrition are issued two rounds of three‐monthly ration, each comprising 25 kg of micronutrient‐fortified corn/wheat soya blend and 3 L of oil. Further, the severely malnourished cases are referred to the community‐based management of acute malnutrition programme.

2.2. Study design

This descriptive study emanates from a larger survey that we conducted in 2013, in 20 districts of the Tigray, Amhara, Oromia, and the Southern Nations, Nationalities, and Peoples Region of Ethiopia. However, this paper is developed based on the data collected from six TSF‐implementing districts. The original study was commissioned by UNICEF and was intended to validate the coverage of CHD activities (VAS, deworming, screening for malnutrition) in the regions. The survey was conducted within 7 days after the end of the CHD events. In each site, administrative data recorded by HEWs during the CHD event were reviewed and compared with the survey findings.

2.3. Sample size and sampling technique

We analysed the data of 1,104 children aged 6–59 months from the six TSF‐implementing districts. As the information was extracted from an existing dataset, prior sample size determination has not been made.

In the original survey, the participants were identified using multistage cluster sampling technique. In 2013, there were 125 TSF‐implementing districts in the aforementioned four regions. Among these, the six districts were selected at random. On the basis of the list of enumeration areas (EAs) acquired from the CSA of Ethiopia, four EAs were selected from each district (total of 24 EAs across the six districts) using a simple random sampling technique. In each of the selected EAs, exhaustive listing of eligible children was made, and the information was used as a sampling frame for the study. Ultimately, from each EA, 46 children were selected via the simple random sampling technique. When nonresponses were encountered, fieldworkers made direct replacements of children from the nearest household.

2.4. Data collection procedure

The survey data were collected from mothers or caregivers of the index children through door‐to‐door interviews. Data were collected by 24 trained enumerators using structured and pretested questionnaires prepared in three local languages (Tigrigna, Amharic, and Afaan Oromo). In the Wonago district, the Amharic version of the questionnaire was presented to the respondents in the local Gedio language by the data collectors. Eight supervisors and four regional coordinators oversaw the fieldwork.

MUAC was measured to the nearest millimetre using nonstretch MUAC tapes. The measurements were made at the midpoint of the left upper arm, without any clothing, with proper tape tension and positioning of the arm. The MUAC was taken in duplicates by the same observer, and if the first two differed by more than 0.3 cm, another pair of measurements was required.

The presence of nutritional oedema was determined following standard procedure. Normal thumb pressure was applied to both feet of the child for about 3 seconds, and if a dent persists, then the child was considered as having nutritional oedema.

The nutritional status of the children was classified according to the national TSF programme guideline (Disaster Prevention and Preparedness Agency [DPPA] & WFP, 2007). Children whose MUAC exceeded 12 cm and had no bilateral leg oedema were considered as nonmalnourished. Moderate malnutrition (MUAC of 11.0–11.9 cm without oedema) and severe malnutrition (MUAC below 11.0 cm and/or bilateral oedema) were also classified as per the protocol.

In each site, the field team reviewed the CHD register of HEWs and extracted information about the children included in the survey, whenever available. Information extracted included MUAC measurement taken by HEWs during the CHD event and the decision made regarding the eligibility of the children for food ration.

During the fieldwork, intensive on‐site supervision was made by eight supervisors and four regional coordinators. Further, the supervisors validated 10% of the filled questionnaires by reinterviewing random subsamples of the study participants. Further, the completeness and coherence of all questionnaires were checked on a daily basis.

2.5. Data analysis

We used SPSS 20.0 software for data entry and analysis. Proportions, as well as appropriate measures of central tendency and dispersion, were used to describe the data. MUAC measurements taken by HEWs during the CHD event were compared with the corresponding survey measurements using paired t‐test analysis. Differences were considered statistically significant at p values less than .05.

During the analysis, all children whose MUAC fell below 12 cm and/or had bilateral leg oedema were considered eligible for TSF ration (DPPA & WFP, 2007). On the other hand, children who were issued a TSF ration card or enlisted in the CHD register as TSF beneficiaries were considered as TSF recipients. Ultimately, as described in Table 1, four indices of targeting were computed. Inclusion error was calculated as the proportion of children who received the ration despite not being eligible. Exclusion error was determined as proportion of needy children who were omitted from the ration. Successful targeting coverage was calculated as the percentage of the eligible children that actually received the food aid (Table 1).

Table 1.

Summary of key indicators used for evaluating the targeting efficiency of targeted supplementary feeding programme

Indicators	Description	Calculation
Inclusion error (%)	Proportion of children who received the ration despite not being eligible	Number of children who received the ration without being eligible, divided by the total number of beneficiary children
Exclusion error (%)	Proportion of children in need who are omitted from the food aid programme	Number of children who are omitted from the scheme despite being eligible, divided by the total number eligible beneficiary children
Successful targeting (coverage; %)	The proportion of eligible children who actually received the ration	Number of eligible children who received the ration, divided by the total number eligible children for the ration
Successful exclusion (%)	The proportion of the ineligible children who actually did not receive the ration	Number of ineligible children who did not receive the ration, divided by the total number ineligible children for the ration

2.6. Ethical issues

Ethical clearance was obtained from the Institutional Review Board of the College of Natural and Computational Sciences, Addis Ababa University. For each respondent, a statement explaining the purpose of study was read, and accordingly, informed consent was secured. Eligible children who were found to be excluded from the TSF scheme were linked with the local HEWs so that they can get rescreened for eligibility.

3. RESULTS

3.1. Sociodemographic characteristics

The survey included 1,104 children aged 6–59 months—184 from each of the six districts. In most (97.3%) of the cases, data were collected from the parents of the index children, whereas in the remaining 2.6%, information was availed from other primary caregivers. Nearly half of the respondents (45.3%) were between 25 and 34 years of age, and the majority (80.6%) had no formal education. The vast majority of the respondents (94.1%) were married. The mean (±SD) household size was 5.9 (±2.0) and ranged from 2 to 13. The mean (±SD) age of the children was 31.8 (±14.7) months, and the boys‐to‐girls ratio was 1.11 (Table 2).

Table 2.

Sociodemographic characteristics of the study participants, Ethiopia, 2013

Variables	Frequency	Percentage
Survey among children 6–59 months (n = 1,104)
Type of respondent
Mother	1,047	94.8
Father	28	2.5
Other primary caregiver	29	2.6
Respondent's age (year)
15–24	267	24.2
25–34	500	45.3
35+	337	30.5
Educational status
No formal education	890	80.6
Primary education (Grades 1–8)	196	17.7
Secondary or higher education	18	1.6
Marital status
Married/living together	1,039	94.1
Others	65	5.9
Child's age (months)
6–11	101	9.1
12–23	247	22.4
24–35	265	24.0
36–47	238	21.6
48–59	253	22.9
Sex of the child
Male	580	52.5
Female	524	47.5

3.2. Nutritional status of the children

The mean (±SD) of the MUAC of the children was 13.9 (±1.2) cm. The majority were in the MUAC category of more than 12.0 cm, indicative of normal nutritional status, whereas 10.5% and 1.2% were moderately (MUAC between 11.0 and 11.9 cm) and severely (MUAC less than 11.0 cm) malnourished, respectively. The oedema examination suggested that 2.1% of the total children had bilateral pitting oedema. Nutritional status classification based on MUAC and occurrence of oedema showed that 13.0% of the children were severely or moderately malnourished (Table 3).

Table 3.

Nutritional status of children in six chronically food‐insecure districts of Ethiopia, 2013

Variables	Frequency	Percentage
Nutritional status of children (n = 1104)
Bilateral pitting oedema
Yes	23	2.1
No	1,081	97.9
Mid‐upper arm circumference (cm)
>12.0	973	88.3
11.0–11.9	116	10.5
<11.0	13	1.2
Nutritional status
Severely malnourished	47	4.3
Moderately malnourished	97	8.8
Normal	960	87.0

3.3. Coverage of screening for malnutrition during the CHD round

Among 1,104 children included in the survey, 605 (54.8%) were screened for malnutrition during the recent CHD event. The leading reported reasons for not being screened were as follows: The health workers did not assess MUAC or oedema despite the child being brought to the CHD site (38.5%), and the caregiver was not aware of the screening programme (34.1%). Other less frequently cited reasons were domestic workload of the mother (9.2%) and illness of the child (3.2%).

3.4. Inclusion and exclusion errors of TSF

Table 4 shows the extent of inclusion and exclusion errors of the TSF programme. Among 1,104 children included in the survey, 103 (9.3%) had received a TSF ration after being screened by HEWs during the recent CHD round. Among the 103 children who were considered to be eligible for the TSF ration, 17 were found to be nonmalnourished in the survey; accordingly, the inclusion error was 16.5%. Of the 144 malnourished children identified in the survey, less than three fourths (72.9%) were screened in the recent CHD round. Further, only 86 (59.7%) were recorded as eligible to receive the TSF ration. Accordingly, the inclusion error was 40.3%. The reasons for the large exclusion error were failure to screen malnourished children during the CHD (67.2%) and potential MUAC measurement errors (32.8%; Table 4).

Table 4.

Inclusion and exclusion errors and successful targeting and exclusion in targeted supplementary feeding programme among children aged 6–59 months in six chronically food‐insecure districts of Ethiopia, 2013

Inclusion of the child in the SFP	Eligibility of the child
	Eligible	Not eligible	Total
Included	86	17	103
Not included	58	943	1,001
Total	144	960	1,104

Note. Inclusion error (%) = (17/103)100% = 16.5%. Exclusion error (%) = (58/144)100% = 40.4%. Successful targeting (coverage; %) = (86/144)100% = 59.6%. Successful exclusion (%) = (943/960)100% = 98.2%. SFP = supplementary feeding programmes.

3.5. Comparison of MUAC taken during the CHD screening and the validation survey

Comparison of MUAC measurements taken during the CHD screening and in the validation survey was made based on the data of 85 children. For the remaining children, either the MUAC measurements have not been made by HEWs or the measurements were given in intervals. The mean together with standard deviation (M ± SD) of the MUAC measured by HEWs (11.8 ± 0.9 cm) was found to be significantly lower than the measurement made during the validation survey (12.1 ± 1.0 cm; t = −4.754; p < .001).

4. DISCUSSION

A vital aspect of successful targeting of food assistance is reduction of exclusion and inclusion errors. This study shows targeting errors are often committed in the TSF programme of Ethiopia. Further, the findings suggest low community coverage of screening for malnutrition and MUAC measurement errors have also led to high exclusion errors. Conversely, systematic errors of MUAC measurement skewed to overestimation of malnutrition might have caused inclusion errors.

Previous studies suggested that even for minimally trained health workers, reliability of MUAC measurement is convincingly high (Ayele et al., 2012; Mwangome, Fegan, Mbunya, Prentice, & Berkley, 2012). However, in our study, MUAC measurements by HEWs were found to be significantly different and lower than the one made by the fieldworkers. The divergence is unlikely to emanate from random measurement errors. Random errors usually bring about balanced overestimation and underestimation; and thus significant departure from the actual mean value is not anticipated. Rather, intentional or systematic errors might explain the mismatch. For instance, health workers sometimes may decide to enrol marginally normal children into the TSF with the notion of avoiding imminent growth faltering; or few front‐line health workers might dishonestly enrol the nonneedy to illegitimately benefit the household or the community. A previous study in Ethiopia witnessed that local officials involved in the selection of food aid beneficiaries are usually pressurized by the public to recruit more recipients from the locality (Sharp, Brown, & Teshome, 2016).

Theoretically, the mismatch between MUAC measurements made by HEWs and those made by the fieldworkers can also be due to errors committed during the survey. However, we assumed this is very improbable because, unlike HEWs who usually make one random measurement, the fieldworkers made the measurements in duplicate strictly following standard procedures.

The efficiency of food aid targeting in Ethiopia has not been extensively explored before. However, the available studies have suggested targeting concerns (Clay, Molla, & Habtewold, 1999; Sharp et al., 2016; Uraguchi, 2011). A national study conducted in the late 1990s surprisingly found no significant association between household food insecurity status and direct food aid recipients, indicative of serious targeting problems (Clay et al., 1999). An evaluation of the PSNP witnessed large rates of inclusion and exclusions errors due to a variety of reasons including an unmanageably high number of eligible people for the aid and inconsistent interpretation of the enrolment criteria by the local officials (Sharp et al., 2016). Another evaluation study documented high percentages of undercoverage (about 25%) and leakage (about 30%) in PSNP and food‐for‐work programmes of Ethiopia (Uraguchi, 2011).

In food assistance programmes, inclusion errors are more acceptable than exclusion errors (WFP, 2016). Yet, in the districts, the TSF has failed to reach 40% of the needy children, implying gross undercoverage problems. The majority of the exclusion errors were committed due to the low coverage of the nutritional screening activity during the CHD. This is supported by the finding that only 55% of children were screened for TSF eligibility during the event. Rigorous promotion of the CHD event, intensive engagement of volunteers, and making the screening sites more accessible to the community may help reduce exclusion errors. Especially, as transition is made from the EOS to CHD and routine HEP, the health system must preserve implementation intensities and keep service coverage high.

The typical strength of the current study is that it complemented data from community‐based survey and administrative records and managed to estimate both errors of targeting. Theoretically, inclusion error can be determined based on good‐quality administrative records of the beneficiaries. However, any attempt to estimate exclusion error requires knowledge about the number of the needy in the population—which is unlikely to be readily available without community‐based studies. The other strong point is that we have successfully completed the validation survey within a week of the CHD event; accordingly, changes in MUAC measurements and recall errors are unlikely to explain the findings of the study.

However, there were some limitations and possible biases to this study. First, though more than a thousand children were included in the analysis, targeting errors were actually determined based on smaller numbers of beneficiaries and needy children. This has undoubtedly affected the precision of the estimates. Second, the study was conducted based on data gathered in 2013; hence, it may not exactly represent the contemporary situation of the programme. Especially, as the programme had been undergoing active transition, targeting errors are also likely to be in dynamic conditions. Further, alterations that had been introduced recently to the TSF protocol (such as changes in threshold values for MUAC and exclusion of severely malnourished children from the TSF programme) might also affect the extent of the errors. Third, though redistribution and sharing of food aid within households is technically considered as an exclusion error (WFP, 2016), this dimension has not been explored in the study. Fourth, as the district health officials and the HEWs were aware of the survey, MUAC measurements might have been conducted more carefully than usual; accordingly, the Hawthorne effect cannot be entirely excluded, and one would expect underestimation in the errors of targeting. Finally, it should be noted that due to possible socio‐economic disparities, the findings of the study may not be entirely extrapolated to the national level.

5. CONCLUSION

The purpose of the study was to examine the extent of targeting errors of inclusion and exclusion in the TSF programme implemented in six districts of Ethiopia. The study found 17% and 40% inclusion and exclusion errors, respectively. The study suggested that systematic error in MUAC measurement skewed to overestimation of malnutrition might have inflated the inclusion errors. Conversely, the low coverage of the screening and possible measurement errors has contributed to the exclusion errors. The degree to which the TSF reaches the intended beneficiaries can be improved through more careful and accurate measurement of the MUAC and maximizing coverage of screening for malnutrition. Further studies are needed to find out why families are not more proactive in assuring that their children participate in the CHD.

CONFLICTS OF INTEREST

The authors declare that they have no conflicts of interest.

CONTRIBUTIONS

SG, TB, and NR designed the study and collected, analysed, and interpreted the data. SG drafted the paper. All the authors critically reviewed the manuscript for intellectual content and approved the final version.

Gebremedhin S, Bekele T, Retta N. Inclusion and exclusion errors in the targeted supplementary feeding programme of Ethiopia. Matern Child Nutr. 2018;14:e12627 10.1111/mcn.12627