Impact of a Malaria-Control Project in Benin That Included the Integrated Management of Childhood Illness Strategy

Abstract

Objectives. To estimate the impact of the Integrated Management of Childhood Illness (IMCI) strategy on early-childhood mortality, we evaluated a malaria-control project in Benin that implemented IMCI and promoted insecticide-treated nets (ITNs).

Methods. We conducted a before-and-after intervention study that included a nonrandomized comparison group. We used the preceding birth technique to measure early-childhood mortality (risk of dying before age 30 months), and we used health facility surveys and household surveys to measure process indicators.

Results. Most process indicators improved in the area covered by the intervention. Notably, because ITNs were also promoted in the comparison area children's ITN use increased by about 20 percentage points in both areas. Regarding early-childhood mortality, the trend from baseline (1999–2001) to follow-up (2002–2004) for the intervention area (13.0% decrease; P < .001) was 14.1% (P < .001) lower than was the trend for the comparison area (1.3% increase; P = .46).

Conclusions. Mortality decreased in the intervention area after IMCI and ITN promotion. ITN use increased similarly in both study areas, so the mortality impact of ITNs in the 2 areas might have canceled each other out. Thus, the mortality reduction could have been primarily attributable to IMCI's effect on health care quality and care-seeking.

In sub-Saharan Africa, malaria is a leading cause of child mortality.^1,2 These deaths can be prevented with insecticide-treated nets (ITNs), indoor insecticide spraying, prompt and effective treatment of malaria cases, and intermittent preventive treatment of malaria in pregnant women.²

To improve children's malaria treatment, the Roll Back Malaria partnership and the World Health Organization (WHO) recommend the Integrated Management of Childhood Illness (IMCI) strategy.^3–5 IMCI, which was developed by WHO and other partners, aims to prevent mortality from all leading causes of child deaths (e.g., pneumonia, diarrhea, and malaria). IMCI has 3 components: (1) improving case-management quality (especially in health facilities) by training health workers to use evidence-based clinical guidelines, (2) strengthening health systems, and (3) promoting community and family health practices.

IMCI has been introduced in more than 100 developing countries,⁶ and studies have demonstrated that it can improve health care quality at health facilities.^7–10 With regard to its effect on mortality, however, the evidence is mixed. IMCI seems to have lowered child mortality in Tanzania,¹¹ but studies in Brazil¹² and Bangladesh⁸ did not find a statistically significant reduction. Thus, despite IMCI being one of the world's most widely implemented child health strategies, its impact on mortality remains unclear.

In 1998, before today's billion-dollar malaria initiatives, the US Agency for International Development launched a subnational malaria demonstration project in Benin that included IMCI. Benin is a low-income country in West Africa with extreme poverty,¹³ endemic malaria, and high mortality for children younger than 5 years (160 deaths per 1000 live births when the project began¹⁴). We conducted this study to evaluate the project's impact on early-childhood mortality, with an emphasis on IMCI's impact.

METHODS

We conducted a before-and-after intervention study with a nonrandomized comparison group (i.e., a plausibility design¹⁵). Mortality and exposure to interventions were not measured at the individual level, so the evaluation had 2 parts: (1) to determine whether mortality declined, and (2) to use process indicators to determine whether mortality declines could be plausibly linked to the intervention. Figure 1 presents the logic model for the second part of the evaluation. It illustrates indicators for showing that project activities could have caused health improvements and for exploring alternative explanations for mortality changes (i.e., confounders of the intervention–mortality association). Because ITN promotion and some IMCI implementation occurred in the comparison area, the figure represents this implementation as contamination.

FIGURE 1.

Logic model for the evaluation of a malaria-control project: Benin, 1999–2006.

Note. IMCI = Integrated Management of Childhood Illness strategy; ITN = insecticide-treated net.

^aHealth-worker supports included supervision, job aids, and nonfinancial incentives. Conceptually, such supports are a facet of the health-systems component of IMCI.

^bConceptually, a facet of the community component of IMCI.

^cIncludes antimalarial drug resistance, births at health facilities, birth spacing, breastfeeding, child vaccinations, HIV prevalence, insecticide resistance, improved water sources, rainfall (a climatic determinant of disease transmission), socioeconomic status (electricity in household, education), tetanus vaccination for women, and vitamin A supplementation.

Setting

The intervention area (Ouémé and Plateau departments; Figure A available as a supplement to the online version of this article at http://www.ajph.org) and comparison area (Zou and Collines departments) had similar populations (approximately 1 million in 1999), numbers of communes (i.e., districts; 16 and 15, respectively), and numbers of public health facilities (about 100 each).¹⁶ Both areas had widespread poverty, weak infrastructure, and comparable climates. Each area included 1 medium-sized city. Mosquito resistance to pyrethroids, the class of insecticide used in ITNs, was detected in both areas.¹⁷ Regarding resistance to chloroquine (the most commonly used antimalarial drug at the time), studies throughout southern Benin revealed that failure rates were increasing, from a range of 14.3% to 23.8% (1998–2001) to a range of 71.8% to 87.5% (2003–2005).^18,19 Reliable data on the number of health workers per population and health facility accessibility were not available.

Interventions

The project's funding was modest (about $450 000/year), and no policy on intermittent preventive treatment of malaria in pregnant women existed at the time. Therefore, in the intervention area the project focused on 2 strategies: implementation of IMCI to improve management of malaria and other important childhood illnesses at outpatient health facilities, and activities to increase ITN use among children and pregnant women (Figure 2). Although ITN promotion is part of IMCI, conceptually speaking (i.e., its community component), in this evaluation IMCI is construed as its health-facility component and, to a lesser extent, its health-system-strengthening component.

The intervention area was the pilot site for IMCI in Benin. From 2001 through 2004, the project provided WHO's 11-day IMCI training to 253 health workers (mostly nurses; Table 1). These workers also received standard WHO supports (e.g., flip chart of IMCI guidelines) and other supports developed by project staff (e.g., IMCI-specific supervision and job aids⁹).

TABLE 1.

Results of Implementation Activities in the Intervention Area of a Malaria-Control Project: Benin, 1999–2006

	Year,a No. (%) or No./Total No. (%)
Indicator of Implementation Activity	1999	2000	2001	2002	2003	2004	2005
IMCI in health facilities
Cumulative no. of IMCI-trained health workers	0	0	118	157	229	253	253
Ill child brought for initial consultation was seen by IMCI-trained health workerb	0/430 (0)	…c	123/313 (39.3)	…	…	188/298 (63.1)	…
Child with a potentially life-threatening illness received adequate treatmentd (analysis of all children, regardless of whether the health worker had IMCI training)b	95/372 (25.5)	…	88/274 (32.1)	…	…	123/277 (44.4)	…
Adequate treatment among children seen by IMCI-trained health workersb	…	…	54/108 (50.0)	…	…	92/175 (52.6)	…
Adequate treatment among children seen by health workers without IMCI trainingb	95/372 (25.5)	…	34/166 (20.5)	…	…	31/102 (30.4)	…
Child with a severee potentially life-threatening illness received adequate treatmentb	20/69 (29.0)	…	17/50 (34.0)	…	…	30/53 (56.6)	…
Health worker received at least 1 supervision visit in the past 6 mo	64/109 (58.7)	…	39/134 (29.1)	…	…	84/130 (64.6)	…
Facility had oral antimicrobials in stock (chloroquine, cotrimoxazole)	80/85 (94.1)	…	91/98 (92.9)	…	…	88/98 (89.8)	…
Facility had all key drugs in stock (chloroquine, cotrimoxazole, ORS, injectable quinine and ampicillin)	32/85 (37.7)	…	50/98 (51.0)	…	…	51/98 (52.0)	…
ITN promotion
Communes with health facility–based ITN promotion (% of all 16 communes)	0 (0)	0 (0)	0 (0)	5 (31.3)	11 (68.8)	16 (100)	16 (100)
Cumulative no. of Members of women's groups trained to promote ITNs	0	0	0	0	334	575	575
Cumulative no. of commune-level community-mobilization eventsf	0	32	64	76	84	100	116

Note. IMCI = Integrated Management of Childhood Illness strategy; ITN = insecticide-treated net; ORS = oral rehydration solution. Ellipses indicate data not available.

^aThe baseline was approximately early 1999 to early 2001, and the follow-up period was approximately early 2002 to early 2004. IMCI scale-up began in mid-2001, after the baseline period; the 2001 survey was conducted in the second half of 2001, also after the baseline.

^bSource is health-facility surveys.

^cNot available; health-facility surveys representing the entire intervention area were only conducted in 1999 (before IMCI), 2001, and 2004.

^dPotentially life-threatening illnesses were any of the following IMCI-defined disease classifications: malaria, very severe febrile illness (e.g., severe malaria or sepsis), pneumonia, severe pneumonia, diarrhea (with or without dehydration), dysentery, anemia, severe anemia, severe malnutrition, measles, measles with complications, severe complicated measles, or mastoiditis.

^eAny respiratory infection with respiratory distress; any febrile illness with neck stiffness; and any illness with convulsions, inability to drink, lethargy, unconsciousness, severe malnutrition, or severe pallor.

^fA commune-level event included activities in several (but not all) villages in a given commune for a given 2-week round of community mobilization.

ITN distribution and promotion in the intervention area occurred through community and health-facility activities (Box A available as a supplement to the online version of this article at http://www.ajph.org). By 2004, all 16 communes had facility-based ITN activities, and 575 members of women's groups had been trained to promote ITNs. Both the case-management intervention and the ITN intervention were supported by community-mobilization activities. From 2000 through 2005, 116 commune-level community-mobilization events occurred. These included activities in several (but not all) villages in a given commune for a given 2-week round of community mobilization.

In the comparison area, UNICEF supported several child-health activities,²⁰ including IMCI training for 56 health workers and ITN promotion (Box B available as a supplement to the online version of this article at http://www.ajph.org).

Three ITN-promotion activities were implemented in both areas: health-education talks to groups of mothers, individual counseling (as per Benin's adaptation of IMCI guidelines²¹), and campaigns for re-treatment of ITNs (Supplemental Boxes A and B).

Several other programmatic characteristics were common to both areas. First, ITNs needed re-treatment annually because long-lasting ITNs were unavailable. Second, ITNs were often unavailable, especially before 2002 and from 2005 through 2006. Third, in practice chloroquine was the first-line antimalarial drug. A policy for artemisinin-based combination therapy was adopted in 2004, but these medicines were scarce during the study period. Fourth, routine preventive and curative services of variable quality and coverage were provided, and several vaccination and vitamin A supplementation campaigns were undertaken.

Data Sources

We analyzed administrative records (both study areas), health-facility surveys (intervention area only), mortality monitoring (both study areas), and household surveys (both study areas; Figure 2).

FIGURE 2.

Time line of implementation and evaluation activities of a malaria-control project: Benin, 1999–2006.

Note. IMCI = Integrated Management of Childhood Illness strategy; ITN = insecticide-treated net.

^aScale-up increases progressively over time; arrows indicate that support for activities continued after initial scale-up.

^bThe Demographic and Health Surveys are national household surveys not conducted by project staff. Results of the Demographic and Health Surveys and the mortality monitoring were available for intervention and comparison areas.

^cResults of the preceding birth technique in the study area reflected the early-childhood mortality risk centered at a time about 2 years in the past, indicated by the diagonal arrow; dashed vertical lines indicate the approximate time to which each 12-month block of data applies.

To measure health care quality for ill children in the intervention area, we conducted health-facility surveys in 1999 (before provision of IMCI training), 2001 (shortly after IMCI training began), and 2004 (the year during which implementation of IMCI in the intervention area was essentially complete). Surveys included a probability sample of outpatient facilities and patients, with cluster sampling. We observed consultations, interviewed caretakers, reexamined children, interviewed health workers, and assessed health-facility characteristics.⁹

To assess mortality trends, we used the preceding birth technique (PBT).^22,23 When women came to maternity units to deliver, trained birth attendants asked 2 questions: (1) Did the woman have a previous live-born child? (2) If yes, was the last live-born child alive today? The proportion of last live-born children who die by the time of a current delivery is the index of early-childhood mortality (IECM). The IECM has been shown in many settings to closely estimate the probability that live-born children will die before the age of 0.8 times the mean interbirth interval.²² This result refers to the mortality risk at a time in the past centered at about two thirds of the mean interbirth interval.²³ With a mean interbirth interval of about 38 months in both study areas, the IECM estimated the probability that a live-born child would die by about age 30 months, and this risk applied to a time centered about 2 years in the past (ranging from about 9 months to 5 years in the past²²; Figure B available as a supplement to the online version of this article at http://www.ajph.org).

Although we characterize the IECM as the probability of dying by age 30 months, the key point is that the mean interbirth interval was essentially constant during the study period (Table 2). Thus, interpretation of the IECM is likely to be the same during the study period regardless of whether the IECM truly estimated the risk of dying before age 30 months or 29 months or some other exact age.

TABLE 2.

Trends in Determinants of Child Mortality in Intervention and Comparison Areas of a Malaria-Control Project: Benin, 1999–2006

	Intervention Area			Comparison Area			Percentage-Point Difference Between Change in Intervention Area and Change in Comparison Area
Indicator	Baseline	Follow-Up	Change, Follow-Up Minus Baseline	Baseline	Follow-Up	Change, Follow-Up Minus Baseline
Determinants in causal pathway with trends favoring lower mortality in intervention area
Child aged < 5 y with diarrhea treated with oral rehydration solution or home-made sugar/salt solution,a %	29.4	43.7	14.3	44.8	33.1	−11.7	26.0
Child aged < 5 y with acute respiratory infection treated at a health facility or by a health worker,a %	23.4	46.9	23.5	28.7	36.8	8.1	15.4
Child aged < 5 y with fever treated at a health facility or by a health worker,a %	28.9	43.1	14.2	32.0	38.3	6.3	7.9
Child aged < 5 y with diarrhea treated at a health facility or by a health worker,a %	13.9	21.6	7.7	23.7	25.3	1.6	6.1
Child aged < 5 y slept under an insecticide-treated net the night before the survey,a %	3.7	27.4	23.7	5.0	24.9	19.9	3.8
Determinants in the causal pathway with trends favoring lower mortality in comparison area
Child aged < 5 y with fever treated with an antimalarial,a,b %	72.9	62.7	−10.2	72.7	64.2	−8.5	−1.7
Determinants not in the causal pathway (confoundersc) with trends favoring lower mortality in intervention area
Household has improved water source,a %	71.0	71.5	0.5	77.0	74.0	−3.0	3.5
Woman with a live birth who had ≥ 2 tetanus vaccinations,a %	40.4	48.9	8.5	57.4	64.2	6.8	1.7
Determinants not in the causal pathway (confoundersc) with trends favoring lower mortality in comparison area
Child aged 6–59 mo received vitamin A supplementation in the past 6 mo,a %	20.1	50.3	30.2	10.3	69.0	58.7	−28.5
Mean annual rainfall,d mm	1072	1297	225 (21.0% increase)	1142	1185	43 (3.8% increase)	17.2
Child aged 6–11 y attends primary school,a %	61.5	68.1	6.6	51.5	71.0	19.5	−12.9
Child aged 12–23 mo vaccinated against measles, by card or mother's history,a %	75.0	54.2	−20.8	76.2	65.7	−10.5	−10.3
Child aged 12–23 mo fully vaccinated,e by card or mother's history,a %	68.0	42.7	−25.3	68.4	51.3	−17.1	−8.2
Male aged ≥ 6 y ever attended school,a %	68.7	70.5	1.8	56.1	65.5	9.4	−7.6
Female aged ≥ 6 y ever attended school,a %	41.5	44.1	2.6	32.0	41.7	9.7	−7.1
Mean interbirth interval during the past 5 y,a mo	38.8	39.5	0.7 (1.8% increase)	36.6	39.0	2.4 (6.6% increase)	–4.8
Household has electricity,a %	28.6	32.2	3.6	11.4	18.7	7.3	−3.7
Child is breastfed,a %	96.3	94.0	−2.3	96.4	97.1	0.5	−2.8
Birth occurs in health facilities,a %	93.1	93.1	0	87.7	90.4	2.7	−2.7
Child aged 12–18 y attends secondary school,a %	15.4	36.5	21.1	11.6	33.5	21.9	−0.8

Note. IMCI = Integrated Management of Childhood Illness strategy.

^aBaseline value measured by the 2001 Benin Demographic and Health Survey¹⁴ and follow-up value measured by the 2006 Benin Demographic and Health Survey.²⁴

^bIn 2001, the antimalarial drugs were chloroquine, sulfadoxine-pyrimethamine, or amodiaquine. In 2006, the antimalarials were chloroquine, sulfadoxine-pyrimethamine, amodiaquine, quinine, artemether-lumefantrine, halofantrine, or artesunate monotherapy. Chloroquine was used for the large majority of all treatments.

^cPotential confounders of the association between project activities and reduced child mortality.

^dOn the basis of data from Benin's weather stations (F.O., unpublished data, 2007). Rainfall data collected in Cotonou (slightly west of the southern edge of the intervention area) were the proxy for the intervention area, and the average of data from Bohicon (Zou department) and Savè (Collines department) were the proxy for the comparison area. Baseline values are means for 1999–2001, and follow-up values are means for 2002–2005.

^eIncludes bacillus Calmette-Guérin vaccine, measles vaccine, 3 doses of diphtheria-pertussis-tetanus vaccine, and 3 doses of polio vaccine.

We added the 2 PBT questions to the standard partogram for tracking the progress of labor. We also provided partogram stocks and trained birth attendants to ask the questions in local languages. Ministry of Health supervisors and project staff visited the maternity units every 1 to 3 months to collect the PBT data, assess the quality of data collection, and provide feedback and support to help maintain high-quality data collection. The PBT system included 119 maternities in the intervention area and 127 in the comparison area, which represented virtually all public and registered private maternities in the study area. The system for collecting PBT data operated from April 2001 to March 2006.

For community-level indicators of intervention coverage, health, and socioeconomic status, we used results from Demographic and Health Surveys (DHSs) conducted in 2001 and 2006. DHSs are household cluster surveys.^14,24 Although national in scope, the DHSs were designed to provide department-level results in Benin. Both surveys were conducted from August to November, which covered the short rainy season. Notably, we could not use DHSs to evaluate mortality changes because their precision is low at the department level for the short time intervals needed to assess project impact.

Analysis

We used SAS version 9.1 (SAS Institute Inc, Cary, NC) to analyze health-facility surveys and PBT mortality data. For hypothesis testing and confidence-interval (CI) estimation, α equaled 0.05.

For the health-facility surveys, analyses were restricted to initial consultations for children aged 1 week to 59 months (the ages covered by IMCI clinical guidelines). The 2 primary indicators of health care quality were the proportion of children with potentially life-threatening illnesses (e.g., IMCI-defined malaria, pneumonia) who received adequate treatment and the same proportion for the subgroup of children with severe illnesses.⁹ Adequate treatment meant the treatment either perfectly matched Benin's IMCI guidelines²¹ or was considered effective according to standard clinical textbooks.^9,25,26 To evaluate time trends and compare the quality of the health care delivered by IMCI-trained workers to that delivered by workers not trained in IMCI, we performed logistic regression modeling with the GENMOD procedure, which accounts for clustering with generalized estimating equations.

For evaluation of project impact with PBT data, the baseline period was approximately early 1999 to early 2001 (data collected from April 2001 to March 2003), before most interventions began, and the follow-up period was approximately early 2002 to early 2004 (data collected from April 2004 to March 2006), when interventions had been scaled up enough to plausibly affect mortality (Figure 2). We used log-binomial modeling with the GENMOD procedure to estimate the relative risk of death. The model included dummy variables for area (intervention vs comparison), time (follow-up vs baseline), and an area × time interaction term.

DHS results were either abstracted from survey reports^14,24 or provided by the survey team (R. Ren, ICF Macro, personal communication, October 2009 through February 2010).

RESULTS

Of 1041 consultations included in the health facility surveys in the intervention area, the proportion performed by IMCI-trained health workers increased from 0% in 1999 to 63.1% in 2004 (Table 1). Of 923 children with potentially life-threatening illnesses, the proportion receiving adequate treatment increased from 25.5% in 1999 to 44.4% in 2004 (time trend P < .001). To determine whether the increase in quality was caused by IMCI, we found that the proportion of children receiving adequate treatment was significantly greater for children seen by IMCI-trained health workers (51.6%) than for those seen by non–IMCI-trained health workers (25.0%; P < .001). Similarly, for the subgroup of children with severe illnesses, IMCI-trained health workers outperformed non-IMCI-trained workers (69.4% vs 26.8%, respectively; P < .001). Supervision was relatively weak, despite efforts to improve it.²⁷ Instances of key drugs being out of stock were commonplace, although virtually all facilities stocked an oral antimalarial and antibiotic.

As quality was improving, health facility utilization increased. The estimated number of children with potentially life-threatening illnesses seen at facilities in the intervention area grew 18.4%, from 304 per weekday in 2001 to 360 per weekday in 2004 (Box C available as a supplement to the online version of this article at http://www.ajph.org).

Mortality

At baseline, the IECM (risk of dying before about age 30 months) was nearly the same in the intervention and comparison areas (105.9 and 106.8 per 1000 live births, respectively; Table 3; Figure C available as a supplement to the online version of this article at http://www.ajph.org). From baseline to follow-up in the intervention area, mortality decreased significantly (relative risk [RR] = 0.870; 95% CI = 0.841, 0.900): that is, by 13.0% (95% CI = 10.0%, 15.9%). In the comparison area, mortality was unchanged (RR = 1.013; 95% CI = 0.979, 1.049). The ratio of these trends (0.859; 95% CI = 0.818, 0.902) was statistically significant: the intervention-area trend was 14.1% lower than the comparison-area trend (95% CI = 9.8%, 18.2%).

TABLE 3.

Mortality Results, Indicators of Mortality Data Quality, and Validity Assessment of Mortality Data: Malaria-Control Project, Benin, 1999–2006

Characteristic	Intervention Area	Comparison Area
PBT mortality results (risk of dying before about 30 mo of age/1000 live births)
Baseline perioda (no./total no.; 95% CI)	105.9 (5997/56 630b; 103.4, 108.4)	106.8 (5565/52 090; 104.2, 109.5)
Follow-up periodc (no./total no.; 95% CI)	92.2 (5953/64 597; 89.9, 94.4)	108.3 (5760/53 210; 105.6, 110.9)
Relative change from baseline to follow-up, % (95% CI)	−13.0d (–10.0, –15.9)	1.3 (–2.1, 4.9)
Indicators of quality of PBT data
Expected monthly reports collected,e % (no./total no.)	98.2 (6763/6890)	99.0 (7242/7316)
Deliveries recorded in the health facility birth register with the PBT questions completed with no obvious errors, % (no./total no.)	97.3 (201 422/207 048)	97.3 (171 634/176 464)
Case scenarios on the PBT questions that birth attendants answered correctly, % (no./total no.)	98.0 (16 765/17 104)	97.8 (17 783/18 176)
Validity assessment of PBT mortality results
PBT mortality results for the period of about 1999–2004 (95% CI)f	98.4 (96.9, 99.9)	109.9 (108.2, 111.6)
Estimated risk of dying before age 2 y/1000 live births, from the 2006 Benin Demographic and Health Survey for 1999–2004 (95% CI)f	105.9 (93.1, 118.7)	102.5 (89.1, 115.9)

Note. CI = confidence interval; PBT = preceding birth technique.

^aData collected from April 2001 to March 2003, which reflects mortality from approximately early 1999 to early 2001.

^bThe denominator is the number of women delivering who had a previous live birth, and the numerator is the number of women delivering who had a previous live birth whose last live-born child had died by the time of the current delivery.

^cData collected from April 2004 to March 2006, which reflects mortality from approximately early 2002 to early 2004.

^dCalculations based on exact PBT results, not rounded results in the preceding 2 rows; e.g., relative change of –13.0% = [(5953/64 597)–(5997/56 630)] / (5997/56 630).

^eAfter health facilities were recruited into the PBT system, 1 report (i.e., 1 set of PBT statistics) was expected from each health facility for each month that the health facility was open from April 2001 to March 2006.

^fIf results for all 4 departments are pooled over the 5-year period, the PBT result is 103.7/1000 live births, and the gold standard DHS result is 104.2/1000 live births. This result suggests that the PBT result might have actually estimated the mortality risk before 24 months of age (the interpretation frequently cited in the published literature^22,23).

PBT data-quality indicators showed high levels of data completeness and birth-attendant knowledge (Table 3). A validity assessment revealed that PBT and “gold standard” DHS results were similar, with PBT 95% CIs always well within DHS CIs.

Household Surveys

The 2001 DHS included 1015 households in the intervention area and 1113 households in the comparison area¹⁴; corresponding samples in the 2006 DHS were 3344 intervention-area households and 3553 comparison-area households.²⁴ Anemia prevalence was similar in both areas at baseline and declined somewhat in both areas (5–8 percentage points; Table A available as a supplement to the online version of this article at http://www.ajph.org). Malnutrition trends were similar.

Plausibility Argument

To determine whether it was plausible to conclude that project activities could have led to reduced mortality, we examined steps in the causal pathways (Figure 1). We have already presented findings from the current study supporting a causal role for 5 steps in the figure (IMCI introduction, training, improved quality in facilities, community mobilization, and ITN promotion). Evidence in support of a causal role for the remaining 2 steps (improved care-seeking and treatment, and increased ITN use) was derived from 6 household-survey indicators. For 5 of these indicators, intervention-area improvements ranged from 7.7 percentage points to 23.7 percentage points (Table 2). These improvements were greater than those seen in the comparison area, although some differences were small. For example, care-seeking for respiratory infections improved from 23.4% to 46.9% (23.5 percentage points) in the intervention area and from 28.7% to 36.8% (8.1 percentage points) in the comparison area. Thus, the difference between intervention and comparison area trends was 15.4 percentage points (i.e., 23.5 minus 8.1; Table 2) and favored lower mortality in the intervention area. The smallest difference between area-specific trends was for ITN use (3.8 percentage points), which suggests that ITN promotion in both study areas was similarly successful.

For the remaining indicator (fever treated with an antimalarial drug), trends surprisingly worsened in both areas (Table 2), but the difference between area-specific trends was negligible (–1.7 percentage points). The reason for the trends worsening may have been that parents recognized that chloroquine was losing efficacy. Malaria was a leading cause of childhood deaths, so a decrease in fever treatment might seem inconsistent with reduced mortality in the intervention area. However, 3 reasons support the plausibility argument. First, the indicator does not differentiate between correct and incorrect treatment, and a survey in the intervention area in 1999 found that home treatment with chloroquine was often ineffective because of underdosing.²⁸ Second, an increase was found in the proportion of febrile children treated at health facilities, where quality of treatment was improving, and this increase was greater in the intervention area than in the comparison area. Third, estimates of cause-specific mortality fractions for Africa around the year 2000 show that the burden of pneumonia (the cause of 21% of deaths of children younger than 5 years) and diarrhea (16%) was similar to malaria's burden (18%).¹ Thus, even if IMCI's impact on malaria was small, improved case-management quality for other diseases could reasonably explain the observed mortality trends.

We also analyzed health-facility survey results on treatment quality and utilization to determine whether IMCI alone could explain the intervention area's 13.0% mortality reduction. We first estimated that the mortality reduction translated into about 400 fewer deaths of children younger than 2 years annually in the follow-up period compared with the baseline period (Box D, available as a supplement to the online version of this article at http://www.ajph.org). We then pooled results from the 2001 and 2004 surveys, and on the basis of these we estimated that IMCI could have caused an additional 12 000 children with potentially life-threatening illnesses to receive adequate treatment in the intervention area per year (Box C available as a supplement to the online version of this article at http://www.ajph.org). If providing adequate treatment truly saved just 4% of these ill children, then IMCI alone could have prevented the 400 deaths per year needed to account for the observed mortality reduction (i.e., 4% of 12 000 additionally treated children/year = 480 deaths/year prevented). The corresponding estimate for the subgroup of children with severe illnesses was an additional 3500 children per year receiving adequate treatment attributable to IMCI. If treatment truly saved just 12% of these seriously ill children, then IMCI alone could explain the mortality reductions. Although the preceding 2 sets of results are not conclusive, they generally support the conclusion that mortality reduction could be plausibly linked to project activities.

To search for explanations other than project activities that might account for the mortality reduction, we examined child health determinants that were not in the causal pathways at the top of Figure 1. These determinants are potential confounders of the intervention–mortality association. For 3 of these determinants (HIV prevalence, antimalarial resistance, and insecticide resistance), department-level data on trends were unavailable. However, higher-level data did not suggest any large differences between intervention-area and comparison-area trends; thus, these determinants were unlikely to have been important confounders. For example, the national HIV prevalence among people aged 15 to 49 years was about 1% for the entire study period²⁹ (0.7% in the intervention area and 0.6% in the comparison area for 2006²⁴).

Area-specific trends were available for 14 indicators. For 2 of these indicators—improved water sources and tetanus vaccinations—trends favored lower mortality in the intervention area (Table 2). Thus, theoretically these factors could have been the true causes of the reduced mortality (i.e., positive confounding^6,30). However, differences in intervention-area and comparison-area trends were so small (1.7 and 3.5 percentage points, respectively) that it is highly unlikely that they could have accounted for the entire mortality reduction. For the remaining 12 indicators (e.g., vitamin A supplementation, rainfall, vaccinations), trends favored lower mortality in the comparison area (Table 2; Figure D available as a supplement to the online version of this article at http://www.ajph.org). These factors could have led to an underestimation of the project's impact (i.e., negative confounding³⁰).

As previously mentioned, ITN promotion and some IMCI were implemented in the comparison area. This implementation represents contamination, which would have led to an underestimation of the project's impact. The small number of IMCI-trained health workers probably did not cause large improvements in treatment quality, but ITN promotion was almost equally successful in both study areas.

Cost-Effectiveness of IMCI

The cost of IMCI from 2001 through 2004 in the intervention area was about $237 000 (Box E available as a supplement to the online version of this article at http://www.ajph.org). Assuming that IMCI truly caused the mortality decline (814 deaths prevented over the 2-year follow-up period), the cost-effectiveness ratio of IMCI was $291 per death prevented. Results from a sensitivity analysis ranged from $233 to $582 per death prevented.

DISCUSSION

In a malaria-endemic area with widespread poverty and weak infrastructure, we found that a malaria-control project using IMCI and ITNs was associated with a 14% reduction in early-childhood mortality. Because ITN promotion was almost equally successful in the intervention and comparison areas, it is possible that the mortality impact of ITNs in the 2 areas cancelled each other out. Thus, the mortality reduction could have been primarily attributable to IMCI's positive effect on care-seeking and health care quality in outpatient facilities. Furthermore, IMCI's cost-effectiveness seemed favorable. This study was done when malaria was usually treated with chloroquine, so IMCI's impact today might be larger because current guidelines recommend more efficacious antimalarials.¹⁷

We examined numerous contextual factors to identify bias. Two factors (improved water sources and tetanus vaccinations) might have caused a small overestimation of mortality impact, but many more factors (e.g., vitamin A supplementation, rainfall, child health activities in the comparison area) might have caused an underestimation of impact. Additionally, the PBT results might have underestimated impact because a small part of the baseline risk reflected mortality shortly after project activities began, and a small part of the follow-up risk reflected mortality before project activities.

Although trends in ITN use and other indicators favored lower mortality in the comparison area, no significant change was observed. There are several potential explanations for this finding. First, the ITN impact might have been small because use was relatively low (24.9%). Second, some indicators (vaccinations and diarrhea treatment) worsened over time. Third, an undetected outbreak might have counteracted programmatic effects. To explore this possibility, especially after the mortality increase in year 3 of PBT data (Supplemental Figure C, available online at http://www.ajph.org), we spoke with comparison-area health officials. They were unaware of increased disease activity, although an outbreak still could have been missed.

Our study had several limitations. First, with an ecological analysis the causal link between interventions and outcomes is uncertain. Second, the analysis of contextual factors was semiquantitative; thus, we could identify potential confounders, but mortality results could not be accurately adjusted. Also, local trends for some factors (e.g., household income, antimalarial resistance) were unavailable, a limitation that has affected other IMCI studies.⁶ Third, health care quality was not measured in the comparison area. For positive confounding, however, health care quality would have had to worsen, which seems unlikely given that some IMCI training occurred.

Fourth, because of budget constraints we had to use existing data sources (primarily DHSs) that were not perfectly timed to project activities, which could have caused an underappreciation of how project activities affected mortality. Evaluation approaches such as continuous surveys³¹ could have greatly improved the temporal-spatial matching of outcomes with interventions. Lastly, although IMCI improved health care quality many consultations were done by non–IMCI-trained health workers, and IMCI-trained health workers performed relatively poorly. Thus, IMCI's potential was not fully realized. This finding is consistent with results of a recent review that found that after IMCI training, one third of ill children needing oral antimicrobials or rehydration typically did not receive these treatments as recommended.¹⁰

Regarding this last point, our finding of reduced mortality despite only modest improvements in case-management quality might be important. This finding suggests that in settings with low coverage of key preventive interventions, improved case management might have quite a large potential to prevent child deaths, which highlights the importance of developing strategies to improve health-worker practices. Furthermore, finding ways to improve performance for prevention and treatment of childhood illnesses would likely translate to improvements in delivery of services in other health areas, such as HIV/AIDS, obstetric care, injuries, and noncommunicable diseases.

Interestingly, 2 other IMCI studies with relatively rigorous designs found mortality reductions that were similar to ours (13% in both studies).^8,11 Thus, despite the limitations of all these studies a picture of IMCI's impact on mortality may be emerging. That said, IMCI is primarily a strategy for scaling up interventions already proven to be efficacious, so indicators of health care quality and intervention coverage might be more logical than mortality as metrics for judging IMCI. If IMCI causes meaningful increases in intervention coverage but mortality does not decrease, then it is the efficacy of the interventions that should be in question.

In summary, we observed lower mortality after IMCI implementation and ITN promotion in the intervention area. Trends in contextual factors suggested that it was plausible to conclude that project activities caused the mortality decline. ITN use increased similarly in both study areas, so the mortality impact of ITNs in the 2 areas might have cancelled each other out, and the lower mortality could have been primarily attributable to IMCI's positive effect on care-seeking and health care quality. Inadequate performance by IMCI-trained health workers underscored the urgent need for effective strategies to improve health-worker adherence to clinical guidelines.