Impact of a health system strengthening intervention on maternal and child health outputs and outcomes in rural Rwanda 2005–2010

Abstract

Introduction

Although Rwanda’s health system underwent major reforms and improvements after the 1994 Genocide, the health system and population health in the southeast lagged behind other areas. In 2005, Partners In Health and the Rwandan Ministry of Health began a health system strengthening intervention in this region. We evaluate potential impacts of the intervention on maternal and child health indicators.

Methods

Combining results from the 2005 and 2010 Demographic and Health Surveys with those from a supplemental 2010 survey, we compared changes in health system output indicators and population health outcomes between 2005 and 2010 as reported by women living in the intervention area with those reported by the pooled population of women from all other rural areas of the country, controlling for potential confounding by economic and demographic variables.

Results

Overall health system coverage improved similarly in the comparison groups between 2005 and 2010, with an indicator of composite coverage of child health interventions increasing from 57.9% to 75.0% in the intervention area and from 58.7% to 73.8% in the other rural areas. Under-five mortality declined by an annual rate of 12.8% in the intervention area, from 229.8 to 83.2 deaths per 1000 live births, and by 8.9% in other rural areas, from 157.7 to 75.8 deaths per 1000 live births. Improvements were most marked among the poorest households.

Conclusion

We observed dramatic improvements in population health outcomes including under-five mortality between 2005 and 2010 in rural Rwanda generally and in the intervention area specifically.

Key questions.

What is already known?

• Much of the evidence that health system strengthening in rural Africa has improved health outcomes comes from studies of targeted regional interventions such as performance-based financing or community health worker programmes, rather than integrated interventions that encompass multiple components including infrastructure and supply chain investments, health management information system, workforce training and incentives at all levels, community health workers and free services for poor patients.

• In addition to these experimental or quasiexperimental studies, a series of case studies have documented individual nations’ Pathways to Achieving Millennium Development Goal 4 target, the reduction of under-five mortality by two-thirds between 1990 and 2015.

• These reports suggest that improvements in coverage of reproductive, maternal and child health indicators explain some, but not all, of the decline in child mortality and that these successes occurred in the context of national gains in health, nutrition and food security, sanitation, poverty reduction and access to clean water.

What are the new findings?

• Coverage of most maternal and child healthcare interventions improved at a similar pace in our rural intervention area and other rural areas.

• Despite experiencing poorer health outcomes in 2005, our rural intervention area caught up to or exceeded other rural areas on 23 of 25 population health indicators by 2010.

• Infant and under-five mortality declined in our rural intervention area even more precipitously than in other rural areas of Rwanda between 2005 and 2010.

Key questions.

What do the new findings imply?

• The process of strengthening national health systems often involves trade-offs between a focus on first testing individual programmes that distributed widely, as is often practised by pilot programmes with multilateral institutions, or implementing multiple simultaneous programmes locally. Our results show that integrated health system strengthening interventions can be locally adapted to enable the rapid expansion of healthcare coverage as well as dramatic improvements in population health outcomes.

• Integrated multilevel interventions can also help narrow the healthcare coverage and outcome gap between richer and poorer members of a society.

• National governments can leverage non-governmental partners to achieve the health-related sustainable development goals through joint implementation of national health policy.

Introduction

The 1994 Rwanda Genocide was followed by a profound decline in population health that persisted for almost a decade. In the aftermath of the killing of nearly 20% of the population, HIV incidence soared, a cholera epidemic among the Rwandan refugees ensued and vaccination rates plummeted. In 2000, the new government launched a development initiative, Vision 2020, of which health equity was a major component, and in 2003, Rwanda established health as an inalienable right.¹ The many healthcare initiatives implemented nationally between 2003 and 2010 included a national health insurance policy,² performance-based financing of health programmes,³ a village community health worker programme,³ scale up of vaccinations,⁴ HIV treatment⁵ and malaria reduction initiatives.⁶ Between 2004 and 2011, antiretroviral therapy coverage increased sevenfold to 94%, and by 2011,⁷ the government of Rwanda spent 10% of public expenditure on health.⁸

In 2005, the non-governmental organisation, Partners In Health (PIH), and the Rwandan Ministry of Health (RMOH) began a collaboration to strengthen the health system in a region of southeastern Rwanda (henceforth referred to as Kirehe/S. Kayonza) where health outcomes were among the worst in Rwanda. Children in this area experienced higher rates of death, acute respiratory infection (ARI), diarrhoea and fever than in the other rural areas (ORAs) of the country.⁹ To address this, PIH and RMOH jointly led a regional effort based on the World Health Organization six building blocks of health system strengthening.¹⁰

The intervention included the renovation and equipping of dysfunctional health facilities, the recruitment, retention and training of a health workforce, the development of a medical record system, the procurement of medical products and technologies, financial support to offset health insurance premium costs and user fees and the development of governance strategies to ensure the longevity of the project. These interventions coincided with major RMOH reforms to coordinate external aid with government policies, scale-up a community-based health insurance scheme and introduce performance-based pay into the district health system.¹¹ Notably, many of these interventions were designed to reduce child mortality rates by increasing the coverage of preventative measures such as vaccination and antenatal visits and through the expansion of community health worker networks and provision of high-quality medical care at primary, secondary and tertiary health facilities. The intervention included an accompagnateur community health worker role not present in the rest of the country to provide patients with HIV/AIDS, tuberculosis and other chronic conditions with daily home visits. Specific aspects of the RMOH-PIH intervention are described in the online supplement and reviewed elsewhere.¹²

An important principle of the RMOH-PIH collaboration in Kirehe/S. Kayonza was to leverage existing resources rather than spend limited resources on building new systems. Routinely conducted Demographic and Health Surveys (DHSs) provide extensive data that can be used to measure health system outputs and population health outcomes.¹³ Here, we assessed the impact of the RMOH-PIH intervention using data from two DHSs sequentially conducted in Rwanda 5 years apart.

Methods

We evaluated the impact of the RMOH-PIH interventions by comparing the temporal trends in health outputs and outcomes between 2005 and 2010 in the intervention target region to those in the pooled population of all ORAs. First, we assessed health system outputs, focusing on a set of indicators meant to capture the coverage of maternal and child health services and then we assessed population outcomes including neonatal, infant and under-five mortality.

Data

We used Rwanda Demographic and Health Survey (RDHS) data collected from 21 338 women living in Kirehe/S. Kayonza (K/SK) and ORAs in 2005 (K/SK: 418, ORA: 8217) and 2010 (K/SK: 2073, ORA: 10 630) (table 1). RDHSs are nationally and subnationally representative two-stage cluster samples conducted roughly every 5 years by the RMOH, National Institute of Statistics-Rwanda (NISR) and ICF International. The surveys collect information from women aged 15–49 on their reproductive health histories, practices and desires; household composition; siblings’ survival; and children’s health and survival. The DHS birth history module included the date of the birth and death of each child born alive, and through the sibling module, the age and date of death for each biological sibling (see online supplement). The 2005 RDHS was underway at the onset of the RMOH-PIH collaboration. In order to expand the sample size to allow us to compare the intervention area to other areas, we coordinated with the NISR immediately following the 2010 RDHS to collect a supplemental sample of 1391 households from 54 primary sampling units (PSUs) in Kirehe/S. Kayonza using the same sampling frame, staff and questionnaires as the 2010 RDHS (figure 1).¹² Most data were collected consistently across the three surveys (2005 RDHS, 2010 RDHS and Supplemental RDHS) although neither stunting and wasting was assessed in the supplemental survey. The response rates for surveyed women were 98% in 2005 and 99% in 2010. The RDHS urban/rural boundaries were adopted from the Rwandan government. Exact urban boundaries are not published, however, urban areas are described as being built up, population dense and having public services and facilities. All areas not meeting this definition are considered rural.¹⁴

Table 1.

Summary of sociodemographic characteristics in 2005 and 2010

	2005		2010		Comparison of group trends
	ORA	K/SK	ORA	K/SK	P values*
Women (15–49 years)
Age	28.6	28.1	28.6	28.7	0.059
Married/partnered (%)	50.6	54.1	51.2	54.4	0.904
Employed (%)	94.6	94.6	86.1	87.9	0.379
Literacy (%)	68.0	63.7	75.2	73.3	0.454
N (weighted)	8877	523	10 957	2084
N (unweighted)	8217	418	10 630	2073
Households
Wealth score†	−0.246	−0.290	0.242	0.293	0.013
Electricity (%)	1.4	<0.1	4.1	4.5	0.175
Finished floor (%)	7.1	5.1	10.7	8.5	0.902
Improved water (%)	27.9	47.5	72.4	65.5	0.014
Members 15+ with primary education (%)	66.7	70.5	75.2	75.2	0.205
N (weighted)	8168	498	10 157	2041
N (unweighted)	7702	394	9891	2031
Primary sampling units
Distance to road (metres)	3105	3275	3221	2960	0.543
Distance to Kigali (km)	50.8	49.6	47.2	46.3	0.972
Elevation (metres)	1763	1488	1800	1527	0.990
January average rainfall (mm)	113	117	113	115	0.483
April average rainfall (mm)	175	176	175	176	0.965
July average rainfall (mm)	83	67	83	69	0.507
October average rainfall (mm)	188	179	188	180	0.417
N (unweighted)	331	17	388	79

*Based on t-test from an OLS regression with year, group and year–group interaction terms.

†With respect to 2005 wealth score definition.

K/SK, Kirehe/S. Kayonza; OLS, ordinary least squares; ORA, other rural area.

Figure 1.

Maps of RDHS strata and primary sampling units in 2005 (left) and 2010 (right) and the PIH-RMOH intervention area in southeastern Rwanda (green). PIH, Partners In Health; RDHS, Rwanda Demographic and Health Survey; RMOH, Rwandan Ministry of Health.

Indicators

We assessed the following child health-related health system output indicators: whether treatment was provided for recent episodes of ARI, diarrhoea or fever in children under age 5 years; whether children under age 2 years received the recommended three doses of diphtheria-polio-tetanus (DPT) or measles vaccine; whether children received vitamin A supplementation between age 6 months and 1 year; whether children were exclusively breastfed for the first 6 months of life; whether at least one, or the recommended four, antenatal care visits took place during the last pregnancy; whether the most recent birth was attended by a skilled health worker; whether the birth was delivered by caesarean section; whether women received postnatal care within 24 hours of delivery; women’s current contraceptive use; and their unmet need for contraception. The following population health outcome indicators were assessed: neonatal, infant and under-five mortality; adult mortality (men, women and combined); recent occurrence of ARI, diarrhoea or fever in children under age 5 years; and stunting and wasting in children under age 5 years.¹⁵

We calculated a composite coverage index (CCI) for both groups to monitor overall healthcare coverage across time in the intervention and comparison areas based on that proposed by Barros and Victoria (2013) but modified to exclude BCG tuberculosis vaccination coverage (which was not available) as an indicator.¹⁶ The modified CCI is a weighted average for a group that includes the prevalence of met need for contraception (FPS), skilled birth attendance (SBA), at least one ANC visit with a skilled provider (ANCS), third DTP vaccination (DPT3), measles vaccination (MSL), oral rehydration therapy (ORT) for children with diarrhoea and care seeking for pneumonia symptoms (CPNM) as follows:

$${\displaystyle mCCI=\frac{1}{4}\left(FPS+\frac{SBA+ANCS}{2}+\frac{2\left(DPT3\right)+MSL}{3}+\frac{ORT+CPNM}{2}\right)}$$

Analysis

We integrated the data from the supplemental survey with that from the 2010 RDHS as follows. Sampling probability weights were recalculated for the combined 2010 dataset. To protect respondent confidentiality, we randomly geodisplaced PSU latitude/longitude coordinates in the supplemental survey up to 5 km within district boundaries according to DHS guidelines.¹⁷ All geographic information was linked to displaced PSU locations in a geographic information system (ArcGIS V.10, ESRI). We combined latitude/longitude coordinates and rural residence information to identify respondents living in Kirehe/S. Kayzona and ORAs. Additional PSU geographic characteristics included straight-line distance to the nearest main road in metres (downloaded from DIVA-GIS database), straight-line distance to Kigali province in kilometres (downloaded from Map Library database), elevation above sea level in metres (from RDHS) and 30-year (1971–2000) average total rainfall in millimetres during the months of January, April, July and October (downloaded from US National Oceanic and Atmospheric Administration Climate Prediction Center). We generated comparable household wealth scores for 2010 using the principle components generated from the 2005 RDHS,¹⁸ and we considered a household to be ‘poor’ if it ranked in the bottom 20% of wealth scores of the pooled 2005, 2010 and supplemental survey datasets. Three rural PSUs (representing 69 households and 70 women) were excluded from the 2005 dataset because the PSU GPS coordinate was missing, preventing the PSU from being linked to either Kirehe/S. Kayonza or ORA.

We compared baseline differences in woman, household and community (PSU) characteristics between the two comparison groups using χ² tests and t-tests and temporal changes in and between groups using ordinary least squares regression with a year, group and year-by-group interaction term. Since the intervention area and ORAs may differ by factors that would have an impact on child mortality but which would not be altered by our health system strengthening intervention, we also compared the following social and geographic characteristics of sampled communities: fraction of each PSU with improved water, fraction of PSU adults who received a primary education, distance of PSU to a main road and to Kigali, elevation and average total rainfall in specified months. Finally, we compared health system outputs and population health measures at baseline.

Kirehe/S. Kayonza had been chosen as the site of the intervention, because it had the highest rate of under-five mortality in Rwanda in 2005; thus, it was not possible to identify another Rwandan rural site with comparable under-five mortality. In an attempt to identify an optimal comparison group for Kirehe/S. Kayonza, we first limited our comparison area to Eastern Province in proximity to the intervention area (see figure 1). We found under-five mortality was higher in the intervention area than the rest of Eastern Province or any other subregion, and comparisons of other indicators were mixed (see online supplement). When a subregion was not identified for comparison, we used propensity score matching with inverse probability of treatment weights to identify comparable communities from ORAs by assessing PSU characteristics that might differ between the intervention and comparison areas but which would not be expected to be altered by the intervention (distance to road, distance to Kigali, elevation and average rainfall). We generated a bias B value to capture the difference in the standard deviation between the means of the groups and an R value that is the ratio of variances in the two groups. Following Rubin, we considered the groups to be balanced if B was less than 25% and R was between 0.5 and 2.¹⁹ Kirehe/S. Kayonza and ORAs were balanced by the R value but not by the B bias value (B=340.9, R=0.54) (see online supplement). Since neither approach identified a more appropriate comparison group, we compared Kirehe/S. Kayonza to all ORAs adjusting models for sociodemographic characteristics that followed different trends in Kirehe/S. Kayonza and ORAs over time.

We used ordinary least squares regression with group, year and group–year interaction terms to model changes in binary health outputs and outcomes, controlling for differences in woman’s age and household wealth. We modelled change in childhood mortality rates using the DHS synthetic life-table approach, which uses the histories of all children a mother reports to have been alive during the previous 5 years.²⁰ Adult mortality rates were based on a 5-year synthetic cohort of respondent’s siblings' births and deaths.²¹ Expected mortality rates were calculated by standardising mortality rates of ORAs to the age structure in Kirehe/S. Kayonza. We estimated mean changes between 2005 and 2010 as the absolute difference in rates, and we calculated variances of trends as the sum of year–group variances. We adjusted for clustering of observations by PSU using Taylor linearised variance estimation in regression models and jackknife repeated replications to estimate variance in all other analyses.²¹ We conducted regressions in Stata V.13 and mortality analyses in SAS V.9.2.

Ethics statement

Verbal consent was obtained for all respondents before interviews took place. Protocols for the Rwanda 2005 and 2010 DHSs were approved by the Rwandan government. Protocols for the 2010 supplemental survey were reviewed and approved by the Partners HealthCare Internal Review Board (protocol #: 2009 P-001941/8) and the Rwanda National Ethics Committee.

Role of the funding source

The funders had no role in study design, data collection, data analysis, data interpretation, writing of this report, nor the decision to submit this paper for publication. The corresponding author (DRT) had full access to all the data in the study and had final responsibility for the decision to submit for publication.

Results

Demographic and geographical data

At baseline, women in Kirehe/S. Kayonza and ORAs were similar in terms of age, marital status, employment and literacy. Households in Kirehe/S. Kayonza had fewer household assets but greater access to improved water sources than in ORAs (table 1). Kirehe/S. Kayonza also differs from ORAs in terms of its geography; this region has lower elevation and lower rainfall during the second half of the calendar year (table 1). Between 2005 and 2010, households in Kirehe/S. Kayonza acquired more assets than in ORAs and the age distribution of women shifted to the right. Access to improved water sources rose more steeply in the ORAs during the study period (table 1).

Health system outputs

Table 2 shows that after we adjusted for household wealth and woman’s age, baseline health system outputs were not significantly different in Kirehe/S. Kayonza with some exceptions: vitamin A supplementation was lower in Kirehe/S. Kayonza than in the ORAs (72.3% vs 84.3%, P=0.005), while contraception use among married women was higher (13.9% vs 8.2%, P=0.011) in Kirehe/S. Kayonza than in the ORAs (table 2, column D).

Table 2.

Health system outputs 2005–2010 in Kirehe/S. Kayonza (K/SK) and other rural areas (ORAs)

Indicator (subpopulation)	A	2005			2010			Trends	Comparison of trends
		B	C	D	E	F	G	H	I
	Region	N	% (95% CI)	K/SK versus ORA (P values)	N	% (95% CI)	K/SK versus ORA (P values)	2005 versus 2010 (P values)	2005–2010 K/SK versus ORA (P values)
Children
Exclusively breastfed by mother (last born 0–5 months)†	K/SK	35	89.50 (76.26 to 102.73)	0.29 (0.966)	130	92.12 (87.09 to 97.29)	5.63 (0.066)	2.07 (0.775)	5.34 (0.478)
	ORA	735	89.21 (86.53 to 91.88)		583	86.56 (83.40 to 89.71)		−3.27 (0.131)
Received third DPT vaccine (12–23 months)†	K/SK	68	91.41 (85.17 to 97.65)	4.05 (0.236)	252	97.16 (95.07 to 99.24)	0.06 (0.960)	5.45 (0.106)	−3.99 (0.270)
	ORA	1344	87.36 (84.93 to 89.80)		1366	97.10 (96.05 to 98.14)		9.44 (<0.001)
Received measles vaccine (12–23 months)†	K/SK	68	73.91 (61.53 to 86.29)	−11.79 (0.071)	252	92.08 (88.36 to 95.80)	−2.95 (0.144)	17.76 (0.007)	8.74 (0.198)
	ORA	1344	85.59 (82.78 to 88.41)		1366	95.03 (93.66 to 96.40)		9.02 (<0.001)
Vitamin A supplement in last 6 months (6–59 months)†	K/SK	330	77.27 (72.74 to 81.80)	−7.04 (0.005)	1272	89.14 (86.48 to 91.79)	−3.93 (0.006)	11.70 (<0.001)	3.11 (0.280)
	ORA	5549	84.31 (82.43 to 86.20)		6556	93.07 (92.15 to 93.99)		8.59 (<0.001)
ARI in last 2 weeks and received treatment (<5 years)†	K/SK	100	18.69 (9.93 to 27.45)	−7.51 (0.115)	116	54.58 (43.47 to 65.69)	18.05 (0.003)	34.53 (<0.001)	25.56 (0.001)
	ORA	1022	26.19 (22.98 to 29.40)		720	36.53 (32.58 to 40.48)		8.97 (0.001)
Diarrhoea in last 2 weeks and received ORT (<5 years)†	K/SK	72	12.12 (0.82 to 23.42)	−5.62 (0.346)	169	34.58 (26.46 to 42.70)	0.64 (0.887)	21.22 (0.003)	6.26 (0.402)
	ORA	886	17.74 (14.76 to 20.72)		931	33.94 (30.52 to 37.37)		14.97 (<0.001)
Fever last 2 weeks and received (<5 years)†	K/SK	138	18.77 (9.95 to 27.59)	6.74 (0.146)	215	11.41 (6.68 to 16.13)	0.55 (0.838)	−7.38 (0.150)	−6.19 (0.248)
	ORA	1619	12.03 (9.82 to 14.25)		1140	10.86 (8.50 to 13.22)		−1.19 (0.484)
Women
1+ antenatal care visit (births last 5 years)†	K/SK	277	96.77 (95.03 to 98.51)	2.14 (0.035)	1058	98.37 (97.54 to 99.21)	0.39 (0.420)	1.45 (0.142)	−1.75 (0.117)
	ORA	4374	94.63 (93.66 to 95.60)		5255	97.99 (97.56 to 98.42)		3.20 (<0.001)
4+ antenatal care visit (births last 5 years)†	K/SK	277	10.35 (6.13 to 14.58)	−2.43 (0.286)	1058	26.94 (23.38 to 30.50)	−8.28 (<0.001)	16.05 (<0.001)	−5.85 (0.057)
	ORA	4374	12.79 (11.33 to 14.24)		5255	35.22 (33.28 to 37.16)		21.89 (<0.001)
Attended by a skilled health worker (births last 5 years)†	K/SK	451	38.74 (29.48 to 47.99)	4.37 (0.369)	1496	64.48 (60.39 to 68.56)	−3.07 (0.180)	23.68 (<0.001)	−7.44 (0.167)
	ORA	7036	34.37 (32.03 to 36.71)		7588	67.55 (65.66 to 69.45)		31.12 (<0.001)
Caesarean birth (births last 5 years)†	K/SK	451	2.29 (−0.11 to 4.69)	0.11 (0.932)	1496	5.09 (3.74 to 6.44)	−0.94 (0.227)	2.33 (0.098)	−1.04 (0.477)
	ORA	7036	2.19 (1.74 to 2.64)		7588	6.03 (5.31 to 6.75)		3.37 (<0.001)
Postnatal care within 24 hours (last birth last 2 years)†	K/SK	169	2.09 (−0.18 to 4.36)	−0.63 (0.612)	523	17.17 (13.28 to 21.06)	0.11 (0.958)	14.74 (<0.001)	0.74 (0.767)
	ORA	2811	2.72 (1.86 to 3.58)		2680	17.06 (15.30 to 18.81)		14.00 (<0.001)
Unmet need for contraception (married women)†	K/SK	283	37.14 (30.39 to 43.88)	−1.38 (0.696)	1133	21.49 (18.55 to 24.44)	2.21 (0.177)	−14.65 (<0.001)	3.59 (0.357)
	ORA	4484	38.52 (36.90 to 40.13)		5619	19.28 (18.00 to 20.57)		−18.24 (<0.001)
Current use of modern contraceptive (married women)†	K/SK	283	13.85 (9.63 to 18.07)	5.61 (0.011)	1133	46.15 (42.45 to 49.85)	1.36 (0.515)	29.67 (<0.001)	−4.25 (0.162)
	ORA	4484	8.24 (7.28 to 9.20)		5619	44.79 (43.00 to 46.57)		33.92 (<0.001)
Modified composite coverage index	K/SK	–	57.9	–	–	75.0	–	–	–
	ORA	–	58.7		–	73.8		–

†Adjusting for mother’s/woman’s age and household wealth.

ARI, acute respiratory infection; ORT, oral rehydration therapy.

Most health system output indicators improved significantly between 2005 and 2010 in both groups (table 2, column H), although for many indicators, the comparison of trends between the two groups were not statistically significant (table 2, column I). Indicators that did not improve over time included exclusive breast feeding for the first 6 months of life and treatment of fever which did not change significantly in either group between 2005 and 2010 (table 2, column H). The proportion of children treated for ARI rose from 18.7% to 54.6% in Kirehe/S. Kayonza and from 26.2% to 36.5% in ORAs (P=0.001), while proportion of women receiving four antenatal visits rose from 10.4% to 26.9% in Kirehe/S. Kayonza and from 12.8% to 35.2% in ORAs (P=0.057) (table 2, column I). Overall health system coverage as measured by the modified CCI increased from 57.9% to 75.0% in Kirehe/S. Kayonza and from 58.7% to 73.8% in ORAs (table 2).

Population health outcomes

After adjustment, we found that most baseline measures of child health were worse in Kirehe/S. Kayonza than in ORAs (10.99% more ARI, P=0.005; 5.41% more diarrhoea, P=0.038; 11.91% more fever, P=0.003; 72 more under-five deaths per 1000 births, P=0.045), with the exception of stunting, which was 5.43% more prevalent in ORAs (P=0.018) (table 3, column D). We noted improvements between 2005 and 2010 for both groups in ARI, fever, neonatal mortality, infant mortality and under-five mortality (P<0.05 for all) (table 3, column H); these improvements were greater for ARI, diarrhoea, and fever in Kirehe/S. Kayonza (P<0.05 for all) (table 3, column J).

Table 3.

Health outcomes 2005–2010 in Kirehe/S. Kayonza (K/SK) and other rural areas (ORAs), adjusting for mother’s age and household wealth

Indicator (subpopulation)	A	2005			2010			Trends		Comparison of trends
		B	C	D	E	F	G	H	I	J
	Region	N	% (95% CI)	K/SK versus ORA (P values)	N	% (95% CI)	K/SK versus ORA (P values)	2005 versus 2010 (P values)	Average annual rate of reduction	2005–2010 K/SK versus ORA (P values)
Children
ARI in last 2 weeks (<5 years)†	K/SK	365	27.24 (19.71 to 34.77)	10.99 (0.005)	1402	8.41 (6.54 to 10.28)	−1.64 (0.125)	−18.64 (<0.001)	n/a	−12.63 (0.002)
	ORA	6288	16.25 (14.75 to 17.75)		7152	10.05 (9.10 to 11.00)		−6.01 (<0.001)	n/a
Diarrhoea in last 2 weeks (<5 years)†	K/SK	365	19.51 (14.52 to 24.49)	5.41 (0.038)	1402	12.18 (10.13 to 14.23)	−0.81 (0.490)	−7.24 (0.009)	n/a	−6.22 (0.030)
	ORA	6288	14.10 (12.98 to 15.23)		7152	12.99 (11.93 to 14.06)		−1.02 (0.202)	n/a
Fever in last 2 weeks (<5 years)†	K/SK	365	37.67 (29.90 to 45.43)	11.91 (0.003)	1402	15.41 (12.71 to 18.10)	−0.52 (0.729)	−22.17 (<0.001)	n/a	−12.43 (0.004)
	ORA	6288	25.75 (23.91 to 27.59)		7152	15.92 (14.76 to 17.08)		−9.74 (<0.001)	n/a
Stunting (<–2SD) (<5 years)†	K/SK	184	47.85 (43.87 to 51.83)	−5.43 (0.018)	204	46.83 (41.19 to 52.47)	0.51 (0.868)	3.64 (0.314)	n/a	5.94 (0.122)
	ORA	3158	53.28 (51.17 to 55.39)		3636	46.32 (44.20 to 48.44)		−2.30 (0.163)	n/a
Wasting (<–2SD) (<5 years)†	K/SK	184	3.22 (0.28 to 6.16)	−1.61 (0.307)	204	2.14 (0.39 to 3.89)	−0.63 (0.503)	−0.45 (0.800)	n/a	0.97 (0.596)
	ORA	3158	4.82 (3.90 to 5.75)		3636	2.77 (2.16 to 3.39)		−1.42 (0.018)	n/a
Under-five mortality rate (per 1000 live births)	K/SK	310	229.8 (158.8 to 300.7)	72.1 (0.045)	1347	83.2 (66.2 to 100.2)	7.4 (0.418)	−146.6 (<0.001)	−12.8	−64.7 (0.081)
	ORA	5116	157.7 (145.8 to 169.5)		6575	75.8 (69.1 to 82.5)		−81.9 (<0.001)	−8.9
Infant mortality rate (per 1000 live births)	K/SK	422	129.6 (77.8 to 181.3)	41.3 (0.116)	1462	49.4 (35.0 to 63.9)	0.5 (0.948)	−80.2 (0.003)	−19.3	−40.8 (0.136)
	ORA	6528	88.3 (79.4 to 97.2)		7403	48.9 (43.3 to 54.5)		−39.4 (<0.001)	−11.8
Neonatal mortality rate (per 1000 live births)	K/SK	452	55.9 (29.9 to 81.9)	17.6 (0.185)	1524	26.2 (15.4 to 36.9)	−0.9 (0.875)	−29.7 (0.035)	−15.2	−18.5 (0.201)
	ORA	7012	38.3 (32.9 to 43.8)		7662	27.1 (23.0 to 31.2)		−11.2 (0.001)	−6.9
Adults
Mortality among women (per 100 000 population)	Observed	3952	497.4 (256.4 to 738.5)	−151.8 (0.233)	18 279	256.8 (178.3 to 335.3)	−72.3 (0.110)	−240.6 (0.063)	−13.2	79.5 (0.556)
	Expected*	74 932	649.2 (584.1 to 714.4)		96 696	329.1 (288.1 to 370.1)		−320.1 (<0.001)	−13.6
Mortality among men (per 100 000 population)	Observed	4059	400.0 (171.2 to 628.9)	−296.0 (0.016)	16 707	325.7 (236.8 to 414.6)	−21.4 (0.673)	−74.3 (0.553)	−4.1	274.6 (0.039)
	Expected*	66 592	696.0 (621.3 to 770.7)		90 199	347.1 (303.2 to 390.9)		−348.9 (<0.001)	−13.9
Total adult mortality (per 100 000 population)	Observed	8011	445.0 (279.4 to 610.7)	−226.2 (0.011)	34 986	289.0 (230.0 to 348.0)	−48.8 (0.158)	−156.0 (0.082)	−8.6	177.4 (0.062)
	Expected*	141 524	671.2 (618.5 to 724.0)		186 894	337.8 (304.7 to 370.8)		−333.5 (<0.001)	−13.7

*Expected represents mortality rates in other rural areas standardised to the age distributions in Kirehe/S. Kayonza.

†Adjusting for mother’s age and household wealth.

ARI, acute respiratory infection.

Importantly, under-five mortality dropped precipitously in both groups during the study period from 229.8 to 83.2 under-five deaths per 1000 live births in Kirehe/S. Kayonza (P<0.001) and from 157.7 to 75.8 in ORAs (P<0.001) (table 3, column H). These changes represent annual reductions in under-five mortality of 12.8% and 8.9% for the intervention and ORAs, respectively (table 3, column I). The greatest reductions in childhood mortality occurred among the poorest households in Kirehe/S. Kayonza with a dramatic drop in mortality from 275.4 to 89.4 deaths per 1000 live births in Kirehe/S. Kayonza (annual rate of reduction: 13.5%) compared with 152.2 to 76.2 deaths per 1000 live births in ORAs (annual rate of reduction: 9.0%). Figure 2 shows that the gains in child survival in wealthier strata were much less steep. By 2010, there were no significant differences between Kirehe/S. Kayonza and ORAs in terms of child health outcomes (P>0.05 for all) (table 3, column G).

Figure 2.

Comparison of trends in under-five, infant and neonatal mortality between 2005 and 2010 in Kirehe/S. Kayonza versus other rural areas by household wealth status.

In 2005, the standardised male adult mortality rate was lower in Kirehe/S. Kayonza than in ORAs (P=0.016), and there was no difference in rates for women (P=0.233) (table 3, column D). Both female and male adult mortality in Kirehe/S. Kayonza dropped between 2005 and 2010 (from 497 to 257 deaths per 100000 among women and from 400 to 326 deaths per 100000 among men) (table 3, columns C and F), and there was no difference in the change in standardised adult mortality rates between the two groups (P=0.062) (table 3, column J).

Discussion

Between 2005 and 2010, coverage of health system outputs improved dramatically in the area of southeast Rwanda targeted by the intervention and in ORAs of the country. This increased coverage was accompanied by steep declines in adult, under-five, infant and neonatal mortality in both settings. Although changes in health system outputs were similar in both groups, the average annual declines in under-five, infant and neonatal mortality were steeper in the intervention area. The differences in these rates of decline in population outcomes did not meet statistical significance, however, possibly because the 2005 RDHS was underpowered to detect subregional differences, and we did not oversample the intervention area at baseline.

These data are consistent with the findings from a recent overview of global trends in child mortality that identified Rwanda as a top performer worldwide in reducing under-five mortality between 2000 and 2015.²² Though Rwanda is classified as a least developed country, its national 9.9% annual rate of reduction in under-five deaths is surpassed only by the upper middle-income nation of the Maldives.²³ This reduction is more than twice the global rate of 4.4% during the same time period.²³

Multiple studies in other settings suggest that national health gains can often mask substantial heterogeneity in health system performance and outcomes.^{24 25} In many settings, impoverished and/or geographically inaccessible areas have experienced slower progress in achieving health goals.^{26 27} In other studies, socioeconomic status did not correlate well with health system performance.²⁸ Our results show that the historically unprecedented improvement in health indicators not only extends to urban and wealthier areas of the country, but is also possible among its poorest and most geographically isolated residents. Notably, almost all of the decline in under-five mortality occurred among the lowest two wealth quintiles in the Kirehe/S. Kayonza region where the intervention included specific components (subsidies of insurance premiums and copays, nutrition support and compensated village-based community health workers) designed to address inequities in access to care.

How were the remarkable improvements in health outcomes made in the regions and socioeconomic groups we assessed? Several previous reports describe the major components of Rwanda’s national health strategy during this period of success.^{22 29} Like Rwanda’s plan to improve health outcomes throughout the country, our intervention was deliberately comprehensive, and it is challenging to disentangle specific components that contributed to the health gains observed. Interestingly, despite the fact that under-five mortality was almost one-third lower in ORAs than it was in Kirehe/S. Kayonza in 2005, the modified CCI of the set of interventions thought to reduce under-five mortality was very similar between the groups. Nonetheless, Kirehe/S. Kayonza initially trailed the ORAs in two interventions most likely to target common causes of death among children in lower-income and middle-income countries: case management of pneumonia and diarrhoea. Importantly, these indicators capture access to curative care rather than preventive measures and may be better proxies for existence of health systems than are interventions such as vaccination or vitamin A supplementation, which can be delivered without investment in the kinds of health system building blocks that were targeted in Kirehe/S. Kayonza. The modified CCI increased similarly in both groups, but Kirehe/S. Kayonza had closed the gap in case management for diarrhoea and achieved higher coverage for exclusive breast feeding and case management of ARI by 2010. In contrast, ORAs outperformed Kirehe/S. Kayonza in measles vaccination, vitamin A supplementation, ANC4 and skilled birth attendants, all of which might be expected to have a less direct impact on overall mortality.

In addition to a focus on equity, the construction or renovation of two district hospitals, and higher coverage of horizontal interventions addressing major causes of child death, the Kirehe/S. Kayonza intervention combined a rigorous programme of compensated village-based community health workers with improvements in infrastructure and staffing of the facilities to which community health workers referred patients for care. The indicators of healthcare coverage assess the frequency, but not the quality, of the care provided, and are therefore coarse tools by which to measure the impact of the range of interventions embodied in the six WHO building blocks. For example, treatment of malaria with an antipyretic rather than artemesin meets the criteria for case management of fever as assessed in the DHS but is unlikely to have a major impact on child mortality. Recognising the challenges of designing an evaluation that captures the relative effects of these components of our care model, we suggest that the impact of the Kirehe/S. Kayonza intervention is partly attributable to the coordinated strengthening of services at multiple levels (community, health centres and hospitals) with a focus on quality of care.³⁰

We note several important limitations to this study. First, mother-reports may be imperfect measures of illness and treatment, especially for such indicators as ARIs including pneumonia and whether the child received antibiotic treatment.³¹ Second, the sample size at baseline was small. Dwyer-Lindgren and colleagues have shown that under-five mortality estimates derived from birth histories with sample sizes under 500 can be biased and usually underestimate the true value of mortality.³² They also show that underestimation of mortality tends to occur when estimates are based on surveys conducted in high mortality settings. With a sample size of only 359 birth records in the 2005 survey of the very high mortality intervention region, it is possible that the true Kirehe/S. Kayonza baseline under-five mortality rate was even higher and the decline between 2005 and 2010 steeper than we estimated.

Third, the cross-sectional nature of the surveys did not allow for tracking changes in individuals, households or communities. Anecdotal evidence suggests that a number of chronically sick individuals and their families from nearby districts moved to Kirehe/S. Kayonza after health system improvements began. The opening of new health facilities and establishment of the PIH-Rwanda headquarters in Kirehe/S. Kayonza also attracted new, better educated employees to the region and stimulated related commerce. The cross-sectional study design does not allow for determining to what extent changes in health were driven by the health system strengthening intervention in the baseline population, compared with other demographic changes in the population that occurred during the intervention period. Fourth, we were only able to measure WHO building block indicators available in the RDHS questionnaire; data about perception or quality of care, for example, could not be included. As in any household questionnaire, selection and recall biases may have also affected estimates.

Finally, the ideal evaluation of this intervention would have involved comparing the 2010 indicators in the intervention group to a population that had identical 2005 health status but which had not received the intervention. As discussed in the Methods section, the intervention site was chosen because it had the highest under-five mortality in the country, and we were thus unable to identify an area with a similarly high rate of under-five mortality. If improvements in child mortality rates are easier to achieve when the baseline levels are high, our finding of a steeper decline in mortality in the intervention group may simply reflect this discrepancy in the baseline rate, rather than any meaningful difference between the intervention and ORAs. However, we find little evidence in the literature to support this ‘low hanging fruit’ hypothesis with most repeated measurements of mortality over time in a specific population demonstrating a linear decline on a log scale.³³

Conclusion

Rwanda experienced historic health improvements between 2005 and 2010, and those improvements were even more pronounced in Kirehe/S. Kayonza where RMOH-PIH rolled out an integrated health system strengthening intervention in 2005. This area had substantially poorer performance on population health outcomes in 2005 but narrowed that gap considerably by 2010. Furthermore, the drop in under-five mortality in Kirehe/S. Kayonza was modestly higher than that of the historic drop experienced nationally and in ORAs. Although we are not able to attribute health improvements in Kirehe/S. Kayonza to the RMOH-PIH programme alone, the RMOH-PIH programme likely played a key role in these monumental health achievements.