Impact of a district-wide health center strengthening intervention on healthcare utilization in rural Rwanda: Use of interrupted time series analysis

Abstract

Background

Evaluations of health systems strengthening (HSS) interventions using observational data are rarely used for causal inference due to limited data availability. Routinely collected national data allow use of quasi-experimental designs such as interrupted time series (ITS). Rwanda has invested in a robust electronic health management information system (HMIS) that captures monthly healthcare utilization data. We used ITS to evaluate impact of an HSS intervention to improve primary health care facility readiness on health service utilization in two rural districts of Rwanda.

Methods

We used controlled ITS analysis to compare changes in healthcare utilization at health centers (HC) that received the intervention (n = 13) to propensity score matched non-intervention health centers in Rwanda (n = 86) from January 2008 to December 2012. HC support included infrastructure renovation, salary support, medical equipment, referral network strengthening, and clinical training. Baseline quarterly mean outpatient visit rates and population density were used to model propensity scores. The intervention began in May 2010 and was implemented over a twelve-month period. We used monthly healthcare utilization data from the national Rwandan HMIS to study changes in the (1) number of facility deliveries per 10,000 women, (2) number of referrals for high risk pregnancy per 100,000 women, and (3) the number of outpatient visits performed per 1,000 catchment population.

Results

PHIT HC experienced significantly higher monthly delivery rates post-HSS during the April-June season than comparison (3.19/10,000, 95% CI: [0.27, 6.10]). In 2010, this represented a 13% relative increase, and in 2011, this represented a 23% relative increase. The post-HSS change in monthly rate of high-risk pregnancies referred increased slightly in intervention compared to control HC (0.03/10,000, 95% CI: [-0.007, 0.06]). There was a small immediate post-HSS increase in outpatient visit rates in intervention compared to control HC (6.64/1,000, 95% CI: [-13.52, 26.81]).

Conclusion

We failed to find strong evidence of post-HSS increases in outpatient visit rates or referral rates at health centers, which could be explained by small sample size and high baseline nation-wide health service coverage. However, our findings demonstrate that high quality routinely collected health facility data combined with ITS can be used for rigorous policy evaluation in resource-limited settings.

Introduction

Health systems strengthening (HSS) interventions have become popular strategies to advance population health gains in low income countries [1–5]. Instead of focusing on disease-specific, or “vertical” programs, HSS interventions improve the platform of health service delivery across all health system components [1, 3]. While the evidence base of successful HSS interventions that improve processes and health outcomes is growing [5, 6], few evaluations have quantified the resulting changes in healthcare utilization, a critical step towards increasing coverage of health services [7, 8]. Understanding the link between investment in systems and uptake can inform health resource allocation and planning efforts in low income settings [9]. However, the cost of collecting novel data to measure the effect of HSS interventions presents a barrier to measuring impact and often results in inadequate evaluation.

An underutilized resource that could be used to address this issue is the wealth of routinely collected service utilization data produced by national health management information systems (HMIS) in low income countries [8, 10]. Analysis using HMIS leverages health systems time series data that are already being used for management and improvement, while allowing for evaluation designs informed by principles of causal inference [11]. A potential reason for limited use of HMIS for HSS evaluations is the perception that data quality is poor [12, 13], despite several positive results following data quality assessments of these systems in low income countries [14–16].

Rwanda, a small, hilly country in East Africa, has embraced the use of HMIS and is well positioned to take advantage of this resource. Rwanda has heavily invested in its Rwanda HMIS (RHMIS), which provides managers and policy makers with healthcare utilization data from all public health facilities in the country. Reports on utilization of outpatient and inpatient services are collected from registers by facility data staff at every public health facility in the country, aggregated at the district level, and are subsequently uploaded to a web-based central repository at the national level [17]. A recent national data quality assessment demonstrated high levels of completeness and internal consistency of RHMIS [15]. An accuracy assessment that sampled facility records across three rural districts found concordance between hard copy reports and electronic RHMIS reports to be 73.3% and concordance between facility registers and electronic RHMIS reports to be 70.6% [18]. Assessments of external validity comparing coverage estimates for family planning and 4 ANC visits in RHMIS to the Demographic and Health household Survey (DHS) conducted by RHMIS analysts were found to be comparable. Institutional delivery estimates were slightly lower in RHMIS compared to DHS in 2010 (57% vs. 69%) [19].

Rwanda has begun to leverage this resource in national evaluations of several health policies, including performance-based financing (PBF) [20], Community Health Worker (CHW) programs [21], and uptake of maternal and reproductive health services following implementation of Human Immunodeficiency Virus (HIV) control programs [22]. These data systems have allowed scientists and policymakers to describe improvements in health outcomes that have occurred in Rwanda over the past decade, which include steep reductions in under-five mortality [23–25], as well as increased coverage of antiretroviral therapy services [26] and maternal health services [27], suggesting improved access to a range of health services across the country [20, 28–31]. While providing important contributions to the health systems literature by documenting health gains in Rwanda, these studies largely relied on cross-sectional or pre-post designs without a control, and so have limited utility for causal inference. Other countries in the region including Uganda and Burundi have also used their national health information systems to assess the impact of national health care financing policies on service utilization [32, 33]. Basinga et al.’s evaluation of PBF relied on a randomized controlled trial design, and Falisse et al. employed a difference-in-difference analysis using two single time points for the intervention and control series in a PBF evaluation in Burundi, but few other analyses using routinely collected national data have leveraged the longitudinal data structure or the ability to sample control series from a national census.

In this manuscript, we evaluate the impact of a district-level HSS intervention implemented by the Rwanda Population Health Implementation and Training (PHIT) partnership in rural Rwanda on district-level health service utilization [34, 35]. Using RHMIS as our data source, we use five years of monthly health center-level time series data to conduct a propensity score matched controlled interrupted time series analysis to estimate the district-level impact of the PHIT HSS intervention on delivery rates, outpatient visits rates, and referral rates for high risk pregnancies. Controlled interrupted time series analysis allows for unbiased estimation of population level effects of an intervention, assuming that no other co-interventions occur at the same time as the primary intervention [36]. This evaluation will inform other global health researchers in Rwanda and elsewhere about the effectiveness of health systems interventions on increasing service uptake, and demonstrate the power of combining national HMIS time series data with counterfactual-based methods to allow causal conclusions to be drawn from HSS impact evaluations in global health.

Methods

Study setting, population, and intervention

Rwanda has a population of 10.5 million, 83% of whom live in rural areas [37]. In 2012, Rwanda had 748 public (government-run) health centers and 174 private health centers, with most care provided at public health centers (85–89%) [15]. The first facility point of contact for patients are public health centers, which provide primary care and maternal and child health services. Each of Rwanda’s 30 districts has roughly fifteen health centers, with the goal of every patient in Rwanda to be living within five kilometers of a health facility. Rwanda’s health system is also served by a network of CHW who administer treatments for childhood illnesses (community integrated management of childhood illness), or refer patients to health centers or hospitals for further care [21]. Health centers in Rwanda provide a minimum package of services that span promotional (child growth monitoring and community health insurance), preventive (vaccination, prenatal and postnatal care) and curative activities (child health care, uncomplicated deliveries, HIV, drug dispensation). These services are offered at all health centers in the country [31]. The CHW program was implemented between 2008 and 2011, and overlapped with the implementation of our HSS intervention.

In 2009, Partners In Health (PIH) and the Government of Rwanda (GOR) entered a partnership to implement a five-year district-wide HSS intervention in two rural districts of Rwanda serving a catchment population of 480,000, which lagged behind the rest of the country in terms of health and social indicators [27, 34]. These districts were chosen because PIH was already supporting the GOR through provision of technical and financial support to two district hospitals and seven health centers in these districts prior to the PHIT intervention. For this analysis, we use ITS to study the impact of the first component intervention that began in May 2010 and consisted of targeted instrumental support to PHIT health centers. This included a data-driven gap analysis to assess facility readiness among the fourteen health centers in the intervention area that had not received partnership support prior to the HSS intervention. Facility surveys were developed to ascertain dimensions of facility readiness, guided by the World Health Organization (WHO) health systems building block framework [9] and based on the Services Provision Assessment [38] and the annual health facility survey which was in use nationally at the start of the intervention. Core domains included infrastructure, human resources, monitoring and evaluation, and supplies with an emphasis on data utilization for decision-making. Partnership representatives met with health facility leadership to discuss prioritization of resource allocation based on survey results. Based on review of the results and these discussions, limited funds (average: 18,000 USD/health center) were transferred to intervention facilities to address prioritized gaps. Specific areas of focus varied by facility and ranged from investments in health center management, infrastructure renovations, medical equipment, salary support for additional health center staff, and social support for vulnerable patients resulting in overall improvement of facility readiness to provide care [35]. Following this work, additional district-wide interventions focused on further strengthening facility and CHW care quality and service utilization in the intervention area [39, 40].

Study design

We developed a conceptual framework describing the pathway from intervention to increased uptake of services among people living in the intervention catchment area (Fig 1). We hypothesized that the improved facility readiness to provide high quality care through strengthened infrastructure, supplies, staff and information systems would be recognized by community members who would be more likely to seek care at the intervention facilities and recommend use by others [35]. This framework was supported by research on factors associated with care seeking behavior have shown that availability of equipment and medicines leads to increased utilization [41], and that perceived poor quality of facilities can lead to reduced care seeking by pregnant women [42].

Fig 1. Rwanda Population Health Implementation and Training health systems strengthening evaluation conceptual framework.

We framed our causal question as follows: what is the difference in level and monthly trend in mean health service utilization rates comparing observed rates from PHIT-supported health centers to the rates they would have had if they had not received the HSS intervention? Controlled interrupted time series operationalizes this question by modeling the counterfactual using segmented linear regression and the post-intervention level and trend from a control series [43].

There were a total of 21 health centers in the intervention districts, seven of which had received support from PIH prior to the PHIT HSS intervention. These seven health centers were excluded because we were interested in the impact of PHIT HSS on health centers which had never received any non-governmental support, leaving us with 14 PHIT intervention health centers [35]. Our control series contained 409 government-run health centers that were also providing services. We sought to limit our control series to public health centers because the staff, infrastructure and equipment at private health centers would not reflect those available at the government-run intervention health centers prior to HSS. We excluded 29 health centers due to incomplete reporting over the study period, and the remaining 380 (93%) were eligible for matching.

Data source

RHMIS data were migrated from a SQL-server database to a DHIS2 platform in 2011. We merged data from the two databases to generate an analysis dataset spanning January 2008 through December 2012.

Outcomes

Utilization of maternal health and outpatient visits were chosen a priori because most of the investments were directed at these services, through investments in improved maternity infrastructure, clinical mentoring and essential medicines and equipment for maternal and newborn care [35]. Using the RHMIS data, we constructed a dataset containing variables on maternal health (new antenatal care registrations, women with 4 standard antenatal care visits, facility deliveries, referrals for high risk pregnancy, family planning) outpatient visits, and child care (diphtheria-pertussis-tetanus (DPT) DPT1, DPT3, and BCG vaccination visits).

We converted these metrics to rates using population estimates from the Ministry of Health [37] and measured differences in the monthly number of facility deliveries per 1,000 women, the monthly number of referrals for high risk pregnancy per 10,000 women, monthly number of outpatient (OPD) visits per 1,000 catchment population, monthly number of DPT1 vaccination visits per 1,000 children 0–11 months, monthly number of DPT3 vaccination visits per 1,000 children 0–11 months, and monthly number of BCG vaccination visits per 1,000 children 0–11 months between the intervention health centers and propensity score matched health centers. Though our outcomes do not incorporate person-time, we refer to them as rates for succinctness. For analysis, monthly rates were aggregated by intervention group (PHIT versus propensity score matched non-intervention series).

Statistical analysis

To account for differences in health center characteristics across intervention and control groups and select control facilities that were similar to intervention facilities at baseline with respect to health center characteristics, propensity scores were derived for each health center using multiple logistic regression models [44]. We limited the number of covariates included in the model due to the small number of intervention health centers (n = 14). We hypothesized that population density, outpatient visit rates, and delivery rates would be associated with success of the PHIT intervention and so we wanted to match intervention health centers to control health centers with similar characteristics at baseline. We chose to restrict the baseline period from January 2008 to December 2009 to provide two full years, and eligible control health centers to those that had no more than four missing HMIS reports during the baseline period. We failed to find evidence of association between delivery rates and treatment after adjusting for population density and outpatient rates, so our final model included population density (continuous, number of people per square kilometer in a catchment area sector), and four covariates that summarized the monthly average outpatient rate at each health center over six month intervals (January 2008-June-2008, July 2008-December 2008, January 2009-June 2009, July 2009-December 2009). To improve precision, each intervention health center was matched to up to ten control health centers, using a caliper match of ±0.05 propensity score units [45, 46]. We assessed the performance of the propensity score match by comparing the standardized difference in health service utilization between intervention and control health centers before and after applying the match [47, 48].

We used controlled interrupted time series analysis to study trends in healthcare utilization variables in the intervention group relative to the comparison group. We used the intervention date of May 2010 and fit time series models to test whether differences in changes in level or trend in indicators were statistically significant between the two groups. Controlled interrupted time series produces two main results of interest. The first is the difference in post-implementation change in level of mean outcome in the intervention relative to the control group, and the second is the difference in post-implementation trend in outcome in the intervention relative to the control group. These results are beta coefficient estimates produced by the time series models. For our study, we interpret these beta coefficients as 1) the difference in mean health service utilization rate from the pre-intervention to post-intervention period comparing the intervention facilities to control facilities, and 2) the difference in monthly change in service utilization rate from pre-intervention to post-intervention period comparing intervention facilities to control facilities. We plotted our results using fitted line segments to visualize these pre and post-intervention changes in level and trend by intervention group. The trend coefficient in our model allowed us to estimate any changes that would arise in our intervention over time, such as delayed improvements following the initial investments into the system.

We used generalized least squares (GLS) models including autocorrelation terms for both moving average or autoregressive processes that were assessed independently for each outcome as described by Wagner et al. [43]. Lag terms were determined using Durbin-Watson tests and autocorrelation and partial autocorrelation plots. We used a GLS model with an autocorrelation lag term of 4 and a moving average lag term of 4 to analyze trends in delivery rates. For OPD, we used a GLS model with a moving average lag term of 1. Finally, we used a GLS model with an autocorrelation lag term of 1 and a moving average lag term of 1 for analysis of rates of referral for high risk pregnancy. We chose to control for seasonality by including a seasonal dummy variable corresponding to three month quarters (January to March, April to June, July—September, and October to December) for facility deliveries and OPD [49]. This seasonality dummy was not included in the model for high risk pregnancy because we failed to find any statistically significant associations between seasonality and trends in this variable. We also conducted post hoc tests for effect modification of the post-implementation effect on healthcare utilization on the multiplicative scale by season in the April-June quarter using a multiplicative interaction term. Analysis was conducted SAS v. 9.4 (propensity score matching) and R v. 3.1.0 (controlled interrupted time series) and. Full details regarding the statistical models are provided in the supplementary appendix (S1 File). Statistical significance was assessed at the 0.05 level, and results are presented as point estimates and 95% confidence intervals (95% CI).

Results

Baseline characteristics

We present baseline characteristics of the intervention and control facilities in Table 1. Intervention facilities covered a much smaller total population than control facilities (254,656 v. 8,474,422) and were distributed over much smaller geographic area. Intervention facilities had slightly fewer monthly new ANC registrations on average during the baseline period of 2008–2009 compared to comparison facilities (median: 44 v. 62), and also fewer monthly outpatient visits (median: 945 v. 1326) compared to comparison facilities. Monthly facility deliveries over the baseline period were similar for both groups. Both intervention and comparison facilities had few referrals for high-risk pregnancies.

Table 1. Baseline health center characteristics of intervention and control facilities.

	Pre-match						Post-match
	PHIT Intervention		Eligible Control Health Centers		Absolute difference	Standardized difference	PHIT Intervention		Propensity Score Matched Controls		Absolute difference	Standardized difference
Total catchment population	254,656		8,474,422				238,693		1,675,011
Districts	2		29				2		25
Number of facilities	14		380				13		86
Population density (people/km2) (median, IQR)	379	[279–408]	495	[392–600]	-116	-1.06	315	[279–379]	390	[307–478]	-75	-0.42
Baseline utilization (2008–2009), median, [IQR*]
Monthly new ANC registrations	44	[25–65]	62	[41–91]	-18	-0.59	46	[25–67]	54	[37–75]	-8	0.43
Monthly facility-based deliveries	26	[17–41]	32	[20–45]	-6	-0.26	28.75	[17–43]	28.5	[18–40]	0.25	-0.11
Monthly referrals for high risk pregnancies	0	[0–1]	1	[0–3]	-1	-0.56	0	[0–1]	1	[0–3]	-1	-0.08
Monthly outpatient visits	945	[596–1,325]	1,326	[875–1,964]	-381	-0.64	894	[569.5–1301.5]	1090	[741–1458]	-196	-1.81

*IQR = interquartile range

Propensity score matching resulted in 13 intervention health centers matched to 86 control health centers on population density and monthly outpatient visits over four six-month intervals between January 2008 –December 2009. Parameter estimates and kernel density plots showing results of the matching are presented in the supplementary content (S1 Fig, S1 Table). Propensity score matching yielded improved balance with respect to facility-based deliveries (standardized difference post-matching compared to pre-matching: -0.11 v. -0.26), population density (standardized difference post-matching compared to pre-matching: -0.42 v. -1.06), and referrals for high risk pregnancy (standardized difference comparing post- to pre-matching: -0.08 v. -0.56). The standardized difference for outpatient visits increased (-1.81 v. -0.64), though the absolute difference between PHIT and control facilities decreased (-196 v. -381) after matching.

Facility-based deliveries

As shown in Fig 2, the average monthly facility-based delivery rate was higher (6.61/10,000, 95% CI: [1.54, 11.67]), and the trend in monthly facility-based delivery rate was declining (-0.21/10,000, 95% CI: [-0.51, 0.08]) in PHIT facilities relative to comparison facilities over the period prior to HSS. After implementation of HSS, there was a small immediate increase in the level (0.67/10,000, 95% CI: [-5.40, 6.74]) and trend in monthly facility delivery rates (0.21/10,000, 95% CI: [-0.19, 0.62]). Following HSS, PHIT facilities experienced significantly higher monthly delivery rates during the April-June season than comparison facilities (3.19/10,000, 95% CI: [0.27, 6.10]). In 2010, this represented an additional 4.1 deliveries/10,000 women in PHIT health centers (13% relative increase) compared to the rates in comparison health centers, and in 2011, this represented an additional 6.8 deliveries/10,000 women in PHIT compared to non-PHIT facilities (23% relative increase). Parameter estimates are reported in S2 Table.

Fig 2. Monthly facility delivery rates (count per 10,000 women in the population) in PHIT health facilities (n = 13) or comparison propensity score matched facilities (n = 86).

Notes: Vertical line indicates start of PHIT HSS intervention in May 2010.

Referrals for high-risk pregnancies

Over the period prior to HSS, the mean rate of high-risk pregnancy referrals at intervention facilities was lower than at control facilities (-0.82/10,000, 95% CI: [-1.26, -0.39]) (Fig 3). Mean referral rates at intervention facilities were 0.035/10,000 higher than the rates in comparison facilities in the period after HSS implementation began (95% CI: [-0.51, 0.58]). The trend in monthly rate of high-risk pregnancies referred increased slightly in intervention facilities compared to the control following the start of the intervention (0.027/10,000, 95% CI: [-0.007, 0.06]). We also found increasing trend in monthly referral rates (0.024/10,000, 95% CI: [-0.0002, 0.049]) in the control facilities following implementation of our HSS intervention. Parameter estimates are reported in S3 Table.

Fig 3. Monthly rates of referral for high risk pregnancy (count per 10,000 women in the population) in PHIT health facilities (n = 13) or comparison propensity score matched facilities (n = 86).

Notes: Vertical line indicates start of PHIT HSS intervention in May 2010.

Outpatient visits

At baseline, the average outpatient visit rate was 26% lower in PHIT health centers compared to comparison health centers (13.3/1,000 people, 95% CI: [-29.1, 2.6]). There were no major differences in level or trend in mean monthly outpatient visit rates in PHIT health centers relative to comparison facilities following implementation of HSS (Fig 4). While there was an immediate 27% increase (18.0 visits per 1,000 people per month) in the rate of outpatient visits in intervention facilities compared to control facilities in the period following HSS, (6.64/1,000, 95% CI: [-13.52, 26.81]), this did not reach statistical significance. Over time there was a decline in the trend in monthly mean outpatient visit rate in the comparison facilities (-0.81/1,000, 95% CI: [-1.69, 0.08]). There was a slight increasing trend in monthly mean outpatient rates following implementation of HSS in the intervention facilities compared to the comparison facilities, but this was not statistically significant (0.073/1,000, 95% CI: [-1.18, 1.32]). Parameter estimates are reported in S4 Table.

Fig 4. Monthly rates of outpatient visits (count per 1,000 catchment population) in PHIT health facilities (n = 13) or comparison propensity score matched facilities (n = 86).

Notes: Vertical line indicates start of PHIT HSS intervention in May 2010.

Other indicators

We found no significant differences in level or trend for rates of childhood vaccinations (DTP1, DTP3, BCG), new antenatal care registrations, or women with 4 standard antenatal care visits in the intervention facilities relative to comparison facilities (S5, S6, S7, S8 and S9 Tables).

Discussion

We present results from one of the first evaluations of an HSS intervention on health service utilization conducted using controlled interrupted time series and routinely collected HMIS data. Our findings showed that the impact of the PHIT HSS intervention on service utilization was constrained to facility-based delivery rates. We failed to find evidence that the HSS intervention resulted in increases in monthly outpatient visit rates or rates of referral for high risk pregnancy over and above those found in comparison facilities. For deliveries, we found that the increase in rates following the PHIT HSS intervention was restricted from the April to June season. For outpatient visit rates, even though we did not find evidence of a significant increase in level or trend in PHIT compared to control health centers following HSS, the level and trend in PHIT health centers remained higher than in control over the post-implementation period despite having lower rates at baseline.

Facility-based deliveries were declining in the intervention health centers prior to HSS, with significant increase in mean delivery rates during summer months over the implementation period. Prior to the intervention, PIH had provided support to the two district hospitals and only seven health centers (excluded from the intervention group) [35]. One possible explanation of the decline in the pre-HSS period was patients choosing to go to health centers that were already receiving non-governmental supports within the intervention districts over their local health centers. There is growing evidence that women will indeed bypass facilities which they perceive as providing worse quality of care [42]. It is possible that positive messages about improved quality following the intervention combined with better access to health centers for deliveries during April-June may have driven the increase in delivery rates. The April-June season overlaps with the end of the short rainy season and beginning of the long dry season in Rwanda, during which roads improve and therefore accessibility to health centers increases.

Following the health center strengthening intervention, implementation of a quality of care intervention began that could also have led to increased utilization of maternal health services began in May 2011 [40]. Improving service readiness at multiple facilities in the intervention area would have allowed more women to access health services closer to their homes—a predictor that has found to be associated with facility-based deliveries in other studies in Rwanda [50]. While previous studies have shown that health facility-level improvements focused just on supplies and other readiness factors do not always lead to increases in utilization in other developing countries [51–53], we demonstrate that in Rwanda, these improvements could have led to increased utilization for maternal health services. We attribute these increases to a comprehensive approach to HSS—our intervention was guided by all six of the WHO Building Blocks and included a strong focus on the quality of care delivered to address multiple components critical to responsive health service delivery [34, 35].

We did not find significant increases in referral rates in intervention health centers compared to control health centers following HSS. Since our health center strengthening intervention also included the provision of ambulances and materials to strengthen communication between health centers and hospitals, we hypothesized that these improvements could have led to increases in referrals for high-risk pregnancies. However, our integrated approach to improving health center readiness through infrastructure renovations, availability of equipment and supplies, staffing and social support could have also decreased barriers to care at intervention facilities. Increased receipt of maternal care at intervention health centers could have led to reductions in management of complicated deliveries through better care and skills. Our finding of suggestive trends in control areas post-HSS suggests that some other national co-intervention may have occurred concurrently with the PHIT intervention, thus mitigating our ability to detect additional improvement.

We failed to find significant increases in outpatient visit rates in intervention relative to the comparison health centers. Our intervention occurred during a period of rapid change in Rwanda’s health system [24, 25]. Many national policies to expand access to facility-based care and decentralize decision-making power were introduced in the years preceding the intervention and during the intervention [54]. We saw suggestion of decreasing trends in monthly outpatient visit rates in control health centers which suggests that this roll-out of community-based treatment of children under-five may have decreased use of services at facilities in the intervention and comparison areas. This hypothesis is supported by a recent analysis of community-based integrated community case management of childhood illness in Rwanda found decreases in facility utilization of under-5 services over the period of implementation (between January 2010 and December 2011) [21]. CHW strengthening was an explicit component of our broader health system intervention and may have been implemented with greater intensity in our districts compared to others, yielding differing patterns of healthcare utilization with regards to outpatient visits [34]. Given that implementation of this national policy occurred alongside our intervention, it is possible that it may have attenuated increases in facility-based outpatient visits during our study period.

The impact of various national policies to increase demand for maternal and child health services in Rwanda has been described in the literature [20, 28–31]. Researchers have argued that the collective impact of decentralization of health policy decision-making, community-based health insurance and the introduction of PBF have led to higher quality of services and increased utilization of public health services across the country, though only the PBF evaluation used a randomized controlled trial design. Further analyses have suggested that the benefits of these interventions have allowed the poorest patients in Rwanda to increasingly access health services over time [20, 30], meaning that the additional improvement attributable to the PHIT HSS intervention would be challenging to measure. We did not see any differences in healthcare utilization at intervention compared to non-intervention health centers across a number of services, many of which started at very high coverage rates prior to implementation of our HSS intervention. Antenatal care coverage and vaccination coverage were roughly 90% nation-wide prior to implementation of our HSS intervention [27], leaving little room for improvement over the study period.

Few evaluations exist to estimate the impact of HSS interventions to improve district facility readiness on service utilization, and fewer still are conducted using designs that allow causal effects to be estimated. Our study provides findings from an evaluation of such an intervention and shows that even in the context of national policies aimed at increasing utilization, facility deliveries in intervention health centers increased relative to control health centers in specific seasons. We would expect that in countries where access to health services is more limited and fewer national health financing schemes are in place to encourage demand, a similar intervention could yield greater increases in healthcare utilization. Health systems researchers in northern Uganda found increases in facility deliveries following a combined community-level and health facility-level intervention that occurred between January 2010 and September 2011 to improve quality of services, though they presented results from their intervention area only [55].

Our study had several limitations. First, while our data source has been shown to have strong internal validity [15], full assessments of reliability and accuracy of RHMIS have yet to be conducted. However, preliminary assessments of external validity comparing RHMIS to DHS estimates for coverage indicators show little discordance [19]. Furthermore, in order to bias our results, RHMIS data quality would have had to have changed differentially at the same time as the HSS intervention between intervention and control areas. The electronic system transitioned from a French SQL-based database to an English DHIS2 database in 2012, which limited variables to those that were consistent across systems, though this affected both intervention and control facilities. We were limited in our ability to match our intervention health centers to control health centers using propensity scores due to small sample size for our intervention group and lack of detailed information on health center characteristics in all public health centers in Rwanda. Further limitations of this analysis include a lack of data on contextual factors associated with health centers that could help explain these findings. Use of geospatial analysis and contextual information at the health center and district level would provide additional information on possible mechanisms through which the intervention was successful or not. Future studies should account for district-level and health center-level interventions in the design, and power their studies to estimate effects at each level. These results are restricted to public health centers; since most care in Rwanda is provided at public facilities, we expect that bias due to differential increases and use of private health centers post-HSS is limited. Our quasi-experimental, longitudinal design controls for the impact of any national co-interventions that were rolled out across the country during the study period, strengthening the validity of our findings. Our HSS intervention was heterogeneous, with allocation of funds towards strengthening health systems building blocks tailored to specific needs of different health centers, making it difficult to attribute impact to a specific intervention component. Since we used aggregate data for our analysis, we cannot assess the impact of the intervention on individual patients to determine whether we successfully targeted the most vulnerable in our catchment populations.

Strengths of this analysis include use of five years of monthly time series HMIS data to allow modeling of counterfactuals following our HSS intervention, and use of propensity score matching to simulate exchangeability of intervention and comparison facilities by matching on baseline trend and health center covariates. Since the HSS intervention start date is clear, and control time series were included to account for other co-interventions that might influence utilization, we are unlikely to have introduced systematic bias in our analysis.

In summary, we evaluated the impact of a heterogeneous HSS intervention on health service utilization in rural Rwanda using routinely collected HMIS data. We found that facility delivery rates increased post-HSS in intervention compared to non-intervention health centers, though this increase was restricted to the April-June quarter in years following HSS, though changes in other patterns of utilization were limited. This example demonstrates how routinely collected HMIS data in Sub-Saharan Africa can be used for quasi-experimental evaluation designs. Time series data allow researchers to move away from pre-post and cross-sectional evaluation designs to quasi-experimental designs using counterfactuals. In addition to informing program implementation and providing useful estimates of impact for health providers, such analyses also encourage strengthened information systems and national engagement in policy research [8]. This manuscript involved collaboration between academic, non-governmental and governmental scientists, and was supported through training funds to develop capacity in Rwanda to study national policies using interrupted time series analysis and HMIS data. We hope other implementers and policy researchers in low and middle income countries will be encouraged by such studies to use their national data sources for implementation and policy research. Further studies using these methods are needed to estimate effects of HSS interventions have on utilization and population health outcomes.

Supporting information

S1 File. Supplementary statistical appendix.

(DOCX)

S1 Table. Results of logistic regression model to estimate propensity scores among intervention (n = 14) and control (n = 380) health centers in Rwanda.

(DOCX)

S2 Table. Parameter estimates for monthly delivery rate generalized least squares model following a health center strengthening intervention in 99 propensity score matched health centers from January 2008 to December 2012.

(DOCX)

S3 Table. Parameter estimates for monthly high risk pregnancy referral rate generalized least squares model following a health center strengthening intervention in 99 propensity score matched health centers from January 2008 to December 2012.

(DOCX)

S4 Table. Parameter estimates for monthly outpatient visit rate generalized least squares model following a health center strengthening intervention in 99 propensity score matched health centers from January 2008 to December 2012.

(DOCX)

S5 Table. Parameter estimates for monthly 1^st ANC registration rate generalized least squares model following a health center strengthening intervention in 99 propensity score matched health centers from January 2008 to December 2012.

(DOCX)

S6 Table. Parameter estimates for monthly 4 standard ANC visits rate generalized least squares model following a health center strengthening intervention in 99 propensity score matched health centers from January 2008 to December 2012.

(DOCX)

S7 Table. Parameter estimates for monthly BCG vaccination rate generalized least squares model following a health center strengthening intervention in 99 propensity score matched health centers from January 2008 to December 2012.

(DOCX)

S8 Table. Parameter estimates for monthly DTP1 vaccination rate generalized least squares model following a health center strengthening intervention in 99 propensity score matched health centers from January 2008 to December 2012.

(DOCX)

S9 Table. Parameter estimates for DTP3 vaccination rate generalized least squares model following a health center strengthening intervention in 99 propensity score matched health centers from January 2008 to December 2012.

(DOCX)

S1 Fig. Kernel density plot for propensity scores in PHIT matching analysis.

(TIF)

Data Availability

We have conducted our study using data from Rwanda's national health management information system. The Government of Rwanda's Ministry of Health has strict policies on sharing government data that prevent us from sharing these data publicly. The Rwandan Government Ministry of Health website has a form that can be accessed at http://www.moh.gov.rw/index.php?id=147. Interested parties can download the form and follow instructions to request access to the data used for this study to verify results. However, this will be at the discretion of the Rwandan Government.

Funding Statement

We gratefully acknowledge the financial support from the Doris Duke Charitable Foundation’s African Health Initiative, through PHIT Partnership funding for implementation of this study (Grant number: 2009P001941). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. HSI was supported in part by the National Institutes of Health research training grant (NIH, T32 CA 009001).