Abstract
Objective
To evaluate the influence of the early phase of Project Fives Alive!, a national child survival improvement project, on key maternal and child health outcomes.
Design
The evaluation used multivariable interrupted time series analyses to determine whether change categories tested were associated with improvements in the outcomes of interest.
Participants
The evaluation used program and outcome data from interventions focused on health-care staff in 27 facilities.
Setting
Northern Ghana.
Intervention
The project uses a quality improvement (QI) approach whereby process failures are identified by health staff and process changes are tested in the health facilities and corresponding communities to address those failures.
Main Outcome Measures
The maternal health outcomes were early antenatal care attendance and skilled delivery, and the child health outcomes were underweight infants attending child wellness clinics, facility-level neonatal mortality and facility-level infant mortality.
Results
Postnatal care changes for the first 1–2 days of life (β= 0.10, P = 0.07) and the first 6–7 days of life (β = 0.10, P = 0.07) were associated with a higher rate of visits by underweight infants to child wellness clinics. There was an association between the early pregnancy identification change category with increased skilled delivery (β = 1.36 P = 0.07). In addition, a greater number of change categories tested was associated with increased skilled delivery (β = 0.05, P = 0.01).
Conclusion
The QI approach of testing and implementing simple and low cost locally inspired changes has the potential to lead to improved health outcomes at scale both in Ghana and other low- and middle-income countries.
Keywords: quality improvement, impact evaluation, time series analysis, maternal and child health, mortality, Ghana
Introduction
Quality improvement (QI) methods, defined as strategies to improve the delivery of effective interventions, have long been used in high-income countries to improve health care and outcomes [1–3], but their application to middle- and low-income countries has been more recent. In 2007 the World Health Organization identified quality as a key component of improved health outcomes and greater efficiency in health-care service delivery [4]. As more countries adopt QI approaches there is a need to document their implementation and effectiveness [5–6]. Most of the currently published literature on assessments of QI approaches in middle- and low-income countries has focused on determining changes in process indicators, perceptions of change or improvements in hospital management [7–14]. While understanding the implementation of QI approaches is crucial, so is evaluating their impact on health outcomes. This paper presents an evaluation of the first phase of a large QI project in Ghana.
Project Fives Alive! is a QI intervention implemented by the National Catholic Health Service of Ghana and the Institute for Healthcare Improvement (IHI) in close collaboration with the Ghana Health Service (GHS). Project Fives Alive! began in July 2008 with a pilot phase and will scale up to all public and faith-based health facilities in Ghana before the project end date of March 2015. The project aligns itself closely with the High Impact Rapid Delivery (HIRD) program for maternal and child health, a national program launched by the GHS in 2006 [15]. The HIRD program is focused on delivering low-cost maternal and child health and nutrition interventions nationwide. Using QI methods and tools, the project aims to improve health outcomes in mothers, infants and children under-five by improving the coverage, quality, reliability and patient centeredness of the HIRD program across all public and faith-based facilities in Ghana. Thus, the project aims to assist and accelerate Ghana's national effort to reach both Millennium Development Goal 4 (a two-third reduction in under-five mortality from 1990 to 2015) and Millennium Development Goal 5 (a three-quarter reduction in maternal mortality from 1990 to 2015). In 2008, under-five mortality was estimated to be 80/1000 live births [16] and maternal mortality was estimated to be 350/100,000 live births [17].
The project's QI theory is based upon the model for improvement [18] whereby process failures are identified, and simple and low-cost change ideas are tested in the facilities and the communities which they serve. The improvement approach emphasizes systems thinking, analysis and learning from data at the local level [1, 18–20]. The project incorporates the IHI's Collaborative Model for Achieving Breakthrough Performance [21] whereby health staff and management teams within a district are brought together to form an Improvement Collaborative Network (ICN). Within an ICN each facility forms a QI team which is responsible for overseeing the development and testing of change ideas. Members from each facility's QI team attend four learning sessions, structured workshops led by Project Fives Alive! staff, every 4 months to learn QI methods and to share progress with other QI teams. Another key aspect of the approach is coaching visits to the health facilities made by project staff in conjunction with district health supervisors. These coaching visits take place during activity periods, the 4-month long periods following each learning session. A detailed description of the project's methodology and implementation strategy is presented by Twun-Danso et al. [22].
This paper is focused on evaluating the pilot phase of Project Fives Alive! from July 2008 to December 2009, which included 27 facilities in 4 largely rural districts/dioceses in Northern Ghana. The particular districts were chosen because they included an even mix of government and Catholic facilities. The facilities included 25 health centers (staffed by midwives, nurses and other health staff but not doctors) and 2 hospitals, which provided comprehensive emergency obstetric and neonatal care. This phase of the project was intended to identify a package of locally tested, successful change ideas that could be rapidly scaled up nationally in the later phases of the project.
Methods
Data
Outcome data
We include two outcomes focused on maternal health—early antenatal care (ANC) (percent of first-time ANC registrants who are in their first trimester of pregnancy) and skilled delivery (percent of total deliveries which are attended by a skilled birth attendant defined as a doctor, nurse or midwife). While increased access to skilled delivery has not been universally linked to improved maternal mortality [23], the promotion of skilled delivery is widely regarded as a key strategy for maternal health programs [24].
Three child health impact indicators were studied as key outcomes—underweight among infants (percent of infants attending child wellness clinics who are low weight for age), facility-level neonatal mortality defined as deaths <28 days of life (facility-level neonatal deaths/facility and community-level live births) and facility-level infant mortality defined as deaths <1 year of life (facility-level infant deaths/facility and community-level live births).
These outcome indicators were obtained by Project Fives Alive! from facility health registers. Health-care workers report on a number of indicators directly into this registers. These indicators are included in routinely reported data sent by facilities to the GHS on a monthly basis.
Change categories
The key independent variable for the evaluation is the type of process change implemented. Change ideas fell into five categories: early pregnancy identification, the promotion of four ANC visits, encouraging skilled delivery and providing postnatal care (PNC) on Day 1–2 and PNC on Day 6–7 of life. Table 1 presents detailed information about the change categories.
Table 1.
Description of the change categories and numbers of facilities testing each change
| Change category | Examples of change ideas | Number of facilities that implemented the change |
|---|---|---|
| Early pregnancy identification | Community stakeholder meetings on the importance of early ANC; registration of pregnant women by community volunteers | 7 |
| Four ANC visits | Increase number of days ANC is offered at the facility; reduce visit duration time; offer ANC as an outreach service; ANC defaulter tracing | 8 |
| Skilled delivery | Consistent use of partographs; provision of midwife's phone number to pregnant women; transport provided to laboring women | 20 |
| PNC Day 1–2 | Detain post partum women for 6–48 h after delivery during which time PNC is provided; if beds/wards are full do a PNC Day 1 home visit; home PNC visits for home deliveries | 21 |
| PNC Day 6–7 | Encourage mothers to make a follow PNC Day 6–7 visit; PNC Day 6–7 home visit; defaulter tracing | 19 |
Program and facility-level characteristics
Descriptive data on facility and program-level factors are presented in Table 2. The facility-level variables included affiliation (government or Catholic), type (hospital or health center) and number of staff. The program level factors included a dummy variable for the individual project officer assigned to work in specific facilities, number of site visits and profession of the QI team leader (midwife versus non-midwife). Facility catchment population and a measure of remoteness (distance from the facility to the district capital) were also included to capture facility-level heterogeneity. The program and facility-level information was collected from program records, census data and through discussions with project staff.
Table 2.
Program and facility-level characteristics
| Variable | n | Percent or mean | Range (if applicable) |
|---|---|---|---|
| Government affiliated (versus Catholic) | 13 | 48.1 | — |
| Health center (versus hospital) | 25 | 92.6 | — |
| Mean number of staff | 27 | 9.0 | 2–20 |
| Mean number of site visits | 27 | 14.6 | 9–18 |
| Midwife team leader (versus non midwife) | 18 | 66.7 | — |
| Project officer | 27 | — | |
| Project officer 1 | 6 | 22.2 | |
| Project officer 2 | 14 | 51.9 | |
| Project officer 3 | 7 | 25.9 | |
| Catchment population (per 1000) | 27 | 9.3 | 2.226.9 |
| Remoteness: distance from health facility to district capital (per 10 km) | 27 | 2.7 | 0.1–14.6 |
Analysis
Data from non-intervention comparison facilities were not available. However, rather than compare pre- and post-intervention means we employ an interrupted time series approach whereby monthly facility-level data were included in a multivariable analysis. Time series analysis is used to detect whether an intervention (or change category tested) is associated with a change in an underlying trend for an outcome variable [25]. Data for the evaluation came from the period of April 2008 to December 2009, and the project did not reach full implementation until January 2009. It was, thus, possible to establish an underlying trend using 9 months of pre-intervention data. (January 2009 is the month after the activity period following the second learning session. The project team has indicated this is the time most QI teams were fully implementing change ideas.) In this analysis, each facility serves as its own control because the pre-change trend is compared with the post-change trend.
The data constituted a time series of monthly cross sections, and the core equation to be estimated was as follows:
 |
(1) |
In this specification Y is the outcome of interest, f and t denote facility and time period, respectively, and CT represents a change category tested. CTf takes on the value 1 for a time period after full implementation of a change and 0 before (the ‘interruption’) in facility f, and X is a vector of facility and program-level variables included in the model. β2 indicates the immediate impact and β3 indicates the longer term impact or trend. The test of statistical significance of β2 determines whether there is a one-time jump in the value of Y at full implementation of the change, while β3 determines whether there is a change in the slope of Y after full implementation of the change (the difference in slope from before the change was fully tested to after). If the coefficient of the change variable is positive then there is a one-time positive jump in the value of the outcome at the time the change was fully implemented (a difference in intercepts between the pre- and post-change lines). If the coefficient of the change variable is negative then there is a one-time negative jump. A negative coefficient for the interaction with time trend indicates that the post-change slope is flatter than the pre-change slope and suggests that the effect of the intervention on the trend is negative. A positive coefficient for the interaction with time trend indicates that the effect on the trend is positive. The actual post-change value of the outcome is given by the sum of the coefficients of the change variable (β2) and time trend (β3). In the regression models a non-linear trend is accommodated through a quadratic term in t multiplied by the change variable. Each change category was included in a separate multivariable time series regression model.
Expected influence of changes on the outcomes
Most change categories were expected to have a primary or direct influence on an outcome while a few would have a secondary or indirect influence. For example, the skilled delivery change category would be expected to have a primary influence on neonatal mortality but not a direct influence on the percent of underweight infants. There could however be some indirect or secondary influences. For example, by delivering her child in the health facility, a woman could become more aware of health services for children in that facility and bring an underweight child for care. However, because of the more distal nature of the secondary influences, only change categories expected to have a primary influence on an outcome were tested in the statistical models.
Results
Descriptive statistics on the program and facility-level characteristics are presented in Table 2. The mean number of staff was 9, while the mean number of site visits was 14.6. There were 3 project officers, and 67% of the QI team leaders were midwives with the remaining leaders being nurses, medical assistants and field technicians.
Table 3 presents the pre-intervention, post-full intervention and overall means of the outcome variables. Notably, neonatal mortality decreased from a mean of 2.5/1000 to 0.9/1000, and infant mortality decreased from a mean of 3.5/1000 to 2.3/1000 from the pre-intervention to post-intervention periods. The mean for skilled delivery increased from 55.9 to 64.7%. The unit of observation for the outcome data is facility-months (number of facilities × number of months of data). The number of facility-months varies for each outcome largely because some facilities do not report on all indicators, and the early ANC change was introduced to the facility reporting systems later than the other indicators.
Table 3.
Pre-full intervention, post-full intervention and overall means of the outcome variables
| Outcomes | Pre-intervention |
Post-intervention |
Overall |
||||||
|---|---|---|---|---|---|---|---|---|---|
| Facility-months | Mean | Range | Facility-months | Mean | Range | Facility-months | Mean | Range | |
| Early ANC (%) | 54 | 33.9 | 0–100 | 308 | 32.9 | 0–100 | 362 | 33.0 | 0.0–100 |
| Skilled delivery (%) | 150 | 55.9 | 0–100 | 272 | 64.7 | 0–100 | 422 | 61.6 | 0–100 |
| Underweight in infants (%) | 244 | 1.8 | 0–38.8 | 305 | 1.2 | 0–33.3 | 549 | 1.5 | 0.0–3.9 |
| Neonatal mortality rate | 145 | 2.5/1000 | 0/1000–117.6/1000 | 266 | 0.9/1000 | 0/1000–54.1/1000 | 411 | 1.5/1000 | 0.000–0.018 |
| Infant mortality rate | 145 | 3.5/1000 | 0/1000–117/6/1000 | 257 | 2.3/1000 | 0/1000–78.4/1000 | 402 | 2.8/1000 | 0/1000–117.6/1000 |
Note: The unit of observation is facility-months. The number of facility-months varies for each outcome largely because some facilities do not report on all indicators, and the early ANC measure was introduced to the facility-reporting systems later than the other indicators.
The monthly means of each outcome variable over time are presented in Fig. 1 (for the non-mortality outcomes) and Fig. 2 (for the mortality outcomes). The vertical line at January 2009 represents the time the project reached saturation. These graphs concur with data presented in Table 3 and provide a visual presentation of monthly variation in each of the outcomes.
Figure 1.
Means of the non-mortality outcomes over time (%). Note: the left y-axis is for the skilled delivery and early ANC outcomes and the right y-axis is for the underweight outcome.
Figure 2.
Means of the mortality outcomes over time (per 1000).
Table 4 presents the multivariable results for the non-mortality outcomes (early ANC, skilled delivery and underweight among infants) and Table 5 presents the multivariable results for the neonatal mortality and infant mortality outcomes. Autocorrelation was first tested for and then corrected by using Prais–Winston regression with a Cochran-Orcutt transformation. Several of the change categories and trend variables (change category × time interactions) were significantly associated with the outcomes. Notably, the early pregnancy identification change category had a strong positive effect on skilled delivery (β2 = 1.36, P = 0.07). The early pregnancy identification trend variable had a negative effect on skilled delivery (β3 = −0.21, P = 0.03) suggesting that the influence of early pregnancy identification changes tapers off slightly after the immediate impact. The post-change slope was positive (β2+ β3 = 1.15); however, indicating overall increased skilled delivery over time.
Table 4.
Multivariate regression results of the non-mortality outcomes: coefficients and confidence intervals for the change categories and trend
| Early ANC, β (95% CI) | Early ANC, β (95% CI) | Early ANC, β (95% CI) | Skilled delivery, β (95% CI) | Skilled delivery, β (95% CI) | Skilled delivery, β (95% CI) | Underweight in infants, β (95% CI) | Underweight in infants, β (95% CI) | Underweight in infants, β (95% CI) | Underweight in infants, β (95% CI) | |
|---|---|---|---|---|---|---|---|---|---|---|
| Early pregnancy identification change | 0.038 (−1.213, 1.289) | 1.356* (−0.098, 2.810) | 0.088 (−0.087, 0.264) | |||||||
| Early pregnancy identification trend | −0.003 (−0.174, 0.168) | −0.215** (−0.411, −0.019) | −0.011 (−0.034, 0.013) | |||||||
| Four ANC visits change | −0.687 (−1.923, 0.549) | 0.524 (−0.853, 1.902) | 0.095 (−0.079, 0.269) | |||||||
| Four ANC visits trend | 0.095 (−0.073, 0.264) | −0.062 (−0.250, 0.126) | −0.012 (−0.035, 0.012) | |||||||
| Skilled delivery change | −0.572 (−1.526, 0.382) | 0.042 (−0.107, 0.191) | ||||||||
| Skilled delivery trend | 0.066 (−0.069, 0.200) | 0.008 (−0.006, 0.021) | ||||||||
| PNC Day 1–2 change | 0.100* (−0.006, 0.207) | |||||||||
| PNC Day 1–2 trend | −0.014* (−0.028, 0.001) | |||||||||
| PNC Day 6–7 change | 0.104* (−0.008, 0.216) | |||||||||
| PNC Day 6–7 trend | −0.014* (−0.029, 0.002) | |||||||||
| Catholic vs. government | −0.113** (−0.222, −0.005) | −0.135*** (−0.235, −0.035) | −0.141*** (−0.245, −0.036) | 0.087 (−0.035, 0.208) | 0.027 (−0.095, 0.149) | 0.046 (−0.084, 0.176) | −0.004 (−0.016, 0.007) | −0.002 (−0.013, 0.009) | −0.002 (−0.014, 0.009) | −0.002 (−0.013, 0.009) |
| Hospital vs. health center | 0.033 (−0.194, 0.260) | 0.074 (−0.147, 0.296) | 0.095 (−0.136, 0.327) | 0.586**** (0.339, 0.832) | 0.655**** (0.391, 0.919) | 0.584**** (0.298, 0.871) | 0.030** (0.006, 0.053) | 0.024** (0.001, 0.047) | 0.026** (0.002, 0.049) | 0.025** (0.002, 0.048) |
| Remoteness (per 10 km) | 0.005 (−0.015, 0.025) | 0.008 (−0.011, 0.028) | 0.008 (−0.011, 0.028) | 0.015 (−0.006, 0.037) | 0.012 (−0.012, 0.035) | 0.016 (−0.007, 0.039) | 0.002 (−0.000, 0.004) | 0.001 (−0.001, 0.003) | 0.001 (−0.001, 0.003) | 0.001 (−0.001, 0.003) |
| Catchment population (per 1000) | 0.001 (−0.010, 0.011) | 0.000 (−0.011, 0.011) | −0.001 (−0.011, 0.010) | −0.025*** (−0.039, −0.011) | −0.023**** (−0.038, −0.008) | −0.021*** (−0.037, −0.006) | −0.001 (−0.002, 0.000) | −0.001 (−0.002, 0.000) | −0.001 (−0.002, 0.000) | −0.001 (−0.002, 0.000) |
| Project officer | ||||||||||
| Project officer 2 | −0.209*** (−0.330, −0.088) | −0.208*** (−0.332, −0.084) | −0.219*** (−0.355, −0.082) | −0.145* (−0.313, 0.023) | −0.128 (−0.306, 0.050) | −0.104 (−0.294, 0.087) | −0.022*** (−0.035, −0.009) | −0.021*** (−0.034, −0.008) | −0.022*** (−0.035, −0.008) | −0.021*** (−0.034, −0.008) |
| Project officer 3 | −0.406**** (−0.579, −0.232) | −0.414**** (−0.594, −0.234) | −0.430**** (−0.613, −0.246) | −0.424**** (−0.630, −0.218) | −0.437**** (−0.659, −0.215) | −0.391*** (−0.624, −0.158) | −0.021** (−0.040, −0.003) | −0.019** (−0.038, −0.000) | −0.020** (−0.040, −0.001) | −0.020** (−0.039, −0.001) |
| Number of staff | −0.006 (−0.017, 0.006) | −0.006 (−0.018, 0.006) | −0.006 (−0.018, 0.006) | 0.013* (−0.001, 0.027) | 0.009 (−0.005, 0.024) | 0.009 (−0.005, 0.024) | −0.000 (−0.002, 0.001) | −0.001 (−0.002, 0.001) | −0.000 (−0.002, 0.001) | −0.000 (−0.002, 0.001) |
| Number of site visits | 0.001 (−0.027, 0.028) | 0.003 (−0.026, 0.032) | 0.003 (−0.025, 0.030) | 0.018 (−0.009, 0.046) | 0.013 (−0.018, 0.044) | 0.018 (−0.012, 0.048) | 0.002 (−0.001, 0.005) | 0.002 (−0.001, 0.005) | 0.002 (−0.001, 0.004) | 0.002 (−0.001, 0.004) |
| QI leader: midwife vs. non-midwife | 0.020 (−0.059, 0.099) | 0.026 (−0.063, 0.114) | 0.020 (−0.061, 0.101) | 0.219**** (0.125, 0.314) | 0.217**** (0.112, 0.323) | 0.249**** (0.146, 0.352) | −0.001 (−0.010, 0.008) | −0.000 (−0.009, 0.009) | −0.001 (−0.010, 0.007) | −0.001 (−0.010, 0.008) |
Note: all models include controls for time, time2 and change × time2, but these variables are not included in the table.
*P ≤ 0.10, **P ≤ 0.05, ***P ≤ 0.01, ****P ≤ 0.001
Table 5.
Multivariate regression results of mortality outcomes: coefficients and confidence intervals for the change categories and trend
| Neonatal mortality, β (95% CI) | Neonatal mortality, β (95% CI) | Neonatal mortality, β (95% CI) | Neonatal mortality, β (95% CI) | Neonatal mortality, β (95% CI) | Infant mortality, β (95% CI) | Infant mortality, β (95% CI) | Infant mortality, β (95% CI) | Infant mortality, β (95% CI) | Infant mortality, β (95% CI) | |
|---|---|---|---|---|---|---|---|---|---|---|
| Early pregnancy identification change | −0.001 (−0.031, 0.030) | 0.000 (−0.035, 0.035) | ||||||||
| Early pregnancy identification trend | −0.000 (−0.004, 0.004) | −0.000 (−0.005, 0.005) | ||||||||
| Four ANC visits change | 0.000 (−0.028, 0.029) | 0.001 (−0.031, 0.033) | ||||||||
| Four ANC visits trend | −0.000 (−0.004, 0.004) | −0.000 (−0.005, 0.004) | ||||||||
| Skilled delivery change | −0.009 (−0.029, 0.012) | −0.007 (−0.031, 0.016) | ||||||||
| Skilled delivery trend | 0.001 (−0.001, 0.004) | 0.001 (−0.002, 0.004) | ||||||||
| PNC Day 1–2 change | −0.006 (−0.024, 0.012) | −0.004 (−0.024, 0.015) | ||||||||
| PNC Day 1–2 trend | 0.001 (−0.001, 0.003) | 0.001 (−0.002, 0.003) | ||||||||
| PNC Day 6–7 change | −0.007 (−0.025, 0.012) | −0.005 (−0.026, 0.016) | ||||||||
| PNC Day 6–7 trend | 0.001 (−0.002, 0.004) | 0.001 (−0.002, 0.004) | ||||||||
| Catholic vs. government | 0.001 (−0.001, 0.003) | 0.001 (−0.001, 0.003) | 0.001 (−0.001, 0.003) | 0.001 (−0.001, 0.003) | 0.001 (−0001, 0.003) | 0.001 (−0.001, 0.003) | 0.001 (−0.001, 0.003) | 0.001 (−0.001, 0.003) | 0.001 (−0.001, 0.003) | 0.001 (−0.001, 0.003) |
| Hospital vs. health center | 0.012**** (0.008, 0.016) | 0.012**** (0.008, 0.016) | 0.012**** (0.008, 0.016) | 0.012**** (0.008, 0.016) | 0.012**** (0.008, 0.016) | 0.025**** (0.022, 0.029) | 0.025**** (0.022, 0.029) | 0.025**** (0.021, 0.29) | 0.026**** (0.022, , 0.030) | 0.026**** (0.022, 0.029) |
| Remoteness (per 10 km) | −0.000 (−0.000, 0.000) | −0.000 (−0.001, 0.000) | −0.000 (−0.000, 0.000) | −0.000 (−0.000, 0.000) | −0.000 (−0.001, 0.000) | −0.000 (−0.000, 0.000) | −0.000 (−0.000, 0.000) | −0.000 (−0.000, 0.000) | −0.000 (−0.000, 0.000) | −0.000 (−0.000, 0.000) |
| Catchment population (per 1000) | 0.000 (−0.000, 0.000) | 0.000 (−0.000, 0.000) | 0.000 (−0.000, 0.000) | 0.000 (−0.000, 0.000) | 0.000 (−0.000, 0.000) | 0.000 (−0.000, 0.000) | 0.000 (−0.000, 0.000) | 0.000 (−0.000, 0.000) | 0.000 (−0.000, 0.000) | 0.000 (−0.000, 0.000) |
| Project officer | ||||||||||
| Project officer 2 | 0.000 (−0.002, 0.003) | 0.000 (−0.002, 0.003) | 0.000 (−0.003, 0.003) | −0.000 (−0.003, 0.003) | 0.000 (−0.002, 0.003) | 0.000 (−0.003, 0.003) | 0.000 (−0.003, 0.003) | 0.000 (−0.003, 0.003) | 0.000 (−0.003, 0.003) | 0.000 (−0.003, 0.003) |
| Project officer 3 | 0.002 (−0.001, 0.006) | 0.002 (−0.001, 0.006) | 0.002 (−0.001, 0.006) | 0.002 (−0.001, 0.006) | 0.002 (−0.001, 0.006) | 0.002 (−0.002, 0.005) | 0.002 (−0.002, 0.005) | 0.002 (−0.002, 0.005) | 0.002 (−0.002, 0.005) | 0.002 (−0.002, 0.005) |
| Number of staff | −0.000 (−0.000, 0.000) | −0.000 (−0.000, 0.000) | −0.000 (−0.000, 0.000) | −0.000 (−0.000, 0.000) | −0.000 (−0.000, 0.000) | −0.000 (−0.000, 0.000) | −0.000 (−0.000, 0.000) | −0.000 (−0.000, 0.000) | −0.000 (−0.000, 0.000) | −0.000 (−0.000, 0.000) |
| Number of site visits | −0.000 (−0.001, 0.000) | −0.000 (−0.001, 0.000) | −0.000 (−0.001, 0.000) | −0.000 (−0.001, 0.000) | −0.000 (−0.001.000) | −0.000 (−0.001, 0.000) | −0.000 (−0.001, 0.000) | −0.000 (−0.001, 0.000) | −0.000 (−0.001, 0.000) | −0.000 (−0.001, 0.000) |
| QI leader: midwife vs. not midwife | 0.002** (0.000, 0.003) | 0.002* (−0.000, 0.003) | 0.002** (0.000, 0.003) | 0.002** (0.000, 0.003) | 0.002** (0.000, 0.003) | 0.001 (0.001, 0.003) | 0.001 (−0.001, 0.003) | 0.001 (−0.000, 0.003) | 0.001 (−0.001, 0.003) | 0.001 (−0.000, 0.003) |
Note: all models include controls for time, time2 and change × time2, but these variables are not included in the model.
*P ≤ 0.10, **P ≤ 0.05, ***P ≤ 0.01, ****P ≤ 0.001.
The PNC change activities were significantly associated with an increased monthly percentage of underweight infants attending the child wellness clinics. The significance of both the PNC trend variables indicates that the immediate increase drops off in subsequent months. However, the overall post-change slopes (β2 + β3) were slightly positive at 0.09 for both the PNC Day 1–2 and PNC Day 6–7 indicating a slightly increased percentage of underweight infants over time.
There were no significant associations between the changes and the mortality outcomes. However, the associations were generally in the expected direction.
In addition to studying the specific change categories tested, the total monthly number of change categories tested was also evaluated in a separate model for each outcome. This was done to understand whether facilities that tested more changes had more improvements in outcomes than facilities that tested fewer changes. This time-varying variable had a monthly mean of 1.3 and a range of 0–4. The total number of monthly change categories tested was included in multivariable regression models with the program and facility variables and was found to be associated with one outcome, skilled delivery (β = 0.05, P = 0.01) (results not shown).
The key findings for the program and facility-level variables, which were included in the multivariable models are summarized here. Catholic health facilities had less early ANC care seeking but greater skilled delivery than government facilities. Hospitals, which often have the sickest patients, had higher mortality and a higher percentage of underweight infants attending child wellness clinics than health centers. Hospitals also had a greater percentage of women in their catchment area delivering with a skilled attendant. The individual project officer had an influence on all the non-mortality outcomes. The profession of the QI team leader had an influence on two of the outcomes. Facilities with a midwife as the QI team leader had increased skilled delivery but higher neonatal mortality.
Discussion
As more middle- and low-income countries adopt QI methodologies, there is an urgent need to evaluate the approaches to understand not just if, but how, such programs could lead to improved health outcomes [26]. Results of this evaluation indicate that the early phase of Project Fives Alive! had an influence on key maternal and child health outcomes. The PNC Day 1–2 and PNC Day 6–7 change categories were significantly associated with an increased percent of underweight infants attending wellness clinics. Perhaps, providers encouraged caregivers to bring infants for follow-up care, particularly those who were low-birth weight. Identifying women and encouraging them to use ANC early in their pregnancy was an effective means of increasing skilled delivery. A greater number of monthly change categories tested was also associated with increased skilled delivery. Though no change categories were associated with mortality, there were declines in both neonatal and infant mortality from the pre-intervention to post-intervention period.
Program and facility-level factors were also important. Changes tested in hospitals may need to be different than those tested in health centers since hospitals tend to receive the sickest patients. (In response to the findings the project is testing a package of changes to address hospital level infant and child mortality in the current scale-up phase.) The individual project officer had an influence on three of the five outcomes, thus indicating the importance of trainings for and supervision of the project staff member who interacts closely with the QI team. Other factors beyond competence may also explain this finding including the nature of the relationship developed between the QI team and the project officer. Facilities that had a midwife as a QI team leader had increased skilled delivery but slightly higher neonatal mortality. It would be important for the project to think about diversity of occupations within a QI team and also perhaps having teams being co-led by individuals in different professions. Consideration should also be given to the midwives' skills in addressing the most common causes of neonatal deaths in low-income countries [27]. A related finding was that Catholic facilities had a greater percentage of skilled deliveries but higher neonatal mortality. Again this stresses the importance of training in neonatal health.
There are several limitations to this analysis. First without comparison facilities it is not possible to control for all potentially confounding factors. However, even without comparison facilities, the methodology of using repeated (monthly) observations from the same facilities both pre- and post-intervention offers a strong evaluation design [28]. A second limitation is that only a maximum of 9 months of pre-intervention data were available. Without a full 12 months of pre-intervention data it is not possible to take into account the full effects of seasonal variation in health outcomes, but our pre-intervention data does include several months of both the rainy and dry seasons. In Northern Ghana the rainy season is from March to September. Given that our full implementation period begins in January 2009 and the rainy season begins in March, seasonality might actually be making it more difficult to see a program effect. (Mortality often increases in the rainy season due to malaria, and use of health services may decline due to poor roads, etc.) This could also be a reason we did not see an effect on the mortality analysis, which was also limited by low power due to low monthly numbers of deaths in the health facilities.
A general limitation of facility-level data is inaccuracies due to transcription errors or missing data as information is transcribed from facility registrars to facility-level monthly reporting forms which are then sent to the district level, compiled, and sent to the national level [29]. A recent study in South Africa suggests that the attention to data encouraged by the QI process can itself greatly improve data quality [30]. In that study, as in this one, the issue of data quality was specifically addressed from an early stage through training and coaching of district health information officers on data QI methods. In addition, potential errors were somewhat mitigated because the facility data were directly obtained at the source, the registrars.
Despite the limitations of this evaluation, this paper presents an impact evaluation of a large, ongoing QI project in a low-income setting. Findings from the early phase of Project Fives Alive! indicate that the QI approach of testing simple changes identified by local staff has the potential to lead to improved health outcomes during the national scale-up of this project and also in other middle- and low-income countries.
Funding
This work was supported by the Bill and Melinda Gates Foundation.
Acknowledgements
The authors wish to thank staff members of Project Fives Alive! who assisted in obtaining the data for this evaluation. We also thank Lloyd Provost and the anonymous reviewers for their helpful suggestions on the manuscript. We are grateful to the Carolina Population Center (R24 HD050924) for general support.
References
- 1.Deming WE. Out of the Crisis. Cambridge: MIT Press; 2000. [Google Scholar]
- 2.IOM. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: Institute of Medicine; 2001. [Google Scholar]
- 3.Berwick DM, Godfrey AB, Roessner J. Curing Health Care: New Strategies for Quality Improvement. San Francisco, CA: Jossey-Bass; 2002. [Google Scholar]
- 4.WHO. Nobody's Business. Strengthening Health Systems to Improve Health Outcomes. WHO's Framework for Action. Geneva: WHO; 2007. [Google Scholar]
- 5.Leatherman S, Ferris TG, Berwick D, et al. The role of quality improvement in strengthening health systems in developing countries. In J Qual in Health Care. 2010;22:237–43. doi: 10.1093/intqhc/mzq028. 10.1093/intqhc/mzq028. [DOI] [PubMed] [Google Scholar]
- 6.Campbell H, Duke T, Weber M, et al. Global initiative for improving hospital care for children: state of the art and future prospects. Pediatrics. 2008;121:984–92. doi: 10.1542/peds.2007-1395. doi:10.1542/peds.2007-1395. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Hoskins EJ, Abdul al-Hamid Noor F, Ghasib SH. Implementing TQM in a military hospital in Saudi Arabia. Jt Comm J Qual Improv. 1994;20:454–64. doi: 10.1016/s1070-3241(16)30090-6. [DOI] [PubMed] [Google Scholar]
- 8.Dohile MB, Mielke E, Mumba FK, et al. Using practical quality improvement approaches and tools in reproductive health services in East Africa. JT Comm J Qual Improv. 1999;25:574–87. doi: 10.1016/s1070-3241(16)30471-0. [DOI] [PubMed] [Google Scholar]
- 9.Nolan T, Angos P, Cunha AJ, et al. Quality of hospital care for seriously ill children in less developed countries. Lancet. 2001;357:106–10. doi: 10.1016/S0140-6736(00)03542-X. doi:10.1016/S0140-6736(00)03542-X. [DOI] [PubMed] [Google Scholar]
- 10.Hermida J, Robalino ME. Increasing compliance with maternal and child care quality standards in Ecuardo. Int J Qual Health Care. 2002;14(Suppl. 1):25–34. doi: 10.1093/intqhc/14.suppl_1.25. doi:10.1093/intqhc/14.suppl_1.25. [DOI] [PubMed] [Google Scholar]
- 11.Legros S, Tawfik Y, Abdallah H, et al. Evaluation of the quality assurance project and BASICS joint project in Niger. Int J Qual Health Care. 2002;14(Suppl. 1):97–104. doi: 10.1093/intqhc/14.suppl_1.97. doi:10.1093/intqhc/14.suppl_1.97. [DOI] [PubMed] [Google Scholar]
- 12.du Mortier S, Arpagaus M. Quality improvement programme on the frontline: an international committee of the red cross experience in the Democratic Republic of Congo. Int J Qual Health Care. 2005;17:293–300. doi: 10.1093/intqhc/mzi042. doi:10.1093/intqhc/mzi042. [DOI] [PubMed] [Google Scholar]
- 13.Mohammadi SM, Mohammadi SF, Hedges JR, et al. Introduction of a quality improvement program in a children's hospital in Tehran; design, implementation, evaluation and lessons learned. Int J Qual Health Care. 2007;19:237–43. doi: 10.1093/intqhc/mzm021. doi:10.1093/intqhc/mzm021. [DOI] [PubMed] [Google Scholar]
- 14.Bradley E, Hartwig KA, Rowe LA, et al. Hospital quality improvement in Ethiopia: a partnership-mentoring model. Int J Qual Health Care. 2008;20:392–99. doi: 10.1093/intqhc/mzn042. doi:10.1093/intqhc/mzn042. [DOI] [PubMed] [Google Scholar]
- 15.GHS. Annual report. 2007.
- 16.Ghana Statistical Service (GSS), Ghana Health Service (GHS), and ICF Macro. Accra, Ghana: GSS, GHS and ICF Macro; 2009. Ghana demographic and health survey 2008. [Google Scholar]
- 17.WHO. Trends in Maternal Mortality 1990–2008. Geneva: WHO; 2010. [Google Scholar]
- 18.Langley GJ. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. 2nd edn. San Francisco: Jossey-Bass; 2009. [Google Scholar]
- 19.Deming WE. The New Economics for Industry, Government, Education. Cambridge: MIT Press; 1994. [Google Scholar]
- 20.Juran JM. Juran on Leadership for Quality: An Executive Handbook. New York: The Free Press; 1989. [Google Scholar]
- 21.Institute for Healthcare Improvement. The Breakthrough Series: IHI's Collaborative Model for Achieving Breakthrough Performance. Cambridge: Institute for Healthcare Improvement; 2003. www.ihi.org . [Google Scholar]
- 22.Twun-Danso N, Akanlu G, Osafo E, et al. A nationwide quality improvement project to accelerate Ghana's progress towards Millennium Development Goal 4: design and implementation progress. Int J Qual Health Care. 2012;24:601–6. doi: 10.1093/intqhc/mzs060. doi:10.1093/intqhc/mzs060. [DOI] [PubMed] [Google Scholar]
- 23.Graham W, Bell JS, Bullough CHW. Can skilled attendance at delivery reduce maternal mortality in developing countries. In: De Brouwere V, Van Lerberghe W, editors. Safe Motherhood Strategies: A Review of the Evidence. Antwerp, Belguim: TTG Press; 2001. [Google Scholar]
- 24.Campbell O, Graham W on Behalf of the Lancet Maternal Survival Series Steering Group. Strategies for reducing maternal mortality: getting on with what works. Lancet. 2006;368:1284–99. doi: 10.1016/S0140-6736(06)69381-1. doi:10.1016/S0140-6736(06)69381-1. [DOI] [PubMed] [Google Scholar]
- 25.Campbell DT, Stanley JC. Experimental and Quasi-Experimental Designs for Research. Boston: Houghton Mifflin; 1963. [Google Scholar]
- 26.Althabe F, Bergel E, Cafferata ML, et al. Strategies for improving the quality of health care in maternal and child health in low- and middle-income countries: an overview of systematic reviews. Paediatr Perinat Epidemiol. 2008;22(Suppl. 1):42–60. doi: 10.1111/j.1365-3016.2007.00912.x. doi:10.1111/j.1365-3016.2007.00912.x. [DOI] [PubMed] [Google Scholar]
- 27.Lawn JE, Kerber K, Enweronu-Laryea C, et al. 3.6 million neonatal deaths—what is progressing and what is not? Semin Perinatal. 2010;34:371–86. doi: 10.1053/j.semperi.2010.09.011. doi:10.1053/j.semperi.2010.09.011. [DOI] [PubMed] [Google Scholar]
- 28.Biglan A, Ary D, Wagenaar AC. The value of interrupted time series experiments for community intervention research. Prev Sci. 2000;1:31–49. doi: 10.1023/a:1010024016308. doi:10.1023/A:1010024016308. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Mate KS, Bennett B, Mphatswe W, et al. Challenges for routine health system data management in a large public programme to prevent mother-to-Child HIV transmission in South Africa. PLoS One. 2009;4:e5483. doi: 10.1371/journal.pone.0005483. doi:10.1371/journal.pone.0005483. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Mphatswe W, Mate KS, Bennett B, et al. Improving public health information: a data quality intervention in KwaZulu-Natal. South Africa Bull World Health Organ. 2012;90:176–82. doi: 10.2471/BLT.11.092759. doi:10.2471/BLT.11.092759. [DOI] [PMC free article] [PubMed] [Google Scholar]